If you ever work with terabyte-scale data, try to avoid unnecessary unnesting/unpacking arrays and date ranges. If you have no choice, materialize the unnested result and partition and cluster it accordingly in preparation for any further joins.

With that bold statement out there, here’s a BigQuery lesson drawn from real-world experience.

Imagine an SCD-2 table with tens of millions of rows containing product data.

For particular subintervals (even individual days) of the [valid_from, valid_to) period, we need to set some attributes (think: marking that the product is on sale, or is getting ready to be discontinued).

The simple and straightforward solution here would be to unpack the intervals with something like UNNEST(GENERATE_DATE_ARRAY(valid_from, valid_to)). Joining with a date dimension would work in a similar fashion.

This means that it is easy to now look up these attributes at (grain + valid_date).

But now we could be talking about several billion rows. Which you might join with other big tables.

The issues with this approach are quite obvious. Say we have a two-year interval. Unpacking this period would mean having 730 rows instead of one, while a majority of them would just be repeated data — there’s no actual change, just some spot changes for individual days or subintervals.

We can of course go ahead and compact this data back using an algorithm like the one I presented in one of my previous posts, but regardless of whether we do that or not, the whole processing would be incredibly expensive and inefficient.

So, is there a solution here? Sometimes there might be.

Say for the two-year interval [2021-01-01, 2023-01-01], we’d need to mark the subperiod [2021-04-15, 2021-05-15] as the period when the product is on sale.

This data can be accurately expressed as the following intervals:

[2021-01-01 → 2021-04-15),

[2021-04-15 → 2021-05-15),

[2021-05-15 → 2023-01-01).

So in fact, we’re splitting the intervals into smaller subintervals based on what was true at each moment. Instead of unpacking and getting hundreds of records, we need only 3. I’ve illustrated this approach in another previous post.

Such a difference in cardinality has deep performance implications, allowing us to process data way more efficiently.

While this might be an unusual corner case, my initial opinion stands — if you can get away without UNNESTing your array or unpacking a date interval, you’ll manage to keep cardinality in check and save a lot of processing power.

If there’s no way around it for your particular case, at the very least, materialize the result (I know, it’s going to be a lot of storage) and tailor the partitioning and clustering according to the querying (especially joining with other big+ tables) downstream.

There’s one concept that is central when working with data and goes beyond specific technology or #SQL dialects. You just can’t miss it. I’m talking about the grain—the granularity of the data.

It’s essentially answering the question: what does one row of this data represent? Is it an order, a line item in an order, or a status update for a line item?

This is so important because you need to know it to query and ETL data correctly. If not, you’ll end up guessing, and might never be 100% sure you made the right choice. Misunderstanding the grain of a table is how you get faulty joins and duplicates—and why DISTINCT is often slapped on indiscriminately to try and fix the problem.

You need to understand the grain to know whether you should COUNT or COUNT DISTINCT, and whether it’s safe to SUM or JOIN data.

Now, if you’ve ever tried to figure out the grain of an external data source with no documentation, you know it can get tricky. Here’s my approach:

➡️ Of course, find out who maintains the data, get your answers, and give feedback on any issues you find. Then ask for documentation or, in extremis, write down your findings. The next person will thank you!

➡️ If that doesn’t work, start with assumptions based on the columns but test to validate or invalidate what you think is true.

➡️ Iterate fast.

➡️ If you see DISTINCT on anything with more than a couple of columns, run the other way, catch your breath, and return to replace them a methodical approach to duplication.

➡️ Leverage ranking functions like ROW_NUMBER() and DENSE_RANK().

➡️ Use COUNT(1) and GROUP BY to test your hypothesis.

➡️ ????

➡️ PROFIT.

I find that working with data daily significantly influences how you perceive the world around you. You begin to notice patterns and details you might otherwise overlook.

Take retail, for example.

In our places, it's common for pharmacies, gas stations, and supermarket chains to offer loyalty cards. Some of these programs are worthwhile, while others are not. Personally, I'm wary of them for privacy reasons.

From an outside viewer like me, their goal lies on the surface: boosting customer loyalty and collecting user data. But why is this data collected? Spending habits, products frequently bought together, evolution of demand patterns - for retailers, this information is a gold mine.

Subscribe now

But what's even more interesting is the data collection process. In some places, clerks will scan their own loyalty cards for customers who don't have one, accumulating points for themselves. That's a huge outlier raking tens and hundreds of transactions in a day. Filter that one out?

Some people even have multiple loyalty cards for the same store, especially when no form is required to get one—they just hand you a new card. How do you accurately identify a unique buyer in this case?

In other cases, an entire family might use the same loyalty card. Just imagine the variety of consumer patterns for that one.

The data teams working with such information must have fascinating day-to-day tasks, involving elaborate data cleaning and processing setups.

Whenever I see the events being collected in real life, I'm already thinking about someone analyzing them with SQL somewhere.

So, as a Data Analyst, Data Engineer, or Data Scientist, it might be beneficial to consider how the data was collected the next time you work on a data task.

So I spend quite a bit of time optimizing SQL data processing pipelines in BigQuery. Scale-wise, I might have an occasional encounter with a 10TB table. I have limited knowledge on what it does 'under the hood', but I find that there's a couple of simple, common sense things you can do from the start.

So here's my thinking when I'm trying to optimize such a problematic query.

Work with less

The simplest and most efficient optimization is to work with less data from the beginning. Perhaps you don't need that much data for the task you're performing.

Say you're processing month-over-month calculations. Do you really need that 3-year old data processed in this query? If not, filter those rows out.

How about the columns? Do you really need those? Keep only what you need.

Do less work

Another big trick is to ensure there is less work do be done. So, can you rewrite the problem you're solving in a way that makes it easier to find the answer (even for you as a human with a pen & paper / Excel )?

For example, I'd always try to limit the number of column I aggregate by, keeping only the essential. I wouldn't need the customer's country and city when aggregating their transactions, I can always look up that later.

Or take Window Functions. They take quite a toll on the performance at these high volumes. So I would use them sparingly. Check out this example of how I've replaced multiple WINDOW function calls with just a single one + PIVOT.

Also, aim to reduce the number of rows that you are working with as early as possible - that is, push aggregation as early as possible.

Is there any repetitive work involved?

Is there any part of the processing that you see is done multiple times? It doesn't need to be identical, but can you find a way to make the computation happen just once? Can you re-purpose a staging table to to serve multiple needs at the same time?

In a more practical fashion, a normal (non-recursive) CTE in BigQuery is evaluated once per each each call, so you're effectively doing that work over and over again.

Can you split things up?

When combining multiple big tables and are doing a lot of operations on this data, things are bound to get ugly.

One way to work around that is to split the process into logical steps and components.

Extract big steps into a separate staging table that is as barebones as possible but is properly partitioned and clustered (on indexed, depending on your platform).

Is there any underlying problem making the data that big?

I’ve recently was struggling with optimizing something that produced a 17TB table, only to realise that the data sources are nowhere that big - and a bad join made the data volume to explode.

So maybe performance is not your main problem to fix?

Check for yourself

Pick a representative data slice. Turn off query caching in your window. Try out different approaches and work out which one is the best.

Check for different data volumes. Do you findings till hold? As learned multiple times on my own skin, test your assumptions and pick a best approach based on empirical results.

Think about maintainability

While this is second-to-last, it should rank closer to the top in terms of importance. Is whatever you’ve achieved there understood by your peers in the team, or whoever might come to replace you? Will they be able to maintain, extend or fix your solution should the conditions change?

No one likes black boxes, so make sure to do document the ‘what’ and ‘why’ of your work, ensuring your actions have a positive net impact on the process in its lifetime.

In lieu of a conclusion: invest your time wisely

Trying to optimize something can be interesting. Time will fly by. Don't get yourself sucked in and keep an eye on the end target. Is all this effort on optimization worth it? Don't spent days of working to save 5$ in processing costs.

Perhaps you should prioritize improving a scheduled query running every 2 hours than a vastly bigger one that is used for yearly reporting?

After all, the smaller one runs 12 * 365 = 4380 times a year, so a small saving here might mean a big difference in the grand scheme of things.

Leave a comment

So you have to write a SQL query and after tirelessly working on it, you get more rows than you expected. It looks otherwise correct, but some of the records you see multiple times.

Now, while the temptation to just slap a DISTINCT in that final SELECT is enormous, don't succumb to it (yes I wrote this word).

Indiscriminate usage of DISTINCT might cover up a few problems with your query or can even come back to bite you.

1. Bad join

It could be that one of your joins is incorrect or you might need, for example, an extra condition there.

Or you've unwillingly created a self-join like a.id = a.id 😁 Been there, done that.

2. Incorrect grain

One of the tables you've retrieved data from has a finer granularity that you think, for example, multiple product variants that share the same id instead of just one product id per row.

Or you're getting multiple status updates throughout the lifetime of an order. Do you need to pick only one of these multiple rows, say the last status update?

Or you're trying to join month-level and day-level attributes together.

Do you need to aggregate or split one of the inputs to match the others?

3. Are you working with nested data?

If you're working with nested data and proceed to unpack it i.e. UNNEST an ARRAY in BigQuery, you'll get the non-nested column repeated multiple times. Ensure you’re correctly processing the nested data as per your requirements.

What should I do? How can I debug a problem with duplicates?

• Pick a subset of data that you can simply analyze at a glance: a day, a single store, a product or one single order.

• Filter the data for just that. Are you getting the expected amount or rows? Is the granularity what you expected?

• Have a look at each table individually that you are using in your query.

• If you know an exact number of rows you should be seeing as output (e.g. total number of orders) you could comment out all the columns and put a SELECT COUNT(1) instead. Comment out the joined tables and add them back one by one. At what point in time does the COUNT result change for the worse?

If you still need to de-duplicate, say because the data is bad at the source (and you know it for a fact) or you receive multiple status updates but only want the last one - do so in an explicit manner, clearly showing how you are doing it and ideally explain in a short comment why you are doing it.

This could look like the following:

QUALIFY ROW_NUMBER() OVER (PARTITION BY order_id ORDER BY status_update_ts DESC)=1

If you're not familiar with QUALIFY and other ways to de-duplicate, check out the comprehensive Medium article about the many ways you can de-duplicate and their pros & cons.

Happy querying!

Starting your journey in Analytics can feel like standing at the base of a big mountain. I’ve seen it around me, and I’ve experienced it myself: there’s an overwhelming amount of information and an endless list of courses and tutorials you can take.

As a Finance undergraduate transitioning into Analytics, I was confused about where to start learning. I signed up for every platform I could find – Coursera, Team Treehouse, Pluralsight, Linux Academy, Cloud Guru, you name it. At first, I learned a lot of things, picked up the basics on Python, brushed up my SQL, learned about what git was. But then, as I was going through them, it felt increasingly like the goal was collecting the certificates at the end, not the journey itself.

Don't Fall into the Trap

It's easy to fall into the trap of this "tutorial hell" – a cycle where you keep consuming courses without ever applying what you've learned. Sure, I’d want a course to learn the basics of Python or how a Data Engineering stack on GCP looks like, but don’t overdo it. Practice > theory.

Let’s see on how to avoid this common pitfall and make the most of your learning journey.

Start with the Basics

Surely being good with Spark or ${YOUR_COOL_NEW_TOOL} is good and commands a bigger salary, but you still need to have your basics right first. A solid foundation in SQL, a decent grasp of Python, a basic understanding of data modeling, working knowledge of git, and minimal familiarity with products in one big cloud Analytics stack - these could be a good foundation when starting your career.

Choose Wisely

Your time and money are limited resources, so start by picking a good course for your topic. Index the information in your mind: understand the features of the tools or frameworks and use cases where they are suitable. Focus on grasping the bigger picture rather than memorizing specifics. You can always look up those later. Nobody wants you to know the exact order of arguments for a specific function; you just need to remember it exists and it can help in X or Y situation.

Get Hands-On Quickly

One of your primary goals should be to learn the basics and then start applying them as soon as possible. There will always be gaps in your knowledge, but it's better to dive in and start doing rather than endlessly chasing another tutorial. Applying your skills quickly helps cement your understanding and builds your confidence.

Course Certificates Don’t Do Much

If you’re looking for your first job or internship, having a lot of certificates from courses might be a small advantage to your name, signaling your motivation to learn, your perseverance, and consistency. But nowadays I’m not even sure that it would matter in how modern CV screening works and with this strong competition.

Work on Personal Projects in the Long Term

Make it a habit to engage in personal projects or guided projects that simulate real-life scenarios.

These are good:

• For your understanding - you encounter problems that tutorials don’t cover and fix them

• For your confidence - you manage to make things work on your own, so you’ll be just fine in a job setting

• For your employability - it’s not only about the keyword in your resume, it’s also about being a problem solver and finding solutions to problems, business or technical

I’ve previously posted about how personal projects can help you grow as a developer:

Here’s two such projects of my own:

• A Portable data stack “Analytics in a Box” with Docker, Dagster, dbt, DuckDB, PostgreSQL and Superset

• A similar one with Docker, Mage, dbt, DuckDB,and Superset

It’s time to build

The key takeaway here is to get building. Start creating, experimenting, applying, and sharing what you learn. Whether it's an individual project stemming from your own idea or a guided project, the experience you gain from practical application is invaluable.

Remember, the goal is to become proficient solve real-life problems and be confident in your skills, not just to collect certificates. So, get out there and start building your path to success in Analytics! Don't give in to tutorial hell.

. So, I found a local mirror and started downloading. Anyway, it turned out openSUSE was too complex for me.

Then, I read that Ubuntu was the newbie-friendly one. Back in the day, there was a service called ShipIt, which sent free Ubuntu copies by mail. So here I am at my local post office in a small 🇲🇩 Moldovan town - they even asked me to open the package so they could check what was inside.

One of the first things I did after installing it? Figuring out how to log in to root automatically without entering the password. No need to enter the long and secure password (inspired by cartoon heroes) every time I needed to sudo apt-get install a package, I thought. I even deleted the other user so the computer would run smoother. 😁

It took me a while, but I was so happy with myself. I spent the next year tinkering and trying to install Windows games on 🐧 Linux. Then I forgot about the penguin altogether for the next 10 years until I became a data engineer.

But some of the things I learned back in the day, a good 18 years ago, like terminal basics and, more importantly, the patience to tinker with tech stuff until I got it done, have always stayed with me.

A couple of takeaways:
🔹 Never give up: keep trying until you get it done.
🔹 Don't waste your effort: focus on what you want to achieve.
🔹 Iterate fast: don't get stuck in minutiae (yes, it's my own word); try, fail, try another approach.
🔹 Learning is like investing: what you put in today will help you down the road.
🔹 Getting smart takes time: you will inevitably do dumb things like I did with the root user (which is not safe; the ⚠️ warnings are there for a reason), but you will come to know better.

The Chernobyl disaster stands out as one of the most horrific technological failures in history. I was struck by how a technology meant for good could cause such devastation when things went wrong. This sparked my interest, leading me to delve deeper into the details. There's even an exceptional TV miniseries that captures the event.

Like many catastrophic events, Chernobyl was a result of a dark combination of factors and coincidences—design flaws, human errors, negligence, technical oversights, flawed processes, and conflicting motivations.

Conflicting motivations? Yes! Despite knowing about the reactor's design flaws, people remained silent for fear of reprisal. Construction shortcuts were taken to meet deadlines and secure bonuses.

On that fateful night of April 26th, 1986, greed likely drove some to rush safety tests for bonuses, while fear paralyzed others from challenging reckless orders.

You might wonder, why does this matter now, 40 years later? And how does it relate to software?

Consider your team or company culture. Are you incentivized to prioritize doing the right thing over meeting deadlines or earning bonuses?

Is there an environment that fosters speaking up when things are amiss? Can you take responsibility for mistakes, even if it means facing criticism?

Developing quality software is not an easy task, and aligning people's interests with the greater good is crucial.

Establishing a culture of openness and transparency, alongside appropriate incentives, should be a priority for anyone leading a team.

In life and software, mishaps occur. However, when people's motivations are aligned, we can collectively work better towards the same goal.

I was following a discussion about the “Self-Service paradox”. Then I've seen another LinkedIn post about the random 'one-off' requests.

And then memories just poured in.

When I just started working in Analytics a decade ago, I was pretty hyped up with this idea of data democratization. As an analyst with too much time on my hands, I thought: with the right infra in place, everyone interested can access all data in the org (minus sensitives of course) and uncover some “insights” that will help improve how things work. Data is fair game for everyone.

Time flies. I’ve picked up SQL and with a couple of years of experience, I was working as a Business Intelligence Developer, working across multiple business areas and industries. The reality struck me.

Even apparently (for my unprepared mind) straightforward things like revenue, there was not any consensus on how it should be calculated. One person might think of it as revenue once the order has been placed, another once it was paid for and yet another one it was handed out for delivery. Or delivered? You get the idea.

Then, I could get odd ‘one-off’ request asking for this or that piece of data. Aggregated at this level, filtering out X and Y. They’re all the same but equally different. People would build entire siloed ecosystems of their own, report packs, presentation decks, all sourced from ‘raw data’ provided by engineers. It became clear that this is not a sustainable track.

Why was this happening? Maybe the analyst wasn’t comfortable with SQL or with the self-service tools of the time like SSAS Cubes + Pivots in Excel. Perhaps there is no data dictionary in place so people don't know what these columns represent. And what are these huuuge integer keys for?

Does this amount include tax or not? How about shipping? What about returns?

Or maybe this data hasn't landed yet in the Data Warehouse and we are due to integrate it sometimes Q3 in 2 years. Or maybe they don't trust out transformations?

Without proper controls, this *free-for-all* leads to conflicting metrics, a skewed understanding of how good a company is doing and overall, and overall, different yardsticks doing the measurements.

With regards to the analogy I’ve used in the title, I believe that the we should treat our Data Warehouse more like a pharmacy and less like a retail store.

Surely, there are over-the-counter products, MDs and treatments. But every one of those comes with instructions and a number of dedicated professionals that can guide and help you when needed. And for some you need a prescription.

I'm very happy when someone uses the data that I help deliver. We have a duty to act as stewards and custodians for the data we are entrusted to process, including the Analytical products it is feeding. Not restrict and keep exclusive to our silos, but inform, educate, collaborate and build for better business outcomes.

The cold winter is here in the Northern Hemisphere, but here’s a very warm ‘Welcome!’ to the second issue of Not Just SQL. It might have taken a while to write it out (busy month, right?), but here’s the latest I wanted to share with you. Let’s dig in!

Exploring SQLMesh - a dbt challenger?

Working in Data Analytics has definitely changed a lot since I first started almost a decade ago. SQL queries are now hopefully saved less often in text files and emailed. Businesses need shorter data-to-insights turnaround, better monitoring, data quality controls they can trust and of course, a better experience for the Analytics staff. This has, in turn, led to an explosion of different projects and products looking to solve a particular pain point.

One such competitive space is the data transformation framework. The elephant in the room in of course dbt, whose mother company dbt Labs is currently valued at multiple billions. There’s also Dataform, acquired by Google and integrated in the their cloud platform.

One new player on the block is SQLMesh. It has a number of very interesting promises, some that could be regarded as differentiators.

• A BIG accent on DataOps: it surely looks like this tool was built by practicing Data Engineers. Easiness of managing different environments, a Terraform-like “plan-apply” workflow to assess and apply changes, attention to concepts like incremental loading are just some of the features that stand out.

• No-copy virtual environments, which I understand Snowflake also has. This means that you not longer need to have multiple *physical* copies of the same data across environments, thus saving storage costs.

• A semantic understanding of SQL, built with SQLglot which transpiles i.e. ‘translates’ it, rather than just generates the SQL code based on a template. This allows it to distinguish between breaking and non-breaking changes (think removing a column that is used by downstream models versus removing a unused one), while also allowing for flexibility to work across SQL dialects.

• Table and column lineage: one of the most common current requirements to Modern Data stack products.

I’ve decided to try it out using this quickstart guide. The example project was using DuckDB as an analytical database, a pairing I appreciate. For starters, I’ve read a parquet file from a PostgreSQL instance, transforming the data using several steps in a pipeline.

One can use the tool with the built-in UI or the cli, and it offers an adapter to use with existing dbt models (with some changes).

The Slack community is active and was able to assist me quickly when I had questions.

While the tool is under active development and has some rough edges, it’s definitely something to keep an eye on. As with any domain, competition is very healthy for everyone involved.

If you’d like find out more, a very interesting analysis comparing three such solutions (two of which are DBT and SQLMesh, can be found below.

The Data Jargon Newsletter

DBT vs SDF vs SQLMesh

For some of you, that headline might not mean much. That’s ok, and this one might not be for you. I’m mixing in some technical posts every once and a while. This is the first of the series. If you’re reading this, I’m going to assume you know what DBT is. You don’t need to know what SQLMesh is, or have an …

Read more

3 years ago · 24 likes · 2 comments · Joe Naso

Data radar

Here’s a some more interesting projects I’ve encountered in the past few weeks.

Datadiff

As a Data Engineer, an important challenge in my work is mitigating risk as part of change management. One of the main goals there is avoiding regressions. This is typically in the form of sanity checks - think queries that would compare the state before and after changes: row counts, count of unique keys, ensuring grain alignment with the requirements, corner cases.

I’ve recently stumbled upon Datadiff by Datafold, an automated regression testing tool, as illustrated in this nice blog post. Deploy it as part of your CI/CD pipeline to get these automated checks.

Malloy

Courtesy to the Lindy effect, SQL has, in my humble opinion, quite a long lifetime ahead of it. Its low barrier of entry, ubiquity and long history (hello, late 1970s) give it quite a gravitational pull. That does not stop people from thinking about alternatives though.

I’ve recently discovered Malloy, an open-source language specifically designed to perform data processing tasks. It can of course translate to SQL. Here’s a taste of how it looks and the equivalent SQL.

Queries reproduced from Malloy github page.

run: bigquery.table('malloy-data.faa.flights') -> {
  where: origin ? 'SFO'
  group_by: carrier
  aggregate:
    flight_count is count()
    average_flight_time is flight_time.avg()
}

SELECT
   carrier,
   COUNT(*) as flight_count,
   AVG(flight_time) as average_flight_time
FROM `malloy-data.faa.flights`
WHERE origin = 'SFO'
GROUP BY carrier
ORDER BY flight_count desc         
-- malloy automatically orders by the first aggregate

Turntable

Does the phrase “how the tables have turned” qualify as SQL joke? You know, like that one with the tables being joined in bar? I’ve also discovered that a turntable is the device that plays vinyls.

On a more serious note though, Turntables is a startup promising to speed up data pipeline creation and management using AI. What caught my eye though, is their free VSCode extension, catering to dbt core users.

It offers no less that column-level lineage, an AI-powered docs auto-filling and model validation before you actually run it. Definitely trying this one.

More than just coding

• Google launches Gemini, a new GPT-4 challenger

• How an engineer saved his company half a million dollars

• Tech conference organizers create fake female speakers to increase appearance of diversity

• Gergely Orosz’s new book - The Software Engineer’s Guidebook

  How saying healthy ‘NO’s can be help you get ahead

Musings Of A Caring Techie

Strategies and examples on how to say “no” effectively and guilt-free

“Yes, sure, I can take that on. Don’t worry about it.” How many times have you said those words and immediately regretted it, knowing that your plate is already full? If you also have a hard time saying “no” to things, I’m with you. Saying “no” is a challenge many of us face…

Read more

3 years ago · 24 likes · 4 comments · Irina Stanescu

Creator update

Searching for a palatable monetization strategy

As a novice content creator, ever since I’ve started writing on a consistent basis, I also pay attention to how other people in tech/data space do it.

There seems to be an ever increasing number of people entering the game. Creating tech content can be quite resource-intensive, so it’s only normal that a lot of people with a consistent following are trying to sell you something. A course or bootcamp, a book, 1-on-1 mentoring sessions. But the important part here is how are they doing it.

How do you create a more palatable monetization strategy? Something that will allow you to earn (or supplement) your livelihood but not disappoint your audience as a creator.

A couple of days ago, I’ve read an interesting post about how an appropriate monetization strategy should look like.

If had to define my stand, I’d say:

• the creator should continue to offer good value for people not willing or not being able to pay (also because, in the attention economy, it’s might be the right business decision)

• paid products or content should be accessible to people at different price points, based on how deep they’d like to engage (the author of the above post mentions a guide → paid community → in-person retreat structure as an example) and afford to spend

• some things of course should be non-negotiable, such as having your audience best interests at heart and not promoting something that you deem immoral or harmful in any way.

This is, of course, a page still to be written for me, but I think it’s beneficial to think about these things beforehand. What would your ideal strategy be?

Tools I use as a content creator

As you may already know, in addition to this page and Linkedin, I publish on Datawise and cross-post on Data Engineer’s Notes, a Medium publication I run.

Content creators use a wide array of tools to get their job done. Now, nobody shared these with me, but I’ve seen a lot of people using them, so I’d like to share a rough list of things I use to help me create content.

Plan:

• Notion for taking notes and writing down ideas for future posts. I’m definitely using it for less than it can.

Design:

• Canva: a free graphic design tool, useful to create slides, graphs and carousel posts for Linkedin

• I’m showcasing a lot of queries to illustrate SQL concepts, snippets which I beautify with Ray and Carbon (pick one that suits you best).

• For presenting tables (typically snippet output), I’m copying the query output as JSON and using a converter to create an ASCII table. This then goes into one of the beautifiers from the previous point. Sadly this doesn’t work with STRUCTS/ARRAYS, so the last resort is taking a screenshot of it.

Analyze:

• Sadly Linkedin does not provide a straightforward way to know the exact hour something was posted, so you need a specialized tool to extract the timestamp when a particular piece of content was posted for your analytics.

Content Recap

Here’s an overview of my articles posted on Datawise since the last issue of Not Just SQL.

SQL 📰

• Correlated subqueries: what are they and how to use them?

• Using ARRAY_AGG in BigQuery

• Cleaner window functions calls with named windows

• Comparing ranking functions

• Create audit fields with STRUCTS

• Using SELECT * with EXCEPT and REPLACE

• De-duplication with ROW_NUMBER vs ARRAY_AGG

• Does order of expressions in the WHERE clause matter?

• Filling up missing values with LAST_VALUE

• Quantified LIKE operator in BigQuery

• Value Tables, SELECT AS VALUE and SELECT AS STRUCT

Python

• What are magic functions in Python and why do we need them

• Comprehensions in Python

Not really about tech

Here are some things that are not at all related to computers but which I found interesting.

• Flywheel energy storage - an energy storage principle powering kids’ friction toys and many other things

• A mild controversy over one of my favorite drinks and what are tea blends

• Why the `knot` unit is still around and how it is useful

• The hoop snake - a legendary creature from English-speaking folklore

• One word origin, different stories: Behemoth , Bahamut and hippos

Hello data fans and internet folks in general! I'm really excited to welcome you to the very first post of my newsletter-blog Frankenstein. My name is Constantin Lungu, and I've spent the last ten years working with data: as a Data Analyst, Business Intelligence Developer and Data Engineer. You can think of this page as the next chapter in my data story, and I'm glad you're here to be a part of it!

Growing in Public as a Data Creator - I

Does “building in public” apply to content creation? Or shall I better call it “growing in public” ? In this recurring series, I will be documenting my journey with writing about tech and personal branding.

Let’s start with a little backstory. I’ve always enjoyed writing on topics I care about. Given my line of work, I engage in technical storytelling as well (yes, I sometimes write documentation at work as well).

Origins

In the distant 2016 I was learning Data Analysis with Python while working as a BI Developer. At this stage, I’m just trying just to save the Jupyter notebooks, so I remember how I did this or that.

Fast forward to 2020. Plenty of time during COVID and I make a couple of posts on Medium, sharing stories about my tinkering with Raspberry Pi and how I studied for an AWS Certification. A story like that is a compilation of issues encountered during a long project, so writing an article takes literal weeks. Sharing them to LinkedIn to very little engagement.

Let’s skip another 2 years or so to 2022, when I do another round of posting, this time around practical tips and tricks tied to problems I encounter while working day-to-day with BigQuery. The turnaround is still very slow - preparing an article is tedious and takes days. I submit them to a number of publications in Medium and get a couple of hundred views a month. Again, sharing them to LinkedIn to a dozen or so reactions.

Early 2023, I continue posting a couple of practical BigQuery walkthroughs. I enjoy writing them - straightforward, practical, easy to write. I also find them useful to myself - explaining something in writing helps you understand it better, iron out any issues, get some interesting feedback about something that could’ve been done in a different way. I decide to set up datawise.dev - my blog, hosted on Hashnode. I’m now cross-posting on Medium (with a canonical links).

Things started getting more interesting in August 2023. At this point I have maybe 1800 LinkedIn followers, but 80% of them are recruiters, so engagement on technical content I post is not great.

I was already following some big creators in the Data world, but I discovered the Creator Mode on LinkedIn and started paying attention to audience as well (before, I used to think only in terms of engagement - comments, reactions, re-posts).

One Friday in August, before leaving for the weekend, I wrote a short post about the ROLLUP command in BigQuery, which I found interesting since I never had to use it.

I was very surprised to see that it received ~270 reactions, easily more than the sum of all the reactions to my previous posts. Here’s the post.

At this moment, I start to get a little bit more interested about how LinkedIn works in general - I learn that the time you post matters, outbound links get posts penalized, engagement in comments matters, nice images boost distribution and so on.

I decided to continue writing short, condensed, reproducible examples inspired by my Data Engineering practice, along the following lines:

• problem scenario → possible solution

• new or less known feature → use cases

• comparison of alternatives → pros & cons and when to use which

It’s October 2023 and I decide to set up this domain and a Substack presence, although my idea about the format is not very precise yet 😊.

Current situation

I strive to write 3-5 times per week, and I have been consistently doing so for the last 10 weeks. There are some promising numbers around content performance and the Medium numbers have never been higher. I’ve also managed to surpass 3k followers on LinkedIn, most of them new followers based on my posts.

There is of course a great deal of variability in the content performance, due to an array of factors I’m analyzing at the moment - stay tuned for more here. Here’s a screenshot of the correlation matrix I’ve asked my data analyst - ChatGPT - to build with the data I’ve collected so far on my LinkedIn posts.

What’s next

It’s of course a little hard to see the future, but I’m eyeing the following next steps:

• refine content: more value in a concise format that I would like to experiment with

• analytics: find out what works best and use it to grow

• more polish: a little bit of marketing magic applied to the content

• diversification: write about more diverse topics that just BigQuery

• personal projects: engage with new technologies and explore them for myself and my audience

  ---

Things I’ve followed 👉

• One of the hottest debates in tech (and on LinkedIn) this month was around Indian top executive statement that youth should be looking at 70-hour work weeks for their country’s economic advancement. I’ve also read about 9-9-6 work schedule and the Protestant Work Ethic while at it.

• OpenAI’s ChatGPT new release adds support for DALL-E3 , so you can now generate interesting right there. I think I’m going to write a post-apocalyptic retro-futuristic novel with illustrations like the below

Content Recap

Here’s an overview of my articles released in October on my blog - Datawise.

SQL 📰

• Swapping partitions for faster and more seamless data processing

• Decomposing big SQL problems into smaller ones

• Using a linter with SQL

• Inner / Cross Join and UNNEST

• Use case for STRUCTs together with window functions

• Using dynamic SQL in BigQuery

• Hashing functions in BigQuery: why, where, how

• Moving data Google Sheets → BigQuery: a short walk-through

• Cheaper & faster data exploration in BigQuery

• What to pick between CTE, Views and Temp Tables

• Set operations: Intersect and BigQuery

Python

• What are context managers and why they are important

Not really about data

Here are things not really related to data or tech, but that I enjoyed reading about in the last month:

• Silbo Gomero - a whistling language spoken (whistled) on La Gomera island of the Canaries

• Soundex - a phonetic way of indexing names by their sounding in English

• Soft serve - didn’t know this type of ice cream has its own name

The buzz is real! Excel now integrates with Python, and it's making waves.

I've started my Analytics path with Excel. I plan trips and manage my finances with it. At work, to this day, I'm falling back to it when other tools fail, so I welcome this update with open arms.

Here's why:

- This introduces a vast array of capabilities to Excel users. Think specialized data processing, visualization, machine learning, automation, and the list goes on.

- This move could revolutionize Data Analytics and potentially even Software Engineering by unveiling new horizons for individuals doing this kind of work already, but not necessarily explicitly having it in their role titles.
Imagine, even if a fraction of the Excel power users harness this, we’re ushering millions into the Python community. Exciting times ahead!

- Perhaps, this could be the curtain call for VBA scripting, which, let's face it, was never a favorite of mine.

But, a word of caution is in order:

- As the famous saying goes, "With great power comes great responsibility." Organizations and educators must spearhead best practices, especially concerning security, maintainability, and validation. Mistakes in spreadsheets have caused chaos before; let's double down on checks and balances.

- There might be an inclination to bypass conventional Software Engineering/Data Analytics channels, thinking, "We can manage in Excel now." But let's be judicious in our tool selection. We shouldn't revert to those dark days of endless tabs driven by opaque macros (with shady VBA, written a decade ago, that no one knows how it works anymore) - the ones where you're advised, "Just click this, and it works."

Python to the people!

Ever wondered how to deepen your understanding of programming and bridge the gap between theoretical knowledge and real-world application? Consider starting a pet project!

Story time

Several years ago I was working in a Business Intelligence specialist role, transitioning from previously working with Excel only as a freshly minted graduate with a Finance Bachelor degree. First, it was building dashboards in Domo, a cloud BI tool with some simple SQL-based transformations. Then came a long trial-and-error and learning-by-doing (at work) journey to learn SQL.

I then understood that to progress I'd need to get myself familiar with a programming language. I chose Python and did so by following a Coursera course named 'Python for Everybody' (University of Michigan/Charles Severance). Data types, variables, control flow - I explored the language bit by bit and tried to code all the assignments.

Next, by the time I was getting a grasp of the basic things, I knew I'd like to do more - but there was no way I could get a job in data with my skill set. So I've decided to work on pet projects.

I was interested in Python and Web Development, so I decided to work on Content Management System (CMS), simply put, a blogging platform. My choice naturally fell on Flask - a lightweight and very popular Python web framework. I spent countless hours trying to get it off the ground, studying how other projects work, what are the best practices in the industry and simply trying to introduce a new feature.

While the project is nothing special in itself, akin to thousands of other hobby projects, here are just a couple of things I've learned about - most of them from scratch:

• git version control

• what is Docker and why do we need it, as well as orchestrating with Docker-Compose

• working with ORM (SqlAlchemy)

• working with Celery (task queue) and Redis

• code linting with Pylint

• jinja templating (which Flask uses) - I use this to this day (in Dataform/dbt)

• unit testing with pytest

• integration testing with a headless browser on Selenium (Splinter)

• basic HTML, CSS (Bootstrap) and Javascript

• a simple CI that would run unit and integration tests on each commit

Now, I might have barely scratched the surface of the above, and some of them might have been taught in a Computer Science Undergraduate program, but for someone with no formal programming acumen, even knowing what these things are and when should we use them was an absolute win. I felt that it gave me a basic understanding of very important things in every developer's toolbox.

🔑 Here are six reasons why you should start a pet project:

1️⃣ Learning by Doing Boosts Problem-Solving Skills: You learn about concepts when you encounter genuine problems and find solutions, reinforcing your learning. You learn how to debug, simulate, emit hypotheses and prove them right or wrong.

2️⃣ Explore Your Passion: The pet project allows you to work on something YOU care about. It's more motivating to solve challenges when you're invested in the outcome. After all, you can build WHATEVER you want.

3️⃣ Expand Your Tech Stack: Waiting to get the chance to work with some technology you are interested in can take forever, or it can never happen. With you in the driver's seat, you can explore on your own and dive into unfamiliar technologies and tools.

4️⃣ Portfolio Building: A finished project shows potential employers tangible proof of your skills, but also consistency, determination and technical curiosity.

5️⃣ Feedback and Iteration: Sharing your project with friends or the developer community can provide invaluable feedback. This iterative process of improving based on feedback is the essence of software development.

6️⃣ Ownership and Responsibility: Owning a project from start to finish gives a sense of responsibility. This can mirror the real-world experience of leading a project at a job.

In essence, while books and tutorials offer foundational knowledge, diving headfirst into a pet project can supercharge your learning journey. Whatever your dream project is – an app, website, or game – dive in!

Remember: even if your creation isn't the next tech sensation, the skills you gain and the confidence you build is an unparalleled investment in yourself. Dive in, and let your creativity flow! 🌊