Posts on Sam's Blog

A Review of Hex

Sat, 04 Apr 2026 00:00:00 +0000

Hex Review

It’s Easter Sunday. After a long day of chocolate mogging and stealing eggs from small children, I’ve decided to check out Hex which calls itself the “The AI Analytics Platform for your whole team”. I’m going to toss a small csv into it, if it does well I might hit it with a 20GB Parquet file.

I’m not going to connect a data warehouse to it (This is wrong, I ended up connecting to MotherDuck), I’m going to treat this as the classic “There’s a dashboard, but I need to download the underlying data and re-manipulate”

I signed up for a free trial and started off with a prompt and some data.

I have not read a single guide, looked at any template or read any docs
I have no idea how Hex works on the backend
I’m just going to figure it out as I go and bumble my way through

The data

I gave it a ~740 line csv with historical March Madness data

Example:

Season	Round	Team	Odds	Result
2026	1	TCU	120	1
2026	1	Ohio State	-142	0
2026	1	Troy	650	0
2026	1	Nebraska	-1000	1

The Prompt

I’ve added the prompt below but the TLDR is we want to evaluate the “accuracy” of pre-game odds for March Madness from a CSV containing 5 years of historical data. The prompt isn’t great, and I didn’t even provide extra contents on what I was working on.

Click to view the entire prompt word for word

I have captured historical odds on NCAA march madness tournaments. The CSV contains data with the year, round, team, odds (in american odds) and result (1 for win 0 for loss). The data is not perfectly clean, some of the odds formatting may be wrong or missing, not all years have the correct number of games.

I’d like to visualize the accuracy of these predictions on a discrete vent over the years have been. Use something like brier scores and/or log loss to do this. If you have a better way feel free to do that as well. Split this by round as well as there are many blowouts or “easy” to call games in the beginning of the tournament.

I’d also like to measure the competitiveness of these games over the years. Have the odds gotten closer together on avg over the years, how is this changes if we look by round?

Dropping in

After issuing the prompt and attaching the csv, I immediately found myself in a “project” or, more descriptively, a “notebook style environment”. I’m prompted to give my project a name and watch as some LLM gets to work on the right side of my screen. I can give the project a status and category, which is nice as I could see these things getting unwieldy pretty fast if you had 20-30 people doing “analysis”.

The first cell

There’s a collapsible cell with a SELECT * of my data displayed as a table. The column structure is there, and I can actually create a new column with a formula that can reference existing columns if I want. If you’ve read my blog you’ll know I don’t like SELECT *, but this doesn’t seem to be auto-materialized for all new projects and this is a csv so no harm no foul.

I can re-write the SQL in the cell to produce a new data-frame, but I can also manually filter it via the UI. Applying the filter via the UI updates the “relation” (I don’t really know what to call this yet), and it also looks like it updates visualizations/cells that depend on the relation (The LLM seems has been hacking away below).

Importantly, comments can be added, responded, reacted, and resolved in a cell. Maybe more importantly? Cells can be triggered in DAG fashion with downstream cells taking dependencies. This is good. It’s kind of feeling like Marimo which was recently acquired by CoreWeave. Although Hex is not open source, and you don’t really own your notebooks they live in cloud land… this is actually wrong, you can import/export notebooks as .ipynb or their Hex yaml situation.

Speaking of DAGs there’s this graph view which is nice. the graph shows me that Hex has decided to SELECT * from my csv, written some pandas transformations, and then created three charts. I can’t figure out how to pop the graph view out (Full screen) or change the size… thus the blurry image. In general, every square inch of my screen is packed with features.

The LLM & Context

Moving over to the right pane there’s a chat window with an LLM. The chat window can host multiple threads with conversations >90 days being deleted. Unlike the graph view, I can make this pane basically eat the entire screen.

I quickly find out who’s baking this cookie (claude), but it won’t share system prompts/instructions. After a bit of back and forth, Claude tells me I should be using Guides which exists inside “context studio”, which seems to be exactly how context is managed in Hex. This made me a little nervous at first as it looked as though you’d need to embed/maintain context in Hex, but they allow connections from outside sources (i.e. context in github updated via github actions) If relevant context is haphazardly stored in a bunch of people’s brains this is still of little use though.

But wtf is the compute here

So now I’m curious how this SQL is getting executed. I ask in my little Claude thread and the weights and biases tell me “it’s querying the uploaded OddsMM.csv file directly using the embedded DuckDB engine”. DuckDB… tight. I can confirm this going into the SQL Cells Introduction. It also seems that you can parameterize SQL using Jinja which is a huge win in my book. It also looks like it integrates well with dbt Cloud which is solid. DBT or something like it is pretty simply how modern data teams should be authoring transformations, there’s just not much of a question at this point. I wouldn’t want to be orchestrating transforms inside of Hex though. I’d be doing this in a data warehouse, orchestrating with Dagster and managing transformations with DBT and only after that connecting to Hex.

Opening the “Environment” on the left tool box shows I’m using a Medium instance with 8GB of memory and 4CPUs. They have a beefy 128GB instance, but I’m getting the feeling that if you need this inside of Hex, you may also need a better data model (Yes, dude who’s visualizing some ridiculous telematics data, feel free to rip into me for that comment). I’ve also got access to python 3.9 -> 3.12 and a bunch of python packages come pre-installed (no polars… which is big sad), but it seems you can just add to your venv with uv.

Other staff in the left most toolbox

There’s a full revision history with version checkpoints and a large space dedicated to a search feature which seems to be able to grep through your current workspace. The variables section stores all the built in variables as well as anything we create in our python code and dataframes. There’s also a data browser which acts as a functional data catalogue for assets inside the workspace. Opening up the column types inferred from my csv: Team & Odds are labelled as having type “object”… which is a bit weird, but I guess they are strings… so arrays… so… objects? Idk, I’m stretching here for why that doesn’t say “varchar()”.

Back in the middle

Coming back to center stage with our data/assets, there’s a nice little chart telling us what is taking the longest inside of our “DAG”.

Cool feature to have rather than setting up timing, %%time on some sprawling notebook.

Continuing on, Claude has written a little pandas to create some new data-frames based on the aggregations I requested. These data-frames are available for view in the data browser. These have become materialized relations (I’m pretty sure) that I can query with SQL in a new cell and are also referenced in the visualizations.

Visualizations

Speaking of visualizations, Hex seems to be using Vega after they performed some sort of acqui-hire of the VegaFusion maintainer. The visualizations seem fine and responsive. There’s a suite of UI tools to edit them a bit (both style and substance). You get access to the generated SQL, which seems to build the relation required for each one.

From the screenshot you can see accuracy_by_season_round is referenced which was created by some pandas in the cell above and exposed in the data browser. I’d like to have full control of my charts as code… and it seems like I can easily do that. I created a cell that references that same data-frame, but creates a basic chart with plotly and it works.

Make an… App?

So it seems like the main other thing you can do is switch out of notebook mode and make an “app”. I turned my analysis into an app (If you read this a while from now this will most likely not be available… sorry not sorry) which seems to expose the visualizations at an endpoint, and they can refresh on a schedule. You can export it, take snapshots, turn it into a presentation. So the apps are basically interactive dashboards that you might expose to an end user rather than tossing them a notebook and saying “run this and look at cell 8”.

These apps can be added to a “library” that contains published knowledge that can be “endorsed”. I don’t know how this solves for people just endorsing a bunch of stuff. I don’t think there’s a re-endorsement system where you have to be like “yeah this is still good”… because all of this manual tagging typically loses meaning. It’s like people flagging workloads as “business critical” to get access to resources faster… it’s a short time before everything is “business critical”. Some kind of continuous mechanism to confirm endorsement might be nice (Or just annoying enough that people would disregard it… its a hard problem admittedly).

Competitors

It looks like DeepNote offers something similar, but I’ve never used it. It seems like every data warehouse in the sky has something like this. Posthog’s platform is similar but more focused on web analytics. ClickHouse’s cloud dw offers a SQL notebook and dashboard experience. Marimo is a great notebook based analytics tool that’s open source (recently CoreWeave is in the picture).

This is in NO way an exhaustive list of the space, but I have to say… Hex is very complete. They are clearly thinking hard about how to nail “last mile” analytics and make integration upstream simple. If I was sitting in a small shop and we needed analysts/data science to produce some standard reports, version them over time, and sprinkle LLMs into the workflow I’m probably reaching for Hex.

I’m seriously trying to figure out the value proposition of something like Looker, PowerBI, Tableau, etc… as opposed to Hex.

People either play around with the filters a bit or they actually just want the underlying data for the dashboard… which Hex quickly provides access to and lineage for.

Increasing the data size

I had some larger parquet files from my Prediction Markets article that I thought it’s be a good test to see if Hex could quickly reproduce a few of the charts. They are big enough that I can’t just drop them into the UI.

The Kalshi Markets dataset is ~2 GB & 30 million rows with data on all historical markets
The Kalshi Trades dataset is ~11 GB & 203 million rows with data on every single trade to ever occur on Kalshi

I loaded my local copies to MotherDuck by establishing a connection to a new database I set up.

Then it took about 30 seconds to give Hex a Read/Write access token, and I could start writing SQL against my tables.

I gave it a pitiful prompt/context and in under 5 minutes it produced some charts that you can view here. Again, to do this well with the data in under 5 minutes with limited context is quite impressive. You can directly compare these to a few charts in the actual article… and they get the point across, I’m sure that with a bit of effort I could get these one’s inside Hex to look quite nice.

The Prompt

Please use the Kalshi data assets (markets & trades) to derive a series of visualizations.

Show me weekly trade volume in a bar chart
Show me contracts per trade month over month avg and median
Show me a stacked bar chart displaying total trade volume by month with the bars segmented into the implied probability of each trade in cohorts of 0-20%, 20-40%, 40-60%, 60-80%, and 80-100%

Overall Thoughts

I’m impressed. It’s a well thought out product. It’s definitely much more than just a “Claude wrapper”, and they are clearly trying to integrate things that modern data teams need. For a quick proof of concept on a personal project I could easily see myself tossing some data in here with some prompting/context as long as I had access to the elevated features of a “Team Plan”. I do think this is geared more toward enterprise, as it’s way overkill for a hobby project, and I can pretty much get all the features I’d want out of this with Claude Code, Marimo, and a little know-how. But they know this and can cater to the “Yeah we don’t want to maintain that but want that functionality” crowd.

I’m always a little weary of notebook development for large teams. How do you stop people from building data assets in parallel that describe the “same” thing but end up with a different result. This is a perennial problem that Hex seems to be great at reducing in comparison to people hand waiving Jupyter notebooks and pivot tables. And holy FUCK can we please realize that there are better things out their than just isolated local notebooks performing data wrangling, visualization, and presentation… they do NOT scale… use Hex.

I’m trying to think of other headwinds here. I mean, this is a lot of product. I’m familiar with these concepts, but if the ideal state is to drop this in front of 20 PMs and say “Hey you can make your own dashboards and data assets now” there would be a significant amount of churn/up-skilling where as Data Science could probably get on board with this very quickly. You still need to solve for upstream, Hex isn’t going to fix your broken medallion architecture and redundant/stale tables. Notebooks are a classic foot gun, if people don’t use the tools that Hex provides (DBT integration, jinja templating, version control, data catalogue, etc…) you could easily run into spaghetti mode just as anything else. Hex can lead your team to the data lake, but they can’t make you…

SQL Buckets

Sun, 05 Oct 2025 00:00:00 +0000

Case Statements:

A common occurrence when writing SQL is needing to bucket some data and look at how it is distributed across said buckets.

Typically I’d write (have an LLM spit out buckets) something like this:

SELECT CASE WHEN COLUMN_TO_BUCKET < 5 THEN 'a. < 5'
            WHEN COLUMN_TO_BUCKET < 10 THEN 'b. >=5 - <10'
            WHEN COLUMN_TO_BUCKET < 15 THEN 'c. >=10 - <15'
            ETC...
            ELSE 'u. >100' END AS COLUMN_TO_BUCKET_DIST
      , SUM(THIS_IS_ANNOYING) AS ANNOYED
FROM TABLE
GROUP BY 1

I have to add a character/number in each bucket to allow for sorting, making writing/updating the lengthy case statement that much more annoying. I have to write all this because unlike other OLAP DBMS, Redshift doesn’t have a bucketing function that I’m aware of.

Or do I…

Solution:

Let’s say I have a table that records when a record was created and when it was last updated. I’ve never used this table before, and I’m interested in the distribution of the number of days between created and updated to see how common it is for a record to have an update n days after it was created.

CREATE TEMP TABLE #DELTA_DIST AS (
SELECT A_UNIQUE_ID_OF_A_ROW
     , DATE_DIFF('DAY', CREATION_DATE, LAST_UPDATED) AS CD_LD
FROM MY_TABLE
WHERE A_FILTER >= '2025-01-01'::DATE
  AND ANOTHER_FILTER = 1
);

Cool, so we want to know how these records look like in some buckets!

We could do this:

SELECT CASE WHEN CD_LD < 5 THEN 'a. < 5 Day'
            WHEN CD_LD < 10 THEN 'b. 5-10 Day'
            WHEN CD_LD < 15 THEN 'c. 10-15 Day'
            WHEN CD_LD < 20 THEN 'd. 15-20 Day'
            WHEN CD_LD < 25 THEN 'e. 20-25 Day'
            WHEN CD_LD < 30 THEN 'f. 25-30 Day'
            WHEN CD_LD < 35 THEN 'g. 30-35 Day'
            WHEN CD_LD < 40 THEN 'h. 35-40 Day'
            WHEN CD_LD < 45 THEN 'i. 40-45 Day'
            WHEN CD_LD < 50 THEN 'j. 45-50 Day'
            WHEN CD_LD < 55 THEN 'k. 50-55 Day'
            WHEN CD_LD < 60 THEN 'l. 55-60 Day'
            WHEN CD_LD < 65 THEN 'm. 60-65 Day'
            WHEN CD_LD < 70 THEN 'n. 65-70 Day'
            WHEN CD_LD < 75 THEN 'o. 70-75 Day'
            WHEN CD_LD < 80 THEN 'p. 75-80 Day'
            WHEN CD_LD < 85 THEN 'q. 80-85 Day'
            WHEN CD_LD < 90 THEN 'r. 85-90 Day'
            WHEN CD_LD < 95 THEN 's. 90-95 Day'
            WHEN CD_LD < 100 THEN 't. 95-100 Day'
            ELSE 'u. >100' END AS CD_LD_DIST
, COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

That is pretty annoying… it is verbose and, if you’ve done this before, you’ll know that catch all bucket at the end will have some spike in records in it that you’ll want to further investigate. So I came up with the following:

SELECT LPAD(FLOOR(COLUMN_TO_BUCKET / 5::FLOAT)::TEXT, 2, '0') -- Creates the sortable digits at the beginning
            + '. ' +
       FLOOR(COLUMN_TO_BUCKET / 5::FLOAT) * 5 -- Generates the first day in the range
            + '-' +
       (FLOOR(COLUMN_TO_BUCKET / 5::FLOAT) * 5 + 4) -- Generates the last day in the range  (+4 as I'm bucketing by 5 days at a time)
            +
       ' Days' AS COLUMN_TO_BUCKET_DIST -- Adds the Days string at the end for clarity
     , COUNT(DISTINCT THING_TO_AGGREGATE) AS AGGREGATED_THING
FROM TABLE_I_WANT_TO_DISTRIBUTE
GROUP BY 1

We can pass in the column to bucket, adjust the bucket size, and also handle the number of buckets. This does use LPAD but FLOOR() is standard. Regardless, lets break it down in detail if you need to rewrite for a different SQL dialect.

FLOOR: rounds a number down to the next whole number
LPAD: prepends characters to a string

On an input of “1” this is what the flow would look like to go from 1 -> 00. 0-4 Days

-- FLOOR of 1/5 = 0 and cast to text
SELECT FLOOR(1 / 5::FLOAT)::TEXT -> '0'
-- LPAD adds a leading zero to single digits
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') -> '00'
-- Just cosmetic, appending a "."
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. '  -> '00.'
-- Now we know that this will iteratively return 0,1,2,3,n... as our input increases
-- And if we multiple by 5 we get the beginning of our bucket
       -- 0 * 5 = 0 (for 0-4)
       -- 1 * 5 = 5 (for 5-9)
       -- etc...
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 -> '00. 0'
-- Now we add another cosmetic piece ( a dash to separate start/end of bucket)
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 + '-' -> '00. 0-'
-- Remember how we generated the start of the bucket 
-- If we know the bucket is going to span 5 days (inclusive of start and end) we just add 4 to the start
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 + '-' + (FLOOR(1 / 5::FLOAT) * 5 + 4) -> '00. 0-4'
--- Add final cosmetic piece ' Days'
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 + '-' + (FLOOR(1 / 5::FLOAT) * 5 + 4) + ' Days' -> '00. 0-4 Days'

We’ve transformed from an input of CD_LD = 1 and assigned it a sortable, bucketable (probably not a word) value -> ‘00. 0-4 Days’

I’m worried about an unconstrained series of buckets… how do I cap it?

SELECT  -- Define our buckets when we are below our threshold
       CASE WHEN CD_LD < 100 THEN LPAD(FLOOR(CD_LD / 5::FLOAT)::TEXT, 2, '0')
            || '. ' ||
       FLOOR(CD_LD / 5::FLOAT) * 5
            || '-' ||
       (FLOOR(CD_LD / 5::FLOAT) * 5 + 4)
            ||
       ' Days'
       
       -- Define final bucket when threshold exceeded
       ELSE LPAD(FLOOR(100 / 5)::TEXT, 2, '0')
          || '. >100' END AS CD_LD_DIST
     , COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

I want to change the increment size

-- Change increment size from 5 days to 7 days
-- 5 -> 7
-- 4 -> 6
SELECT LPAD(FLOOR(CD_LD / 7::FLOAT)::TEXT, 2, '0')
            || '. ' ||
       FLOOR(CD_LD / 7::FLOAT) * 7 --
            || '-' ||
       (FLOOR(CD_LD / 7::FLOAT) * 7 + 6) -- Generates the last day in the range  (+6 as I'm bucketing by 7 days at a time)
            ||
       ' Days' AS CD_LD_DIST -- Adds the Days string at the end for clarity
      , COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

I have more than 99 buckets, the two digit solution wont work:

-- Pad an additional 0 with LPAD
-- 
SELECT LPAD(FLOOR(CD_LD / 5::FLOAT)::TEXT, 3, '0') -- 2 -> 3
            || '. ' ||
       FLOOR(CD_LD / 5::FLOAT) * 5 
            || '-' ||
       (FLOOR(CD_LD / 5::FLOAT) * 5 + 4) 
            ||
       ' Days' AS CD_LD_DIST -- Adds the Days string at the end for clarity
      , COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

Hope that’s helpful! Maybe there’s a better solution, I didn’t do an exhaustive search!

Your Data Warehouse Ignores Your Primary Key

Thu, 19 Jun 2025 00:00:00 +0000

The agenda…

Today we’re digging into the curious case of “Primary Keys” in modern data warehouses (think Redshift, BigQuery, Snowflake). On the menu:

A high-stakes request from our VP
What happens when a PRIMARY KEY doesn’t do what you think it does
A hands-on experiment in Postgres to see why OLAP systems choose not to enforce a PK
A dash of data quality, some opinions, and a tangent or two

A little background

Lets take Redshift for example. In Redshift, primary keys are informational only; they are not enforced. However, the query optimizer will trust our declaration and use it to create more efficient execution plans. How silly of Redshift to trust us like that.

This is not shocking to the seasoned data warehouse connoisseur, but to the uninitiated this may look unnerving. So for anyone unfamiliar here’s what I mean.

Our VP needs to know

Our VP needs to know if Bob is hungry or not to understand whether our start up is doing good or bad.

Yikes, we deleted all our data when the U.S. Space Command was investigating us for fraud! We can’t tell our VP that we don’t know anything about Bob so lets insert some random records into our datawarehouse. We’ll need to start from scratch!

First lets make a new table and define a primary key.

-- REMEMBER! When Storing strings make sure to take up as much space as possible 
-- Defining these columns as CHAR pads strings with blanks to ensure we waste as much space as we can 
-- Everyone will love you for this, and you'll probably get promoted!
CREATE TABLE CHECK123 (
SOME_ID CHAR(4096) PRIMARY KEY, <-- Ahh, a nice PK
AN_ATTRIBUTE CHAR(4096),
ANOTHER_ATTRIBUTE CHAR(4096),
SOME_NUMBER INT4
);

Alright lets manually insert our correct data!

INSERT INTO CHECK123
VALUES ('abc123', 'Bob', 'Hungry', 2) <-- Hmmm...
     , ('wow', 'Big Tom', 'Fast', 86)
     , ('this', 'Dale', 'Thoughtful', 12)
     , ('is', 'Doug', 'Tired', 9)
     , ('pretty', 'Hammer time', 'Happy', 6)
     , ('annoying', 'The bug unit', 'Angry', 6)
     , ('abc123', 'Whopper', 'Sad', 6) <-- Hmmm...

And run our important query that our VP needs. We don’t have time for data quality checks so lets throw it into Excel and send it everywhere.

SELECT SOME_ID
     , COUNT(1)
FROM CHECK123
GROUP BY SOME_ID;

abc123,2 -- UH OH!!!!
wow,1
this,1
is,1
pretty,1
annoying,1

SELECT DISTINCT SOME_ID FROM CHECK123;

abc123 -- UH OH!!!!
wow
this
is
pretty
annoying
abc123  -- UH OH!!!!

Oh no… we have duplicate records even after writing SELECT DISTINCT. But I thought SELECT DISTINCT would remove duplicates? Well, yes, but… we declared SOME_ID as a PRIMARY KEY, effectively telling Redshift, “I solemnly swear this column contains no duplicates.” The query optimizer, trusting us completely, chooses the most efficient execution plan: it just scans the column and hands it back without performing a costly deduplication step. Why would it? We already promised it was unique.

Imagine someone gives you a book with 1,000,000 unsorted names in it and tells you they are unique. Then they say, “cross out any duplicate names and hand the book back to me”. You’re initial reaction is to immediately throw that book back as checking for uniqueness of one million names sounds… well… awful.

In this case, the query optimizer is just handing us the book back. If we were to run:

EXPLAIN SELECT DISTINCT SOME_ID FROM CHECK123

We’d get something back like:

XN SEQ SCAN on check123 (cost=0.00..0.07 rows=7 width-304)

That’s just a SEQ SCAN handing us back the entire column because we’ve already told Redshift these values are unique. In contrast, what if we scan a non-PK column for uniqueness:

SELECT DISTINCT AN_ATTRIBUTE FROM CHECK123;

And and there you have it, now we get:

XN Unique  (cost=0.00..0.09 rows=7 width-304)
    -> XN SEQ SCAN on check123 (cost=0.00..0.07 rows=7 width-304)

You’ll find XN Unique in query plans when DISTINCT or UNION (which implies distinctness, unlike UNION ALL) are used. Our DBMS now returns a unique result set from values it does not know are unique. I go on a tangent about this below if you’re interested.

Click to view a tangent on XN Unique & XN HashAggregate

If we have an index on the column deduplication can be fast, I’ll link the Postgres docs on B-TREE Indexes specifically the section on deduplication. But what if we don’t have an index… Redshift doesn’t even have indexes… Redshift just has metadata (min-max values) for blocks written to disk called Zone Maps.

Redshift docs notes that XN Unique: “Removes duplicates for SELECT DISTINCT queries and UNION queries.” This gets weird because I’d assume with this operator that we are sorting and plucking unique values from this column. However, there is no XN Sort in that plan. XN Sort:“Evaluates the ORDER BY clause and other sort operations, such as sorts required by SELECT DISTINCT queries. Our SELECT DISTINCT query above doesn’t have an XN Sort it only contains an XN Unique, maybe XN Unique implies an XN Sort? I generated an explain plan for a relation whose rows were 100% unsorted according to STV_TBL_PER and that plan did NOT include an XN Sort either…

I think a couple things could be happening:

The optimizer can choose between different strategies to perform our SELECT DISTINCT and in these cases it’s actually choosing a hashing strategy but just labeling it as XN Unique.
XN Unique implies a sorting operation, and we’re just not being shown that in the plan

For example if we take:

SELECT DISTINCT AN_ATTRIBUTE FROM CHECK123;

And re-write it as:

SELECT AN_ATTRIBUTE FROM CHECK123 GROUP BY AN_ATTRIBUTE;

We get:

XN HashAggregate  (cost=0.09..0.09 rows=7 width-304)
    -> XN SEQ SCAN on check123 (cost=0.00..0.07 rows=7 width-304)

This is similar to XN Unique but the costs are different cost=0.00..0.09 & cost=0.09..0.09 respectively. The first number is the relative cost of returning the first row for this operation. The second value, in this case 0.09, provides the relative cost of completing the operation. So I’d surmise that depending on how we write this query:

XN HashAggregate is a blocking operator designed for GROUP BY clauses; it must consume its entire input to correctly compute aggregate functions like COUNT() before returning any rows.
XN Unique is a specialized operator for SELECT DISTINCT that can implement a streaming (non-blocking) hash algorithm, allowing it to output unique rows as soon as they are encountered.

This seems to explain why Unique can have a zero startup cost while HashAggregate’s startup cost reflects processing its entire input.

On another note, I’m only able to get an XN Sort operator via explicitly using ORDER BY:

SELECT DISTINCT AN_ATTRIBUTE FROM CHECK123 ORDER BY AN_ATTRIBUTE;

XN Merge  (cost=1000000000000.19..1000000000000.20 rows=7 width=304)               
  Merge Key: an_attribute                                                          
  ->  XN Network  (cost=1000000000000.19..1000000000000.20 rows=7 width=304)       
        Send to leader                                                             
        ->  XN Sort  (cost=1000000000000.19..1000000000000.20 rows=7 width=304)    
              Sort Key: an_attribute                                               
              ->  XN Unique  (cost=0.00..0.09 rows=7 width=304)                    
                    ->  XN Seq Scan on check123  (cost=0.00..0.07 rows=7 width=304)

Anyway, our VP is pissed because we sent trash, but why would an OLAP DBMS do something like this?

PK in name only

We can see that our primary key’s exist in name only. The DBMS is not going to enforce them, its going to assume they are right, and if you want to enforce them then you’ll need to write that logic yourself.

So what could be the reasons why thoughtful and well payed, Engineers, TPMs, Directors, and VPs building these cloud data warehouses decided to allow people to designate primary keys that aren’t enforced? I’ll boil the arguments down to the following:

Write performance

What do we mean by write performance, and can we actually take a look at a DBMS and see this impact on performance?

Yes, we can:

spin up Postgres inside of a container
create tables with/without PK
write data to said tables
measure performance
and even look a little deeper at what Postgres is doing

A couple caveats before, so I don’t get dunked on…

Single Node vs. MPP: I’m running Postgres inside a container on my ~~[kindle](Insert sponsored link)~~ Mac, which is completely different than the orchestration/networking effort that is required for a cloud data warehouse with dozens of compute nodes
Row-Oriented vs. Columnar: Postgres is row oriented, an entire row (SOME_ID, AN_ATTRIBUTE, ANOTHER_ATTRIBUTE, SOME_NUMBER) is stored together. Redshift utilizes columnar storage, each column is stored in a separate block(s). This is fantastic for aggregates SELECT SUM(SOME_NUMBER) but makes single-row operations inefficient. If we wanted to update or insert a new row in Redshift while enforcing the primary key we’d have to:

Search for the existence of/find the id of the record we are updating in its block (If it doesn’t exist then insert)
Then find the corresponding attributes in their blocks, so on and so forth for every column
Just to reconstruct one tuple for an update… the opposite of what columnar stores are designed for

This leads into the second point people will make… that this isn’t what the DW is for. You can now see that these inserts are going to be gnarly, why not have some sort of slowly changing dimension that would allow for faster writes and control changes to facts in an OLTP DBMS better suited to handle transactional workloads.

Insert with & w/o a PK

Anyway… lets do our little test:

First we pull the docker image and run a container in detached mode:

docker run --name postgres-test -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres

We can check the status of our new container with docker ps -a then we can run docker exec -it postgres-test psql -U postgres to get a psql shell. We’ll create two tables, one with a PK and one without, and use generate_series() to curate some faux data for us to write to these tables, something like:

SELECT i, 'some payload data for row ' || i PAYLOAD FROM generate_series(1, 10) s(i);

-- Which will produce the following ID + PAYLOAD 
 i  |           PAYLOAD          
----+------------------------------
  1 | some payload data for row 1
  2 | some payload data for row 2
  3 | some payload data for row 3
  4 | some payload data for row 4
  5 | some payload data for row 5
  6 | some payload data for row 6
  7 | some payload data for row 7
  8 | some payload data for row 8
  9 | some payload data for row 9
 10 | some payload data for row 10
(10 rows)

So lets create our two tables, one with a PK one without:

CREATE TABLE no_pk_test (
    id  INTEGER,
    payload TEXT
);

CREATE TABLE with_pk_test (
    id  INTEGER PRIMARY KEY,
    payload TEXT
);

Click to view a deep dive into Postgres internals

Now we have two tables and with_pk_test has a primary key. We can see it has an associated index:

SELECT oid, relname, relkind FROM pg_class WHERE relname IN ('with_pk_test', 'with_pk_test_pkey');

-- Returns:
  oid  |      relname      | relkind 
-------+-------------------+---------
 16393 | with_pk_test      | r <-- Our Relation
 16398 | with_pk_test_pkey | i <-- Our index
(2 rows)

Each table and index is stored in a separate file. We can also see the initial size of this index, Postgres has allocated the first 8kB page for our B tree:

-- 8192 bytes
SELECT pg_size_pretty(pg_relation_size('with_pk_test_pkey')) AS index_size;

We can get the OID of all databases with:

SELECT * FROM pg_database;

-- Returns (truncated Columns):
 oid |  datname  
-----+-----------
   5 | postgres
   1 | template1
   4 | template0
(3 rows)

-- We can see our current database with: 
SELECT oid FROM pg_database WHERE datname = current_database();

-- Returns:
 oid 
-----
   5
(1 row)

Now we can exit psql (\q) and use bash instead: docker exec -it postgres-test bash. We can cd to the base directory and list the directories.

cd /var/lib/postgresql/data/base && ls` # shows 1  4  5

Nice… 1, 4, and 5 match the OID for the databases we listed above, and now if we cd to 5 and ls -lh we’ll find:

-rw------- 1 postgres postgres    0 Jun 23 16:30 16388 <-- OID of our relation no_pk_test
-rw------- 1 postgres postgres    0 Jun 23 16:30 16391
-rw------- 1 postgres postgres 8.0K Jun 23 16:30 16392
-rw------- 1 postgres postgres    0 Jun 23 16:31 16393 <-- OID of our relation with_pk_test
-rw------- 1 postgres postgres    0 Jun 23 16:31 16396
-rw------- 1 postgres postgres 8.0K Jun 23 16:31 16397
-rw------- 1 postgres postgres 8.0K Jun 23 16:31 16398 <-- OID of our index with_pk_test_pkey

Alright, with that deep dive out of the way lets insert 10MM records using generate_series() into these empty relations:

-- INSERT 0 10000000
-- Time: 3846.560 ms (00:03.847)
\timing on
INSERT INTO no_pk_test SELECT i, 'some payload data for row ' || i PAYLOAD FROM generate_series(1, 10000000) s(i);

-- INSERT 0 10000000
-- Time: 6038.888 ms (00:06.039)
INSERT INTO with_pk_test SELECT i, 'some payload data for row ' || i PAYLOAD FROM generate_series(1, 10000000) s(i);

Yep, closing in on twice as long (3846.560 ms vs 6038.888 ms). And just for fun we can check on the size of our tables/index.

table_name	table_size	indexes_size	total_size
“public”.“with_pk_test”	651 MB	214 MB	865 MB
“public”.“no_pk_test”	651 MB	0 bytes	651 MB

And for even more fun we can see this time to insert explode further after adding a secondary Index.

CREATE TABLE multi_index_test (
  id INTEGER PRIMARY KEY,
  name VARCHAR(256) NOT NULL, -- Our new table has a new column (name)
  payload TEXT
);
CREATE INDEX idx_name ON multi_index_test (name); -- Add a secondary B+Tree index

INSERT INTO multi_index_test
SELECT
  i,
  'user_name_' || (i % 100000),
  'some payload data for row ' || i
FROM generate_series(1, 10000000) s(i);
INSERT 0 10000000
Time: 30859.168 ms (00:30.859) <-- *cough*

There is NO free lunch, sure you can make your reads faster, but at what cost?

Are people even using the index?
Whats the ratio of writes to reads on this table anyway?
I wonder how many B+ Trees I can fit in my 2012 Chevy impala LT?
These are the questions we need to answer

Merge Upsert

What happens if we insert additional records into our tables that already have 10,000,000 records in them. We could compare the data warehouse style insert where we insert everything vs a merge upsert where we update records that already exist and insert new records that don’t.

Let’s regenerate IDs spanning 9,000,001 -> 11,000,000 so we get a mix of new and already existing records and perform our two writes.

-- Creating a new table with the records as discussed above 
CREATE TEMP TABLE new_batch AS
SELECT i, 'new payload for ' || i FROM generate_series(9000001, 11000000) s(i);

We can perform our basic insert into no_pk_test:

INSERT INTO no_pk_test
SELECT * FROM new_batch;
INSERT 0 2000000
Time: 598.475 ms 

-- 12MM Records now, 10MM old and 2MM new
SELECT LEFT(payload, 10), COUNT(1) FROM no_pk_test group by 1;
    left    |  count   
------------+----------
 new payloa |  2000000
 some paylo | 10000000
(2 rows)

Now what about our upsert? We can use On Conflict courtesy of Peter Geoghegan, Heikki Linnakangas, Andres Freund and Jeff Janes.

INSERT INTO with_pk_test
SELECT * FROM new_batch
ON CONFLICT (id) DO UPDATE SET
    payload = payload;
INSERT 0 2000000
Time: 3287.132 ms (00:03.287) 

                                QUERY PLAN                                
--------------------------------------------------------------------------
 Insert on with_pk_test  (cost=0.00..33414.40 rows=0 width=0)
   Conflict Resolution: UPDATE
   Conflict Arbiter Indexes: with_pk_test_pkey
   ->  Seq Scan on new_batch  (cost=0.00..33414.40 rows=1869440 width=36)

-- 11MM Records now, 9MM old and 2MM new
    left    |  count  
------------+---------
 new payloa | 2000000
 some paylo | 9000000
(2 rows)

-- We can also use merge, slightly slower 
-- Time: 4593.521 ms (00:04.594)
MERGE INTO with_pk_test pk
USING new_batch n
ON n.id = pk.id
WHEN MATCHED THEN
  UPDATE SET payload = n.payload
WHEN NOT MATCHED THEN
  INSERT (id, payload)
  VALUES (id, payload);
MERGE 2000000
Time: 4593.521 ms (00:04.594)
                                          QUERY PLAN                                          
----------------------------------------------------------------------------------------------
 Merge on with_pk_test pk  (cost=398687.23..523481.91 rows=0 width=0)
   ->  Hash Left Join  (cost=398687.23..523481.91 rows=1869440 width=48)
         Hash Cond: (n.id = pk.id)
         ->  Seq Scan on new_batch n  (cost=0.00..33414.40 rows=1869440 width=42)
         ->  Hash  (cost=207833.88..207833.88 rows=10979388 width=10)
               ->  Seq Scan on with_pk_test pk  (cost=0.00..207833.88 rows=10979388 width=10)

The merge upsert takes much longer (598.475 ms -> 3287.132 ms). These are two different kinds of writes, but its generally indicative of what you might be trying to accomplish in an OLAP system vs OLTP. Hopefully this sheds some light on what people mean when they bring up the “performance” argument.

These constraints should be enforced upstream

This is idea #2.

And this is opening a can of opinions, hot takes and worms. I think in theory maybe this works, but in the real world there are certainly people with relations sitting in their datawarehouse that could benefit from the safety of PK enforcement or, at least, have that option. So on one hand we have quick writes/optimized plans, and on the other hand all those performance gains are a wash because we have to write those deduping checks anyway and people are confused that their PRIMARY KEY has duplicates in it.

Opinion Time! I don’t understand why the syntax in Redshift couldn’t have been changed to fully remove confusion. If you read the documentation its clear, but why call it a PRIMARY KEY when it could have been PRIMARY KEY NOT ENFORCED… a little clarity over brevity. Databricks has the RELY option. Snowflake doesn’t enforce them in their standard table offering but do enforce them for hybrid tables. BigQuery uses NOT ENFORCED.

(1048064189, Have fun out there!), (1048064189, Hope you enjoyed the blog!), (1048064189, Take Care!)

Query Banks

Mon, 31 Mar 2025 00:00:00 +0000

That query bank should be a semantic layer

I’ve done it myself:

“Oh we should just cache all these commonly used queries somewhere central, so we always have a nice definition for someone to use.”

You could even extend the functionality of this “Query Bank” and train an LLM, or perform code reviews, or implement search, etc…

Sweet, now anyone can grab some code and use it! They can paste it here and in this scheduled job there, science can use it and slam it into a notebook, and others can run ad hoc workloads against it.

No, ahh! The BANK IS CLOSED

A query bank misses the entire point of DRY, you end up with a hairball of copy/paste code that has no reference to its origin.

Imagine we took all package management and said you know what instead of depending on our source code you should just copy and paste it into your project and reference it there!

That is what the “Query Bank” is doing because you can't inject the latest definition(query) at run time. If you CAN do this then you don’t have a query bank, you have an actual data model, fleshed out with materialized relations that have some established dependency chain which you can reference in your workflow.

The query bank is just a bandaid for broken process/data models:

No lineage of where code has been copied
No ability to update code across all copies
Limited ability to notify users of changes to upstream code
Most likely repeating ourselves which is extra work for people and DBMS
Will 100% cause discrepancies between “Sources of Truth”
No ability to ensure code is being copy/pasted into the correct context

Creating tables establishes consistency, durability and determinism that’s not possible with a “Query bank”.

The Bank is open

Make a table, document the table, expose the table, manage access, and share the table.

Ask yourself a few questions before you act:

Should I really send this code to someone or should I just DEFINE this once and allow them to read from the definition?
This job, model, notebook whatever you want to call it is already like 1000 lines of SQL should I really pile on more logic to this or maybe its time ot break down into a series of relations?
I’ve seen this Temporary table, CTE, etc… in a bunch of people’s queries should I just paste it into mine or is there something better I can do?

It’s your problem until you fix it or find the owner

Here’s a challenge as an example. If you see a long case statement being shared around. Yes, literally just a case statement without any context… take the initiative and see if you can’t centralize that and store it is a relation for the people needing it.

It’s most likely they don’t know any better and will continue to do this until they have a nasty data quality issue/argument.

Table Creation

Mon, 17 Mar 2025 00:00:00 +0000

Collection of thoughts on table creation.

This is specifically regarding tables for analytics (OLAP queries, Data Warehouse, etc…). This is also specific to a larger organization, I don’t necessary apply these when working locally with duckDB per se.

Get a good understanding of who’s going to use the table, what they need, and when they’ll need that data by. This will help inform your data model and manage dependencies. If you are working directly with another team have them list out all of the requirements and set a date when this list will be final.

When working on a request for our finance partners, I was starting on it and everyday they wanted to add this, change that…. nip that in the bud… have them curate what they want with the expectation that it will lock once you start building it. This will force them to stop, think critically, and figure out exactly what they need. I have only had to make significant updates to this table once in the last 14 months because of this. (I guess I can’t prove causation, but I think it helped)

Don’t try and do too much in a single relation, you may think you are doing people favors by adding columns, nested data structures, and complexity but there’s a delicate balance here. A classic mistake is building extremely wide tables. These tables are for everybody and nobody because each aspect of the table wont be “good enough”. The goal is commonly to try and bring in hundreds of attributes at a specific granularity, but it turns into a dependency and data quality nightmare without being able to satisfy many questions without making comprises that nobody can accept.
Document your stuff. I’m a firm believer that what you build is only as good as your documentation. Don’t continue to contribute to the sea of blank column descriptions and opaque DAGs that your stakeholder/peers will have to waste time figuring out. You will also waste your own time explaining things to your stakeholders over and over. Instead, create a thorough explanatory resource and point people to it.
Publish it to a shared data catalog if possible… I’ve made this mistake countless times of creating local tables, them gaining a solid user base and then having to later share it and manage access across various people/software entities. A few caveats in that I would not do this if you’re still developing the table and making fast iterations… but otherwise it’s going to give you the ability to manage access, control updates, configure dependencies, and cascade relevant notifications so much easier.
Write tests and perform code reviews. We are working on better testing frameworks to role out.

The last thing you want is people/down stream jobs consuming trash that could’ve been blocked from being written to the table in the first place.
Having someone review your code/table is a no-brainer. Writing SQL against the table and using it as it would be used in production will uncover oddities/issues that would otherwise plague consumer.

Stop building on top of bad decisions

Thu, 16 Jan 2025 00:00:00 +0000

One step back, two steps forward:

Stop building on top off bad decisions and fix the bad decisions.

Lets say you work with a bunch of people that are setting weekly goals. Each year goals are appended in a particular format to a table referenced by everyone to present goals throughout the organization.

Long long ago in a galaxy far far away someone decided set the goal date column as type VARCHAR() so now you have goals associated with strings that range from ‘2025-05’ to ‘2025-02-05’ to ‘02/05/2025’. In fact for a single goal, the weeks may be recorded in wildly different formats each year based on whoever uploads them.

THIS IS A BAD PROCESS.

This should be screaming bad process, as you disentangle different week to week formatting changes each year or worse what if someone updates it in the middle of the year and your logic just suddenly breaks… Your best way to test that logic in the first place would have been to establish a proper schema and not use a VARCHAR to store a date, but we’ve absolutely blown the rails off that one.

CREATE TEMP TABLE TEST123 AS (
SELECT '2024-12-05'::DATE AS SHIP_DATE
);

-- ERROR: invalid input syntax for type date: "2024-49"
-- Yes! This is what we want
INSERT INTO TEST123 VALUES
('2024-49');

Stop putting band-aids on and fix it… that’s the only way this is going to get better. End of story. But there’s a lot of important dependencies on this table we’d have to change a lot of logic… its a big risk There’s a much bigger risk in parsing semi-random strings to match dates then making it right.

Change your automatic response from ~~“What do I need to do to make this work”~~ to ~~“How do I fix this and make this better for everyone moving forward”~~. Everyone will benefit and your organization will be more flexible and testable.

Flexible and testable, good qualities to live by.

DRY Analytics

Sun, 24 Nov 2024 00:00:00 +0000

I was talking to one of our partner teams in a developing market. They are a newer analytics department of savvy Business Analysts & SCMs (Supply Chain Managers).

Long story short, they are working on this new metric and I was like, How are you filtering for our specific products in your marketplace? And they were like: Oh we have this case statement that we include in all of our jobs…

SELECT COl1
     , COL2
     , COL3
     , ...
     , CASE WHEN PRODUCT_WEIGHT > 45 THEN 'OUR_PRODUCT'
            WHEN PRODUCT_LENGTH >= 37 AND PRODUCT_WIDTH >= 33 THEN 'OUR_PRODUCT'
            WHEN CATEGORY = 'SOME_CATEGORY' OR EARTH_IS_ROUND_FLAG = 'Y' THEN 'OUR_PRODUCT'
          WHEN COLOR = 'RED' THEN 'OUR_PRODUCT'
          ELSE 'NOT_OUR_PRODUCT' END AS PRODUCT_DEF
FROM BIG_TABLE
WHERE PRDDUCT_DEF = 'OUR_PRODUCT'

Quick aside: When I write Job, Profile, Model, or Asset I’m talking about SQL that creates some result set and is most likely scheduled to run at a specified interval or once all dependencies have satisfied.

I was like ohhhh man can I save you some pain. Maintaining this kind of logic in tens or worse hundreds of jobs is going to inevitably suck! At some point someone’s going to come along and say:

 Hey… color = 'BLUE' products should also be in our list of products.

Now when this day comes and you’ve got a bunch of different teams that all maintain product definitions… someone is going to have to coordinate this migration and get the timing right. Heat flashes across your forehead and your palms get a little sweaty. For so long we’ve been copying and pasting case statements but we’ve officially reached a crescendo of misaligned definitions and numbers not matching up. The CAT is way out of the bag on this one, it does NOT want to go back in the bag and you need it back in the bag if you ever want to make it to the FIFe World Show.

This change will need to be cascaded across all the teams that maintain such a definition

a. Some teams might disagree and say Blue products shouldn’t be included

b. It may be difficult to locate everyone who is defining products so there will be some level of uncertainty that everyone has been notified

c. And still others will ask, since when did we define OUR_PRODUCT with a color dimension????
Somebody won’t have bandwidth to implement the change and will need to push the date back, others may need time to figure out where exactly they are defining product definitions in their pipeline, someone’s OOTO. This will bump around a bunch of teams and there’ll be several calls to organize.
Some teams may have different data models, we only keep the latest definition, we keep a versioned definition over time, we have a denormalized table and qualify row_number() over(partition by product_id order by last_updated_date desc) = 1 every single time to derive the most up to date definition.
Now that we’ve updated the definition do we need to rewrite all historical tables that were derived using the previous definition? Now those blue products are OUR_PRODUCT but at any point time pre-change they would never be OUR_PRODUCT and would not reside in any stored result in any relation. And don’t forget this is happening across multiple teams in parallel and that change will need to be cascaded through several sequential jobs!

Ok so how do we avoid pure pain?

Maintain a centralized definition.

DENORM_PRODUCT_DEFS [Captures all product changes over time] denormalized table appending all changes to products over time. Our products may change color or a vendor my adjust the packaging changing its dimensions this can all impact the classification of a specific product. This table could hold when the records were first inserted, when they were most recently updated, and denotes the active definition. This will save you a bunch of LEAD/LAG stuff if you’re reading from this table a lot.
VERSIONED_PRODUCT_DEFS [Tracks historical versions for auditing and reproducability] versioned table that contains a snapshot of the product definition at a specific point in time. Back in JUNE we used this definition which would have resulted in this classification of these products and now in OCTOBER we use this definition which would have resulted in this classification of these products. This table would nicely be partitioned on version and would also be great to have columns denoting start and end dates for the version.
FACT_PRODUCT_DEFS [Current, streamline view for active use cases] only the most up to date definition for products that is pre-filtered to only include your products.

So now we have profiles that look like:

SELECT B.COl1
     , B.COL2
     , B.COL3
     , B...
FROM BIG_TABLE B
    INNER JOIN FACT_PRODUCT_DEFS F ON B.PRODUCT_ID = F.PRODUCT_ID

And even better, now we can update how this reference is configured and if we have this query written fifty times everything points back to that reference and is updated [ref dynamically resolves table and schema names, so changes to a single reference propagate across all queries]:

SELECT B.COl1
     , B.COL2
     , B.COL3
     , B...
FROM {{ ref("BIG_TABLE") }} B
    INNER JOIN {{ ref("FACT_PRODUCT_DEFS") }} F ON B.PRODUCT_ID = F.PRODUCT_ID

Distinct logic across teams is pain. Host a central definition. Argue about said definition, update said definition, cry about said definition, whatever… but allow everyone to read from a single definition.

Any time you see the same code written over and over alarm bells should be going off, there is a fire in the kitchen! Not only is this just MORE lines of code, but every instance its re-written is another opportunity for logic to become stale or be slightly miswritten. DRY DRY DRY. Do not pour water on the grease fire in the kitchen, suffocate it with a deluge consistent data/reporting instead.

This is such a common source of the “This number doesn’t match that number” problem. This is the “but it works on my machine” equivalent for day to day analytics.

The numbers don’t match up because distributed hard coded definitions are hosted across teams using different dependencies running at different times… I would be FAR more perplexed if the numbers DID match up. In fact I’d be completely dumb founded. The numbers have absolutely no business to be matching up because they don’t have a data model/governance to support them actually doing so. Its not even automatically testable because there’s no way to determine where the code diverges without, I guess, literally parsing jobs as text to rip out the dependencies and business logic.

There are so many different “types” of what I described above. For instance, possibly you have a data migration campaign and need to prefix all of your jobs with a new Database. In current state, hundreds of profiles have FROM clauses like FROM SCHEMA.TABLE but now needs to be FROM DATABASE.SCHEMA.TABLE to every single one… hopefully this illustrates why the above idea of ref is so helpful.

I’m not here to praise DBT but the Jinja templating solution to make SQL more dynamic handles some of these problems nicely. I think a better data model can resolve some nasty issues, but if you’re repeating logic or referencing specific relations over and over across profiles, assets, whatever… be prepared for pain.

If you’re working with an analytics team, one of the worst things that can happen is an erosion of trust in data quality from that team’s stakeholders. If you ordered crayons from Amazon and you receive pencils, or you set up an AWS billing alarm to send you an email when your spend has exceeded $100 and it doesn’t… you lose trust. One of the key pillars of a successful analytics team is building trust in their outputs (Also, move quickly I need this data yesterday lets go lets go… but that’s a given).

This is the unglamorous plumbing that underpins a successful analytics organization. While you can never fully control the madness that occurs in a 30 tab excel file authored by ten people off of windows shared drive you can rest easy knowing that at least whatever was copy pasted in there was reputable and “correct”.

Wide Tables

Sun, 06 Oct 2024 00:00:00 +0000

A few considerations before building that extremely wide table.

Every once in a while, an idea comes along to build a table with a wide array of columns at a particular granularity. This is usually associated with “self-serve analytics” and/or an attempt to define a “source of truth” table. Wide tables, self-serve analytics, and source of truth tables aren’t inherently good or bad… but there are some considerations to this type of table that I’m going to rant about.

When I talk about a single “wide table” I’m kind of talking about a denormalized table just an especially wide one with many loosely related columns. The opposite of this would be a normalized design where one would “store different but related pieces of information in separate logical tables (called relations)”. I don’t want to linger on this topic too much but typically a denormalized scheme speeds up reads at the expense of writes, and vice versa for a normalized schema.

I’ll also mostly be talking in the context of OLAP, not OLTP.

Lets define our wide table

For the sake of the rest of this thing lets define a wide table.

You work for a company that sells things. You record data on what was sold, how much was sold, what was projected to be sold, how much of that thing you had, how many distinct customers bought that thing, how quickly did you get that thing to your customer… I’m not going to hash out the full DDL for this table but you get the picture.

It should be apparent the sheer number of unique metrics that can be generated just from selling things!

You know what would be really nice? What if we decided on some granularity say… for a particular thing in a particular week we just record every metric about that thing for that week! So, a unique row in our table will be a combination of Thing + Week… and then we’ll store the name of the thing, the inventory, the shipments, the forecast, etc…

Note that this is a BATCH solution where data is written periodically at specific intervals in time… NOT a streaming solution where shipments are recorded in real time. Maybe the upstream sources this table is generated by are populated via some Kafka/Redpanda/Kinesis situation.

Why would you want to do this?

Maybe this table is for an LLM or ML based query generation solution, pulling data for non-technical users with ad-hoc business questions, maybe it’s to simplify the SQL associated with these ad-hoc queries to move joins/other logic upstream, speeding up reads (SELECT), or maybe the table needs to have a ton of columns for some other reason. That’s fine! Standardizing definitions is generally a good thing and improving the speed at which ad-hoc questions are answered is also good!

Dueling parties wielding Excel files generated by random queries stored and iterated on in Notepad, resulting in different outputs is always annoying. It’d be easier to trouble shoot if everyone was relying on the same source and using some form of version control (I know I’m only dreaming at this point). It’d be even better if you didn’t need to join any other relations on that source, simply SELECT & FILTER and you’re good to go, thus, our wide table.

What are some issues? [Begin Rant]

This table is now a dependency bomb.

All of these columns are derived from upstream tables, your inventory data is coming from scan events at your warehouse, outbound shipments from another source and data science is producing the forecast in a local Python Notebook triggered via CRON that writes to the windows shared drive, you’ve asked if this could be moved to sage maker and the resulting dataset cataloged and directly written to the data-warehouse but that’s “below the line” for this quarter.

You could very well now have a ball of 70 dependencies.

If any of those dependencies are delayed every metric will be waiting… so instead of just having the guy who needs to report inventory levels to the director on Monday pissed, everyone is pissed.

Now you’re like ok… if any dependency hasn’t satisfied by Monday at 5:00 am, it doesn’t matter I’m just going to force the job so that at least some people get their data. However, now you’ve just written dirty rows to this table… the inventory guy is still pissed because inventory is half of what it should be because you forced the data set before the latest scan events were written to the upstream table. You also KNOW that by forcing dependencies you’ve got incorrect data in your table and you’re just going to have to re-run your transform job again and actually let the dependencies satisfy this time. If you have a single transform job that means running everything again on your compute even if it’s already been written to the table successfully. You’ll also need some mechanism to notify the inventory guy that his numbers are going to be garbage Monday am, so he doesn’t accidentally report something ridiculous to the director.

Inventory guy does not like you he thinks you’re an ass, you think he’s an ass, inventory guy keys the right side of your 2022 Toyota Avalon, the Toyota Avalon is now discontinued and you’re sad because you should’ve waited and purchased the Toyota Crown, but you didn’t realize the Crown was coming out so you bought an Avalon, anyway… right hook to inventory guy! Now you’re being charged with assault! Now you’re in jail, your arraignment isn’t until Tuesday, but you need to get inventory numbers out by Monday! Who’s going to force those dependencies???

What about storage?

With this wide table, we’ll have the name of the product, maybe the product belongs to a broader group of products and we’ll throw the description of the product in there as well.

These facts about the product may change… but probably fairly infrequently.

In our table every week the same values will be included for those products over and over. Instead, they could be stored in a separate table that contains each product, the description, color, etc… and then that could be joined to our fact table. Now we are again beginning to talk about Normalization which is critical to think about when setting up your relations.

A few solid rebuttals to the above statement given that we are mostly talking about an OLAP situation:

Modern columnar storage formats like Parquet are efficient at compressing data, especially repeated values in a column… something like MVC
Data storage is relatively cheap… if we are only talking in the millions of rows, it’s just not going to be a major driver of cost
Denormalized data often performs better for analytical queries, with columnar storage, queries will only need to scan the columns as denoted by the query

A few points on Inserts/Updates/Deletes

This is where denormalization can be slower and get more complicated. Let’s say we want to change the product group that one of our products belongs to in the denormalized table. We’ll want all historical data to be attributed to the new group so we’ll need to update the previous product group in every row.

We can also launch a new product, but it’s possible data science hasn’t finished the production forecast for that product yet… now the forecast column is NULL because we aren’t going to stall inserting the rest of the columns for that product just because the forecast is NULL. However, people that read from this table will need to understand that the forecast is NULL and choose how to deal with it accordingly. If an analyst is tasked with deriving forecast accuracy, and they notice some NULLs and just NVL(forecast_column, 0)… that’s just not true, science has a beta forecast for the product and its certainly > 0… there’s a reason we decided to launch the new product after all, we predicted customers will want to buy it!

To be fair, if you launch something like the Pontiac Aztek maybe anticipating 0 demand is the correct choice when attempting to sell an angry kitchen appliance of a car.

Where will your data model break down?

Q: Hey I know we have that inventory column, could we break it out into the different inventory dispositions (Sellable, In transit, and Damaged)?

A: Sure, now our table increases in... width...

Q: Hey I know we have shipments for all of last week, and its only Thursday but I’d really to see what we shipped out Mon/Tues/Wed of this week to see where we are at.

A: We cant really answer that with the demoralized table at the current granularity… even if it was daily we’d probably be waiting on dependencies and wouldn't be able to get back to this person until Friday. Now this ask simply becomes a query against the upstream shipments table for the latest data.

Q: Hey I’m using that wide table you made but I want to see how the descriptions for this product have evolved over time; however, all the descriptions are the same in this table?

A: This question will need to be directed towards another table that contains this info as the denormalized table has been updated to only reflect the latest definition.

Just as with anything there are tradeoffs. Above, I’m posing specifically tricky questions of the table, but it would do extremly well with basic aggregations and reducing the knowledge burden for end users of all the different upstream tables.

I only bring up these points because I think they are relevant to think about when designing how you want to capture your data. Again, there is no right or wrong, it’s simply context/data dependent and the right choice will be different across different teams. However, there is a clear right and wrong when it comes purchasing a Pontiac Aztek; I’ll leave that exercise to the reader.

[6 / 4 = 1 = 1.5]: Type Coercion and Precision in SQL

Sat, 06 Jul 2024 00:00:00 +0000

An overview of what’s to come:

I was originally writing this as a high-level introduction to send people when they encounter things like 6/4 returning 1 instead of 1.5 in some Database Management Systems (DBMS)… overtime its sprawled far beyond that. The points below serve as a TLDR, with a sprawling discussion expanding afterward.

Operations like division in SQL can be confusing for novices or anyone without a programming background
In some DBMS, 6/4 may equal 1 because, dividing two integers a.k.a. integer division returns an integer (the remainder is discarded)
```
 Casting one of the operands as a float or numeric/decimal will ensure floating point
 division. Be careful with the precision needed in your calculation as there are
 tradeoffs to using certain types.
```
Syntax and behavior will be different for various DBMS, but if you want 6 / 4 to equal 1.5 and not 1 then you can use something like:

PostgreSQL:
   SELECT 6::FLOAT / 4 AS result -- Type casting using "::" operator
   SELECT 6 / 4::FLOAT AS result -- Casting the divisor
   SELECT CAST(6 AS FLOAT) / 4 AS result -- Using the CAST function

MySQL:
   SELECT 6 / 4 AS result -- MySQL automatically performs type conversion
   SELECT 6 / CAST(4 AS DECIMAL(10,2)) AS result -- Explicit CAST to DECIMAL
   SELECT 6 / CONVERT(4, DECIMAL(10,2)) AS result -- Using the CONVERT function

SQLServer:
   SELECT 6.0 / 4 AS result -- Implicit conversion using decimal point
   SELECT CAST(6 AS FLOAT) / 4 AS result -- Explicit CAST to FLOAT
   SELECT CONVERT(FLOAT, 6) / 4 AS result -- Using the CONVERT function

Oracle:
   SELECT 6 / 4 AS result FROM DUAL -- Automatic type conversion
   SELECT 6 / TO_NUMBER(4) AS result FROM DUAL -- Using the TO_NUMBER function
   SELECT 6 / 4d AS result FROM DUAL -- Explicit type declaration

SQLite:
   SELECT CAST(6 AS REAL) / 4 AS result -- CAST to REAL
   SELECT 6.0 / 4 AS result -- Implicit conversion using decimal point
   SELECT ROUND(6.0 / 4, 2) AS result -- Throwing ROUND in there

-- Methods may have trade-offs in terms of precision, performance, and portability.
-- The choice depends on your specific requirements and the DBMS you're using.

This is NOT an exhaustive list of solutions/syntax for all DBMS as there are many DBMS

The idea here is that if you desire a specific output, explicitly casting an operand(s) is the safe bet. If you are the data producer, understanding the consumption patterns and applying the correct type will be critical for your consumers downstream
It’s crucial to understand the database management system (DBMS) and environment you’re working in before curating datasets, creating models, or exporting data for reporting

I think that’s a fair synopsis of what I’ve written below. If you’re interested, I’ll continue to cover division across a variety of DBMS, the behavior of data types/operations in SQL, Hotdogs, Docker, rug pulls and BIG DOGS!.

Observations about SQL Writers

Division is a frequent source of logical and syntactical mistakes for novice SQL users. SQL has some round edges for non-technical users like 1 based indexing. 1 based indexing is the SQL standard, and its more straight forward for people who would typically begin counting at 1 and have no familiarity with 0 based indexing found in languages like Python, Java, or C.

Division is more of a sharp edge. Not because it’s difficult to do division, but when you take someone who has always done =6/4 in Excel and received a result of 1.5 and now you show them that it evaluates to 1 when they write SQL against a database… they don’t anticipate that, nor would they have thought to check for it!

Some Typing and Coercion Background

Many users of SQL coming from a “business” background will have encountered different data types in Excel, but they will likely not have given any thought to Static vs Dynamically typed programming languages. I’m now going to ~~play with fire~~ lay out some definitions to provide some context to the previous sentence that are debatable, but I believe they provide a good mental model for SQL.

SQL is typically considered to be Statically typed, meaning:

Every data item has an associated data type defined at compile-time

SQL does have predefined data types for columns in table definitions, which are determined at “compile-time” (more accurately, at table creation time). However, there are dynamic aspects… SQL allows for implicit type conversions in many operations, and the actual type checking often occurs at runtime.

SQL is also most properly defined as Strongly Typed:

Every data item has an associated data type, defining its behavior and allowed usage

SQL uses a mix of static and dynamic typing. Data types are associated with column definitions at table creation time, but many type-related decisions and checks occur at runtime. SQL’s type system allows for implicit conversions and exhibits some characteristics of both strong and weak typing, with the specific behavior often depending on the particular SQL implementation.

Great, so when I think of SQL I’ll think Strong and Static, thanks! Well… no, like I mentioned, this behavior depends on the DBMS. SQLite uses a more general dynamic type system where the datatype of a value is associated with the value itself, not with the column it sits in. This behavior can be modified with the “STRICT” table option keyword.

An in-depth image for visual learners (I’m an aspiring artist):

Ok… that’s cool, but in SQL what happens when I divide an integer by a float or what if I evaluate an integer and a string for equality? In a more general sense we are asking: How do DBMS handle type coercion?

Type Coercion: is an implicit process where the language runtime automatically converts a value from one type to another. This is done to make operations between different data types possible without explicit instructions from the programmer.
Type Conversion: is an explicit process where the programmer manually converts a value from one type to another using specific functions or methods. This requires a deliberate action and is often clearer in terms of code readability and intent.

You’ll see that DBMS have extensive facilities for evaluating mixed-type expressions. I’m adding the Postgres docs here because they are good, but there are other docs for other DBMS.

Why should I care, and how about a few examples?

To make this implicit conversion explicitly clear, lets directly compare two values that are explicitly cast as different types:

SELECT CAST(5 AS INT) = CAST('5' AS VARCHAR(10))

Postgres: ERROR: operator does not exist: integer = character
SQLite: 1
Redshift: true

Some programming languages are quite strict when doing comparisons or operations on multiple types. Here is an example in Rust where someone has attempted to add an integer and a float and received an error.

fn main() {
    let float = 5.0;
    let integer = 4;
    let result = float + integer;

    println!("Result: {}", result);
}
//error[E0277]: cannot add an integer to a float
//     let result = float + integer;
//                          ^ no implementation for `{float} + {integer}`

Executing SELECT 5.0 + 4 or even SELECT '5' + 4 in most (maybe all?) DBMS returns 9.
So then what about SELECT CAST(5 AS INT) + CAST('4' AS VARCHAR(10))? This still returns 9 in some DBMS but NOT in Postgres:

In Postgres, an undecorated string literal (e.g., ‘4’) without an explicit type is treated as an unknown type. This is actually one of a few Pseudo-Types that allows Postgres to delay type resolution until more context is available. Here is a nice response from Tom Lane posted 13 years ago.

Using pg_typeof we can see that SELECT pg_typeof(CAST('4' AS VARCHAR(10))); returns character varying while SELECT pg_typeof('4'); returns unknown. Due to the specific casting of ‘4’, the second example errors (integer + character varying) while SELECT '5' + 4 evaluates an unknown + integer which Postgres implicitly coerces to an integer result of 9.

However, this is where you need to be careful. For instance, Redshift is a fork of Postgres, but… guess what: SELECT CAST(5 AS INT) + CAST('4' AS VARCHAR(10)) returns ‘54’. A few other interesting anecdotes from Redshift:

----------------------
-- Updates
----------------------
CREATE TEMP TABLE SOME_TUPLES (
    MY_NUMBER INT,
    MY_WORD VARCHAR(100)
);

INSERT INTO SOME_TUPLES(MY_NUMBER, MY_WORD) VALUES
(4, 'Hello'),
(6, 'Howdy');

-- Redshift has no problem updating the VARCHAR(100) MY_WORD column with a value I explicitly cast as an INT
UPDATE SOME_TUPLES SET MY_WORD = 4::INT
WHERE MY_NUMBER = 4;

-- How about a decimal instead? No problem
UPDATE SOME_TUPLES SET MY_WORD = 4::DECIMAL(38,0)
WHERE MY_NUMBER = 4;

----------------------
-- What about AVG()?
-- I see a lot of people use AVG in SQL that actually don't want the arithmetic mean
----------------------
WITH CHECK123 AS (
    SELECT 6 AS NUM
    UNION ALL
    SELECT 4 AS NUM
    UNION ALL
    SELECT 1 AS NUM
    )
SELECT AVG(NUM) FROM CHECK123 -- Results in 3

----------------------
-- Oh, Also
----------------------
-- Returns 2 in Postgres/Redshift
-- Returns 1 in SQL Server
SELECT CAST(1.99 AS INTEGER)

Ooooook so can we go back to Division now?

In some DBMS, SELECT 6 / 4; would result in 1 due to integer division, other DBMS will still return 1.5 even after being passed two “integers” to operate on.

Integer division is the division of two integers resulting in another integer. It discards the fractional part of the division and returns only the whole number quotient
In Python print(6/4) returns 1.5 (round edge); however, print(6//4) would return 1, discarding the remainder. We’ll see later on that DuckDB has similar syntax!

However, SELECT 6.0 / 4; or SELECT CAST(6 AS FLOAT) / 4; would result in 1.5 as the DBMS performs implicit type coercion and returns the result to the client as a float.

Floating-point division is an operation involving at least one floating-point number that results in a floating-point (decimal) number. Floating-point division is used when precision is important, and the fractional part of the result is significant.

Different DBMS have different defaults for this coercion behavior. Sometimes its: “Hey Big Dog looks like you gave me two integers two divide, I’m going to assume that the remainder is important to you and not return an integer” OR “Big DOG! You just served me up two integers and guess what… you are getting an integer back!”

SQL Standard is for the DBMS to refer to their users as Big Dog, just like Tiger.

I’m only talking about type coercion, but keep in mind that there are A LOT of DBMS and they do A LOT of things differently.

    Quick side note, in DuckDB there are two division operators: / and //.
    They are equivalent when at least one of the operands is a FLOAT or a DOUBLE.
    When both operands are integers, / performs floating point division (5 / 2 = 2.5)
    while // performs integer division (5 // 2 = 2).

On top of this, admin can make configuration changes to the DBMS you’re using, possibly making its behavior different than what I describe here! Developing a sound understanding of the environment you plan to write queries in before you or some text to SQL engine starts spinning off queries is important!

If you’d like to compare for yourself, SQL Fiddle is a nice online SQL compiler. You can also use docker. With docker installed, you can pull the official MySQL image and try for yourself:

# Start a new MySQL container named 'testsql' with the root password set
# The -d flag runs the container in detached mode (in the background)
$ docker run --name testsql -e MYSQL_ROOT_PASSWORD=ilovemysql -d mysql:latest

# List all running Docker containers
# This should show the 'testsql' container we just started
$ docker ps

# Start an interactive bash shell inside the running 'testsql' container
# The -it flags allow for an interactive terminal session
$ docker exec -it testsql bash

# Inside the container, start the MySQL client
# The -u flag specifies the user (root in this case)
# The -p flag prompts for the password we set earlier
$ mysql -u root -p

# Now we're in the MySQL prompt. We can run SQL commands here, for example:
SHOW DATABASES;
SELECT 6/4;  # This will return 1.5000 in MySQL

# To exit the MySQL client
EXIT;

# To exit the container's bash shell
$ exit

# Stop the 'testsql' container
$ docker stop testsql

# List running containers again to verify 'testsql' has stopped
$ docker ps

# List all containers, including stopped ones
# This will still show 'testsql', but in a stopped state
$ docker ps -a

In Practice

In my day job, 99% of the time people want floating point division, and I will advise them to explicitly cast one of the operands. For example in Redshift I would write: SELECT 6/4::FLOAT.

A good question might be: “Why bother casting if one or both of my columns is already a Decimal/Float, isn’t it irrelevant?”

In current state you would be 100% correct, but are you 100% sure nobody will alter the table and change that one column that was a Decimal to an INT…

Hey remember that old column we created as a Decimal type, we only write integer values between +/- 100000. We can save some space by storing it as an integer… nobody will have any problem with that! Sounds good: ALTER TABLE TBLONE ALTER COLUMN COLUMNONE TYPE INTEGER USING (NULLIF(COLUMNONE, ‘’)::INTEGER);
Hey that table we produce yeah we are actually deleting that column you use, but its ok because you can recalculate it using a JOIN on this other table to bring in another column… make sure it doesn’t break your transform… thanks! (We sent the email about the migration 3 months ago but must have missed you.. haha sorry lol… well we are making the change on Friday, thanks so much!)

I typically prefer writing defensive SQL and checking what I’m being defensive against rather than having the data producer pull the rug on me. Which is why I prefer to be the data producer, and why I prefer not to buy Meme Coins.

I’ll leave a final furthering thought for anyone interested. I’ve gone ahead and stolen this little exercise from the MySQL docs and re-written it for Redshift (should work identically in Postgres as well)

-- Three variables: i (an integer), d (a decimal with 10 digits total and 4 digits after the decimal point), and f (a floating point).
-- The WHILE loop runs 10,000 times, incrementing both d and f by 0.0001
-- After the loop, RAISE INFO prints the values of d and f.
CREATE OR REPLACE PROCEDURE test_precision()
LANGUAGE plpgsql
AS $$
DECLARE
    i INT := 0;
    d DECIMAL(10,4) := 0;
    f FLOAT := 0;
BEGIN
    WHILE i < 10000 LOOP
        d := d + 0.0001;
        f := f + 0.0001;
        i := i + 1;
    END LOOP;

    RAISE INFO 'DECIMAL result: %, FLOAT result: %', d, f;
END;
$$;

-- Call the stored procedure
CALL test_precision();

Here’s a hint if you’re curious about the output, or you can just smash this stored procedure into ChatGPT and it’ll rattle off a coherent answer the choice is yours.

Temporary Tables

Sun, 05 May 2024 00:00:00 +0000

The database management system (DBMS) I’ll reference today is Redshift (AWS Cloud Datawarehouse offering) which is based on Postgres. I’ll dive into some of the anti-patterns around Temporary Tables I’ve seen abused during my work experience, why they don’t make sense, and how to fix them.

Lets get it out of the way… Temporary Tables are not “better” than Common Table Expressions (CTEs) and CTEs are not “better” than temporary tables. It is entirely dependent on the context, the data, the DBMS, the query, upstream tables, etc… because at the end of the day the stakeholder wants the 1MM row output in excel so they can make a pivot table. They don’t care if it was procured via CTEs, TEMP Tables, or that you put another hole in the home-office dry wall.

A few other notes:

In Redshift (and many other DBMS for that matter) there are plenty of excellent use cases for Temp Tables (I’ll provide a few at the end)
Be wary of Premature Optimization. If analytics workloads are running in an acceptable amount of time and not incurring unacceptable expenses, then effort is probably better spent elsewhere

Read, Write, Read, Write:

Internally, most analytics queries we run scan TBs of data (Redshift Spectrum costs $5 per TB of data scanned from S3, but hey… screw it SELECT * it’s a business expense, Right!, Right?). Most commonly, large Transform or Extract jobs are a series of Temporary tables, there is nothing ~~necessarily~~ wrong with this. The issues come about when large chunks of unfiltered data are passed through this series of Temporary tables, being read and written over and over again.

SELECT statements encased in a CREATE TEMP TABLE AS will read data from the upstream source and write it to disk.
In a particular session these tables can be seen in STV_TBL_PERM, they will be collected after the session is closed (obviously permanent tables will not).
Repeatedly reading and writing data too and from temp tables throughout the course of a query can be costly both monetarily and temporally.

Issue 1: Create a temp table just to SELECT everything out of it

Uselessly writing the entire result set to disk just to read it all back out. Just run the final SELECT, there is no need for the temp table. This also presents an additional opportunity for a statement to get stuck in the que behind other workloads.

------------------------------------------
-- NOT GREAT, I SEE THIS WAY TOO OFTEN
------------------------------------------
CREATE TEMP TABLE FINAL AS (
SELECT EVERYTHING
FROM SOME_SCHEMA.BIG_TABLE
);

SELECT *
FROM FINAL

------------------
-- BETTER
------------------
SELECT EVERYTHING
FROM SOME_SCHEMA.BIG_TABLE

Issue 2: Unnecessary data and losing optimizations

Redshift is a columnar database. Each column is stored separately, meaning if you don’t need a column Redshift can ignore scanning the data entirely. Yes, if you're using Spectrum for your OLAP queries and your data is sitting in S3 in abunch of .csv, Redshift will have to scan the whole file... but its probably time for Parquet? In the below SQL:

Redshift is forced to read and store columns BT.COL4, BT.COL5, BT.COL6 , BT.COL7 in TEMP_1 even though they are never referenced again in the rest of the query.
The query optimizer can’t parse further down in the query that all that was wanted were some fairly specific predicates; it could have skipped many blocks of data. Each of the select statements is on its own island in a way. Redshift stores data in 1MB blocks, with additional metadata about these blocks. This meta data (a.k.a zone maps) is what allows the DBMS to avoid scanning irrelevant blocks.

CREATE TEMP TABLE TEMP_1 AS (
SELECT BT.COL1
            , BT.COL2
            , BT.COL3
            , BT.COL4
            , BT.COL5
            , BT.COL6
            , BT.COL7
FROM THIS_IS_A.BIG_TABLE BT
);

-----------------------------------------------------------------------------
-- Now that we’ve read everything and written to disk lets read it back out
-----------------------------------------------------------------------------
CREATE TEMP TABLE TEMP_2 AS (
SELECT BT.COL1
            , BT.COL2
            , BT.COL3
            , ST.COL9
FROM THIS_IS_A.BIG_TABLE BT
	INNER JOIN TINY.SMALL_TABLE ST ON BT.COL1 = ST.COL1 AND BT.COL2 = ST.COL2
);

------------------------------------------------------------------------------------------
-- FINAL SELECT, lets read everything back out, after creating TEMP_2 for… no reason?
------------------------------------------------------------------------------------------
SELECT *
FROM TEMP_2
WHERE COL1 = ‘I Should Have’
    AND COL2 = ‘Applied these predicates’
    AND COL3 = ‘Earlier

SQL is a declarative language, you indicate to the DBMS what you want and the DBMS goes off and determines the best way to retrieve that for you. The series of TEMP TABLES above augments this process and confines the DBMS to only performing rudimentary optimizations without broader context.

These optimizations can be implemented manually by the author of the query, but would otherwise be done automatically by the DBMS. Its like paying your chauffer a bunch of money to drive you to the airport, but, instead, you wrestle him for the keys, tie him up in the back seat of the car, turn on Maroon 5 (You love Maroon 5), and drive to the airport yourself? Why do that?

Issue 3: Sort Keys, Multiple Sort Keys???

Something that seems to be fairly common is defining a SORT KEY within the CTAS syntax when the column used in the sort key should already have been filtered on in the first place during the creation of the Temp Table. This is essentially identical to my point above about filtering after the Temp Table Creation. Defining multiple sort keys at this point just makes things worse.

The kicker is that Redshift will attempt to determine the optimal SORY KEY and DIST KEY for any Temp Table based on the query plan. Manually specifying it in the temp table only to filter it out later is just a bit painful to see and flies in the face of aggregate early and often. More info can be found in the advanced table design playbook. This is probably a good time to mention that its important to make sure table statistics are up to date.

------------------------------------------------------------------------------------
-- THE ADDITIONAL SORTING MAKES THE WRITE MORE EXPENSIVE WITH NO DOWNSTREAM BENEFIT
-- WE'VE DISTRIBUTED THE DATA ACROSS THE NODES IN THE CLUSTER USING COL4
------------------------------------------------------------------------------------
CREATE TEMP TABLE TEMP_1 SORTKEY(COL1, COL2, COL3) DISTKEY(COL4) AS (
SELECT BT.COL1
     , BT.COL2
     , BT.COL3
     , BT.COL4
     , BT.COL5
     , BT.COL6
     , BT.COL7
     , …
FROM THIS_IS_A.BIG_TABLE BT
);

SELECT BT.*
FROM TEMP_1_SORT BTA
        INNER JOIN ANOTHER_TABLE ATA ON BTA.COL4 = ATA.COL4 -- AT LEAST WE COLLOCATED THE JOIN COL...
WHERE BTA.COL1 = ‘Yikes’ -- SHOULD HAVE FILTERED ABOVE
;

In regard to distributing on COL4 someone may think:

“I want to save data in a Temp Table and specify a distribution key, so that I can improve my JOIN later on”

There is certainly some thought going into this argument, but there is no free lunch.

The redistributed data may improve the later join by collocating data across the cluster’s nodes, however, all that shuffling across different nodes still had to occur when the temp table was created!
While this is a valid argument, it would be important to benchmark the run times to actually determine what is most optimal here.
If the current query patterns don’t utilize the table’s partitions (sort keys) and rarely join on the column its distributed on… it might be time to think about adjusting the upstream table rather than having all subsequent queries cache the table in a temp table just to sort and redistribute it before further processing.

When are temp tables a good choice?

If the temporary table is going to be read from numerous times, it may be best to filter once and read several times from the aggregated data. This was especially handy when Redshift used to lack ROLLUP, CUBE, and GROUPING SETS
Using Temp Tables to chunk up pieces of a query can be very handy in development, as they are persisted for the duration of the session, things like tests, logical checks, and quick aggregations can be run against them
Using a series of CTEs within a temp table can be a happy medium, doing more work and writing out less often
Honestly, I’d take 100 temp tables over having to parse through some nested sub query mess

SELECT FROM(
    SELECT FROM(
        SELECT FROM(
            SELECT
            FROM
            WHERE (COL1 IN (SELECT FROM))

        )
    )
)

Posts on Sam's Blog

A Review of Hex

Hex Review

The data

The Prompt

Dropping in

The first cell

The LLM & Context

But wtf is the compute here

Other staff in the left most toolbox

Back in the middle

Visualizations

Make an… App?

Competitors

Increasing the data size

Overall Thoughts

SQL Buckets

Case Statements:

Solution:

I’m worried about an unconstrained series of buckets… how do I cap it?

I want to change the increment size

I have more than 99 buckets, the two digit solution wont work:

Your Data Warehouse Ignores Your Primary Key

The agenda…

A little background

Our VP needs to know

PK in name only

Write performance

Insert with & w/o a PK

Merge Upsert

These constraints should be enforced upstream

Query Banks

That query bank should be a semantic layer

No, ahh! The BANK IS CLOSED

The Bank is open

It’s your problem until you fix it or find the owner

Table Creation

Collection of thoughts on table creation.

Stop building on top of bad decisions

One step back, two steps forward:

DRY Analytics

Ok so how do we avoid pure pain?

Wide Tables

A few considerations before building that extremely wide table.

Lets define our wide table

Why would you want to do this?

What are some issues? [Begin Rant]

This table is now a dependency bomb.

What about storage?

A few points on Inserts/Updates/Deletes

Where will your data model break down?

[6 / 4 ~~= 1~~ = 1.5]: Type Coercion and Precision in SQL

An overview of what’s to come:

Observations about SQL Writers

Some Typing and Coercion Background

Why should I care, and how about a few examples?

Ooooook so can we go back to Division now?

In Practice

Temporary Tables

Read, Write, Read, Write:

Issue 1: Create a temp table just to SELECT everything out of it

Issue 2: Unnecessary data and losing optimizations

Issue 3: Sort Keys, Multiple Sort Keys???

When are temp tables a good choice?

[6 / 4 = 1 = 1.5]: Type Coercion and Precision in SQL