Projects on Sam's Blog

Robinhood Gambling

Fri, 22 Aug 2025 00:00:00 +0000

The Robinhood gambling factory

Robinhood has a nice mobile experience, they have good products/deals, and they also have a credit card.

There has been a lot of talk about Robinhood engaging with human “gambling instincts”.

Just search “how has Robinhood gamified investing” and you can watch confetti rain down as someone yolos their life savings into GME 0 DTE calls

Again, I’m not saying the product is all bad, but there is one piece which is astonishingly ridiculous to me.

The Credit Card

I’m not going to write about the Casino-like aspects of the brokerage product… but there is also the Robinhood CREDIT CARD!

I opened the card a few months ago, payed the $50 Gold annual subscription, applied, and got the card. The card is kind of nice… the gold is a little gimmicky, but the 3% back on everything and 5% on travel is good for someone that doesn’t want to swap cards around for each kind of purchase.

I can just nail down 3%, convert those points to cash that lands straight back into Robinhood, and earns 4% APY. Am I leaving a few bps on the table… sure, but everything is contained in the same ecosystem, transfers are fast, and 4% APY on my uninvested cash is fine.

The Mystery Box

For the first time, I go to exchange my points for cash, 1 point = 1 cent. I’m not going to fuss around with gift cards/shopping stuff, I want my cash, and I want to go make 4% on it.

After opening up the rewards tab, I’m forced to scroll ALL THE WAY TO THE BOTTOM to get to the “Cash Back” section. I’m scrolling through shop with points, travel with points, buy Robinhood merch with points and Low and Behold I see “The Mystery Box”… what the fuck is “The Mystery Box”.

Well, I’m glad you asked, and you can retrieve the full details here… but, in short, the Mystery box is LITERALLY an online casino.

That’s right, You can shuttle 1000 points (~$10) into this thing and come out with… less than $10! Amazing, lets break it down.

A few things to start, the odds do sum to 100%, so you’re guaranteed to win one of these things. Each win is independent and the odds don’t change as far as I can tell. So lets look at expected value:

“Cash” Prizes:

10,000 Points @ 0.50%: 10,000 Points = ~$100, $100 * 0.50% = 50 cents
7,500 Points @ 0.75%: 7,500 Points = ~$75, $75 * 0.75% = 56.25 cents
5,000 Points @ 1.50%: 5,000 Points = ~$50, $50 * 1.50% = 75 cents
2,500 Points @ 2.50%: 2,500 Points = ~$25, $25 * 2.50% = 62.5 cents
1,500 Points @ 4.00%: 1,500 Points = ~$15, $15 * 4.00% = 60 cents
1,250 Points @ 5.00%: 1,250 Points = ~$12.50, $12.50 * 5.00% = 62.5 cents

Total EV: $.50 + $.5625 + $.75 + $62.5 + $.60 + $.625 = $3.6625

Hmmm… I hope we catch up a little bit on the cashback prizes.

Cashback Prizes:

A few notes on cashback:

We must factor in the fact that we would have earned 3% on these purchases anyway, thus, we have to take cashback% minus 3% to calculate the true net benefit
To maximize the value you must spend up to the cap. This means you must WANT to purchase from where you win and be LIQUID/WILLING enough to make said purchase e.g. spend $3500 in the next 24 hours.

Assumptions:

I’m going to assume the opportunity cost of capital is $0, this is especially generous as the $500 you spend at Target could be used for investing in 0 DTE options on… Robinhood!…
For prizes with no Cap, I’m going to assume average cost (if I can find it) + 25%, otherwise just going to insert a reasonable guess
For prizes with a Cap, I’m going to assume the reward is maximized
I’m going to assume the above conditions hold for 33% of the items below. This is a big assumption, but it is fair given the variety of options and the previous conditions. This means we’ll cut our EV calculation for this section in 1/3 when it’s all said and done.

Disagree with my assumptions? I simulated the whole thing so you can test for yourself and see if it’s worth it for you. This is actually ripping the rewards straight from api.robinhood.com (and doing so poorly), so it will inevitably break at some point, but it’s fun for now!

10% cashback on next grocery purchase ($500 cap) @ 5.00%: $500 * (10-3)% = $35 * 5% = $1.75
10% cashback on next automotive gas purchase ($300 cap) @ 6.00%: $300 * (10-3)% = $21 * 6% = $1.26
5% cashback for 24 hours ($3,500 cap) @ 4.00%: $3500 * (5-3)% = $70 * 4% = $2.8
10% cashback on next Uber or Lyft ($300 cap) @ 7.00%: $300 * (10-3)% = $21 * 7% = $1.47
15% cashback on next Starbucks purchase @ 8.00%: $25 * (15-3)% = $3 * 8% = $.24
4% cashback for 24 hours ($3,000 cap) @ 4.50%: $3000 * (4-3)% = $30 * 4.5% = $1.35
5% cashback on next Amazon.com purchase ($500 cap) @ 8.00%: $500 * (5-3)% = $10 * 8% = $.80
5% cashback on next restaurant order ($500 cap) @ 8.00%: $500 * (5-3)% = $10 * 8% = $.80
5% cashback on next DoorDash or Uber Eats ($300 cap) @ 8.00%: $300 * (5-3)% = $6 * 8% = $.48
5% cashback on next Target purchase ($500 cap) @ 6.00%: $500 * (5-3)% = $10 * 6% = $.60
5% cashback on your next phone bill @ 6.00%: ($141 * 1.25) * (5-3)% = $3.525 * 6% = $.2115
5% cashback on your next Lululemon, Alo Yoga or Vuori purchase @ 8.00%: $300 * (5-3)% = $6 * 8% = $.48
4% cashback on next spa purchase @ 7.00%: $300 * (4-3)% = $3 * 7% = $.21
5% cashback for 30 days ($10,000 cap) @ 0.25%: $10000 * (5-3)% = $200 * .25% = $.5

Starbucks, Lululemon, Alo Yoga or Vuori, and Spa purchases I think are fair estimates. The phone bill at least has a source. Feel free to throw some different estimates in there and test it out for yourself.

To find the total EV, we will sum the individual values:

$1.75 (Grocery)
$1.26 (Gas)
$2.80 (5% for 24h)
$1.47 (Uber/Lyft)
$0.24 (Starbucks)
$1.35 (4% for 24h)
$0.80 (Amazon)
$0.80 (Restaurant)
$0.48 (DoorDash/Uber Eats)
$0.60 (Target)
$0.2115 (Phone Bill)
$0.48 (Lululemon etc.)
$0.21 (Spa)
$0.50 (5% for 30 days)

And get: $12.9515 which we can divide by 3 to get $4.3172.

Adding this to our cash prizes, we expect an EV of $3.6625 + $4.3172 = $7.9797!

$10 in… $7.9797 out. You can play around with the assumptions but unless you’re ready to maximize every single one of the Cashback prizes… every single time you roll the dice on this Mystery Box you’re handing $2 back to Robinhood in the form of them not having to pay out the true value on their 3% rewards program.

This is the definition of a Casino. Robinhood will argue that the program is more than fair as it’s positive EV for someone willing to maximize every single one of these rewards. We can go around in circles about this, but if you land on the 5% cashback on next DoorDash or Uber Eats ($300 cap) @ 8.00% the maximum value one can accrue is .02*300 = $6…

Funny Thoughts (This is not legal or financial advice)

Ok so there are three uncapped offerings:

4% cashback on next spa purchase
15% cashback on next Starbucks purchase
5% cashback on your next Lululemon, Alo Yoga or Vuori purchase

Now if you haven’t head of Konstantin Anikeev let me introduce you. Konstantin curated this beauty:

Purchasing Gift Cards: Using his American Express 5% ‘cash back’ Rewards Program, he Purchased Visa gift cards at grocery stores or pharmacies where the 5%Rewards applied.
Although Visa cards cannot be converted to cash, they can be used to purchase Money Orders. He would purchase the Money Orders with the Visa gift cards.
Deposit the Money Orders to his Bank Account.
Pay the Amex Balance.
Repeat to the tune of $6.4MM in purchases and $320,000 in ‘profits’ by converting has 5% cashback in excess of fees

There’s also an interesting tax ruling in Konstantin Anikeev v. Commissioner, but this should now draw your eyes to that uncapped 15% starbucks purchase. The 5% and 4% uncapped offerings are interesting, but 15%…

Lets say we call up Starbucks and tell them we have a $20,000 limit on a card paying 15% on our next purchase, how could they turn down this opportunity. Better yet, what if Starbucks wanted to outfit a few stores in Lululemon and Lululemon wanted to buy starbucks for its employees… we could facilitate this trade, in which case I’d be hitting the mystery box like crazy.

Arbitrage

Sat, 14 Jun 2025 00:00:00 +0000

Winning Every Time

I’m going to summarize arbitrage sports betting. I think I’m going to add this to “projects” although the artifacts/results of my time spent in this space aren’t really relevant to this post. Also anytime I reference myself, me, or I in this post I’m actually talking about Jonathan

Is that click bait? I’d argue its not because its true, and I don’t make any money or care who reads this so… no I don’t think so.

The scene

In 2021 or 2022 as sports books were rolling out across the U.S. they all had large promotions “Bet $5 get $200 in free bets” kind of thing. This was essentially just free money as there are many places that will freely show you how to optimally liquidate your free bets into cash. If you read through that link you’ll see that what you use is a hedge against your free bet… so

if your free bet wins you get a nice payout minus your hedge.
if your free bet looses you get a nice payout from your hedge minus nothing… the bet was free… or the cost of becoming eligible to receive the promotion.

This is all fun and stuff, but I quickly liquidated all the cash out of these opportunities and was left with a crippling addiction… just kidding… but gambling addiction is no joke and extremely sad. I was curious to know if sports books ever mis-priced lines, how lines were created, or if you could pit two sports books against each other…

Background on the books

Online Sports books are no better than playing roulette at a brick and mortar casino. They play on human biases and tendencies toward lottery like payoffs to extract value from players over time. Those who consistently win are banned or limited as value extraction is a one way street.

Sportsbooks typically have well informed lines. A betting line refers to the odds that a sportsbook lists for a wager. There are different ways to display odds. I’m going to specifically talk in American odds just because that’s what I’m used to, not because they are superior in anyway (I think most professionals use decimal odds, but I could be wrong).

In American odds, the favorite is shown with a (-) and denotes the amount of $ you’d have to bet to win $100, if the odds are -400 you’d have to bet $400 just to win $100 meaning that “thing” you’re betting on is likely to happen. If the odds are +400 we are betting on an underdog, in this case, if we bet $100 we’d win $400. We can also determine the implied odds underlying the odds we are being shown.

For example:

+400 could be re-written to note the we are betting $100 to win $500 (We get our $100 bet back + $400 in winnings) -> 100/500 or 20% implied odds. For our favorite at -400 we are betting $400 to win 500$ (We get our $400 bet back + $100 in winnings) -> 400/500 or 80% implied odds.

These lines are constantly updated with the latest information and these books are also keenly aware of the action that is occurring on a particular line. I won’t go into detail here but this article has info on how books manage risk on specific lines.

So are we just going to look for lines that are “off”: This team should only be a +200 underdog but the book has them at +500? Given this information above, we are unlikely to catch sports books utterly mispricing a particular line… but maybe we can catch them being just so slightly off. We also have to remember that because these books are just extracting money from people there is a lot of competition to be an extractor. What I mean is that there are a lot of ways to gamble on sports, even Robinhood has begun adding basic predictions markets on sporting events. When all of these different places offers odds on an outcome; sometimes, they begin to disagree with one another. This is what we can take advantage of.

Taking both sides

What do I mean by “pit books against each other”. To explain this I want to walk through how these books function.

We are going to be a sports book for a second. Lets say sports team A is playing sports team B. At our book we think this is going to be a close match so we offer team A at +110 (slight underdog) and team B at -110 (slight favorite). I play the scenarios out in the below drawing.

Two players one on either side of the line place bets so that their total payout is equivalent ($210):

You can see our sports book makes no money regardless of the out come (It’s pot of $ generated by the players must be payed out in full to satisfy the bets regardless of the outcome), its just facilitating transactions, and it’ll be in the red after it has to pay its variable/fixed costs. This is why you’d never see these odds at a sports book outside of gimmicks like the Super Bowl coin toss, promos, etc… You’ll see lines like this:

What changed? The odds on Team B went from -110 to -130 (It became more of a favorite, a.k.a the sports book pays out less money on the same amount wagered). Previously, player 2 only needed to bet $110 to win $100 now they have to bet $130 to win $100. I added the implied odds of the specific outcome in the first diagram with a 47.62% chance our underdog wins and a 52.38% chance our favorite wins. These sum to 100%.

Now lets check out the implied odds in the +110 / -130 scenario… 47.62% and 56.52% respectively… these don’t sum to 100% anymore, now we get (104.14%) uh oh. NOW our book is making money:

if team A wins the books makes $20
if team B wins the book breaks even.
We calc the EV here assuming the true odds are -110/+110 as (.4762 * $20) + (.5238 * $0) = $9.52

Wow so our book has an EV of $9.52 every time two people put on this bet! And looking at little further if we divide our EV by the Pot of money: $9.52/$230 we get ~4% remember our new totaled implied odds 104.14%… 104% - 100% = 4% this is the books “hold”. Hold in sports betting refers to the percentage of money that sportsbooks keep for every dollar wagered. The percentage will vary by wager type. Sportsbooks can’t just maximize hold because they have… competition. If I can get +130 in book A where as book B offers +150 on the same bet… you’re going to bet with book B every time… sports books need volume they need your bets to make money.

And this is what these books want, they want as high volume of these transaction as they can get. That’s why they put on promotions and offer free bets and create Ads with tigers and famous people etc…

But do all books have the same odds? No. Just as books create lines that ensure they win, we can look for favorable lines as well. These books don’t exist in a vacuum they compete against each other for a pool of betters, many of which are sensitive to how great of a hold a book has on a particular line. As sports books update their lines, mis-price things, or disagree with each other we can take advantage. Lets look at this below.

In this scenario the same player is taking both sides of a line across two sports books. Sportsbook 1 offers +110 while sportsbook 2 offers the other side at -105. Lets look at what happens.

Bam and now we can see that no matter the outcome of this game, player 1 makes $5 or comes out even. Summing the implied odds 47.62% + 51.22% = 98.84% we have engineered an inverse hold (1.16%), where the player always comes out on top regardless of the outcome. Our EV on this bet, assuming true odds of +110/-110 is (47.62% * $5) + (52.38%* $0) = $2.38 and don’t forget $2.38/$205 = 1.16% (Our inverse hold).

Wow so if I have $205 I can instantly make 1.16% on it if I can find these lines?

And if you take that same $205 and use it again and again every night for a month I could make even more $?

And if I increased my bet sizes to $10,000 could I scale up the magnitude of my profit?

Yes, in our contrived +110/-105 example if we bet $10,000 on +110 and $10,500 on -105 we’d have an EV of (47.62% * $500) + (52.38%* $0) = $238… I like $238 better than $2.38.

What if we found something even better like +110/+110?

Yes that would be nice.

So what’s the catch?

Books aren’t dumb, they will see you are consistently hitting soft/slow lines. Lines with low hold or lines that weren’t updated fast enough to avoid mis-pricing across other books. They will ban you or limit you to only being allowed to bet a few dollars at a time.
It’s harder than you think to find these odds. Odds scraping is difficult. Scaling is hard. If you buy odds… that’s going to cost you $, and they are slightly delayed. You are going to struggle to place bets via an api, most books don’t have one, so you’re going to have to build something to place these bets via the UI or manually do it (Doesn’t scale well).
Odds can change as you take both sides as there will inherently be some latency between the two bets. If they change in the wrong direction and you already have exposure to one side… now you’re just trying to limit your losses.
I said before that books will limit you, they can also void your bet. That’s right, if the sportsbook realizes it gave you too good of odds it will just void the bet. Bad analogy alert: Imagine you open a 12 month CD at 4.5%, sign the paper, add the money done… you wake up the next day and the CD isn’t there any more. You call the bank… “Yeah umm we actually don’t want to give you 4.5% so we closed the account… how about 3%”. Sports books can and do just that… remember you are on their platform and this stuff is relatively “new” they get to do what they want (to a point) with little recourse. Its like if you build a YouTube channel and they strike down your video… you can try and appeal but at the end of the day you’re on their platform and play by their rules wether or not your video was justifiably taken down. And who is to determine justifiably? I don’t know that’s a topic for another day.
Books will ban you. Internally it’ll be called the fraud detection team or money laundering blah blah blah, and I’m sure they do a good amount of that… but in reality its the smart money detection team “don’t let us lose money at all costs if someone wants to win they can do it elsewhere.”
In the United States there is this thing called death, and there’s this other thing called taxes… both tend to happen to people. Speaking of taxes, there is another thing called taking the standard deduction, and there’s other things about qualifying as a “professional gambler”. Now what do all these “things” mean? I don’t know, I am not an expert but you should READ UP ON THIS before you start your lucrative arbing at scale.
Side note, Arbitrage exists in other markets as well, look at the trades SBF was putting on back in the day.

Here SBF is taking advantage of price differences between markets (buy low, sell high), his major challenges were essentially bridging the two markets. If people were willing to sell BTC for $10,000 and others willing to buy it for $15,000 in a single efficient market the prices would converge and the opportunity would be quickly eaten up. But because there was a lack of information, hurdles to cross in putting on the trade, and crypto being the wild west he was able to capitalize on this opportunity.

So if we could overcome all the issues above (I’m sure some people do) we could generate a nice profit. But it is NOT easy to do, NOT easy to remain undetected, and NOT something you could accomplish over night. You WILL need $ to get started, you WILL need to adapt to changes and you WILL run into issues as you scale.

Good luck.

NY Times Connections Solver PT. 1

Sun, 16 Jun 2024 00:00:00 +0000

Olive Oil:

During my daily morning ritual of waking up at 4:30 AM and immediately ingesting a ½ cup of [Trader Joes Extra Virgin Spanish Olive Oil] (Sponsored link), I decided I’d like to try and solve the NY Time’s Connections. So, with greasy fingers, lets begin.

This is part one, where I:

Review the rules of Connections
Gather historical Connections games into a structured dataset
Come up with some strategy for a program that’s somewhat capable of playing the game well

In part two, I’ll try and generate a solver for the game. I am not an expert on NLP, nor do I know much about word embeddings, nor do I know much about word embeddings given context, but I’m quite curious to learn. Here in part one, I’m going to nail out something semi-interesting and treat part two as a learning experience.

Overview/Definitions:

Connections is one of many NY Times games. It began in June 2023 and releases a puzzle daily. Outside of drinking straight oil, it’s one of my favorite things to do in the morning… so why not ruin it by figuring out a way to convert all historical games into a table and write SQL against it!

Connections is quite simple. You’re provided a 4x4 grid of word tiles. Words are grouped into sets of four based on a connecting category of varying difficulty. I’ll write about some patterns I discovered in the categories later.

Above is a picture containing the Connections game from 6/05/2024. Funny enough, this particular game so-happens to contain the most common word used across the history of connections which is Ring used, now, 11 times.

In this example, the four words “Sick”, “Nice”, “Cool”, and “Sweet” can be used to form the “Awesome!” category. This category was designated the “easiest” of all categories, although this definition differs from person to person as not everyone would associate Sick as a synonym for “Awesome!”. Personally, I associate sick with what happens after I drink my ½ cup of oil which, in my opinion, is pretty awesome… so, for me, it makes sense, but you do you. Rinse and repeat with the remaining 12 words, and that’s Connections.

Defining a few things to stay on the same page moving forward:

Category: A grouping of 4 words in a game
Words: A single tile in a game
Difficulty: How “Straightforward (easy)” or “Tricky (hard)” a category was in a game.

How does ChatGPT-4o Do?

I’ll start with a single days example from June 8th , 2024. This example aligns with a status quo Connections game, you’ll see why I say this in the analysis section.

**SPOILER ALERT** Below is the solution to the puzzle with the difficulty, category and words.

[Easiest] Grab One’s Attention: ABSORB, ENGROSS, HOLD, RIVET
[Easy] Document with Video: FILM, RECORD, SHOOT, TAPE
[Medium] ANAGRAMS: PASTEL, PETALS, PLEATS, STAPLE
[Hard] PRETTY ___: GOOD, PENNY, PLEASE, WOMAN

I made sure that the model is familiar with the game; however, at this point, I don’t offer it any previous connections data or answers to go off of.

Prompt:

"Solve today’s game, providing the category defining each word grouping.
Words: Pleats, Please, Hold, Record, Woman, Film, Good, Pastel, Penny,
Staple, Tape, Rivet, Absorb, Petals, Engross, Shoot."

Response:

Words related to movies: Shoot, Film, Tape, Record [CORRECT]
Words related to types of fasteners: Rivet, Staple, Pleats, Hold [WRONG], but I also considered this possibility
Words related to artistic or aesthetic qualities: Pastel, Penny, Woman, Good [WRONG] but close
Words related to engagement or involvement: Absorb, Please, Petals, Engross [WRONG]

When provided limited context or historical data, the LLM typically gets the two easier categories right (some cluster of closely related words). However, without any additional context, the model tends to miss the harder categories sometimes even letting mistakes spill back into the easier ones causing additional misses. This isn’t news to anybody, but illustrates that better prompts get better responses… if you’re going to go Oscar the Grouch mode (his birthday was June 1 BTW) expect a garbage in -> garbage out scenario.

Let’s provide an additional prompt denoting that the possibility of certain categories exists.

Prompt:

“These are incorrect, take into account that answers can involve grouping words
 that have similar meaning with the addition of a given prefix or suffix, homophones,
 and anagrams can all be at play to create a category. Please try again.”

Response:

Recording-related terms: Record, Tape, Film, Shoot [CORRECT]
Anagrams of ‘Staple’: Pleats, Pastel, Staple, Petals [CORRECT]
Words that can follow ‘Please’: Please, Hold, Absorb, Engross [WRONG]. The words almost form the easiest category, but it forces the prefix concept unnecessarily.
Terms associated with monetary or value: Penny, Good, Woman, Rivet (stretch, but possibly in terms of securing value) [WRONG] Switching “Rivet” and “Please” would yield a correct puzzle, but the categories are off significantly. It evens notes “Rivet” is a stretch.

While this attempt recognizes the anagram, the “Pretty ___ “ category seems to be fairly elusive for it and there is no idea of “Attention” in any category.

Final time… I provided a month’s worth of historical connections games with the categories, their associated difficulty and the four words. This, initially, caused it to regress significantly, providing an output that no longer followed the rules of the game. After re-orienting back to the task on hand, it butchered its previous progress and… umm… gave a response that added “Woman” to a category called “Objects”…

After providing thirty Connections puzzles, I found it was possible for the model to get them correct but only on easier days and after several prompts. If the model receives an extra hint, such as “I’m sure these three words together are in category “X”, but I don’t know the fourth word,” it usually manages to provide a correct answer.

There are probably better prompts, more prompts, or a curated LLM that could accurately solve most of these puzzles. There is certainly a good amount to explore in prompting an LLM and trying to get it to solve Connections.

Retrieving historical Connections games:

I figured a good place to start would be to create a table containing all historical connections games properly labeled with words, categories, difficulty, and date. This dataset would serve as the basis for understanding common patterns in Connections, as well as a clean set of labeled training data I could use for whatever solution I envisioned.

While the NY Times has a Wordle Archive, there is no such Connections archive. Luckily, there are a myriad of websites that host all historical connections games.

So, I LOCKED IN and spent ~5 days copying…

Nah… actually, I sent a couple cold emails to any address I could find on these sites asking if they would share a file containing the games and answers. No response. I did a quick search of Github and Reddit to see if I could uncover any leads, but no… ok sooooo web-scraping, Yay!

I wrote a quick script to scrape one of the historical Connections sites into a text file. It quickly loops through each month of connections data on the site, extracts the paragraphs with the games and writes them to a text file with a separator. I’m going to leave the URL out, but you can find the site quite easily if you look. After writing 335 connections games to the text file, I convert it to a csv and remove duplicates. Now, nobody needs to scrape. They can just steal from me.

This is probably a good time to point out that this whole process should be a DAG, but I dont care about robustness or repeatability right now so… shortcuts!

Analytics (Cleaning):

Cool, so now that we have our “Clean” CSV, we can copy it directly into a table using duckDB and explore all these historical games. With duckDB you can directly COPY a CSV from disk into a table in a database. Ten lines of code later, and we’ve got a table.

# duckdb.connect(dbname) creates a connection to a persistent database
conn = duckdb.connect("connections.db")

# Create DuckDB table
conn.sql(
"""
DROP TABLE IF EXISTS CONNECTIONS_DATA;
CREATE TABLE CONNECTIONS_DATA (
    GAME_DATE DATE,
    DIFFICULTY VARCHAR,
    CATEGORY VARCHAR,
    WORD VARCHAR
);
"""
)

# COPY CSV from disk
conn.sql(
f"""COPY CONNECTIONS_DATA FROM '{clean_csv_location}' (FORMAT CSV, HEADER);"""

In our new table:

Each game consists of 16 rows, one for each word
A category, on a particular day, can be described with four rows

I choose this because I didn’t want to deal with any unnesting logic for viewing individual words, which would be necessary if I collapsed a game into 4 rows and saved the words in an ARRAY/LIST or just LISTAGG’ed the strings together.

FYI, duckDB adopted the SQL standard of 1 based indexing a couple years ago.

You may have noticed I put “CLEAN” in quotes earlier. I’ve got some half-baked web-scraping and transform logic I cheffed up in ~ 1 hour, so I figured there was about a 1% chance I missed some stuff… and it turns out I did indeed miss some stuff.

Date-category combinations should have four rows. A count of rows (CNT) divided by four should return 1, and then, 1 mod 1 should equal zero. However, the below query returned six instances where CNT equaled 1.25. I discovered hyphenated words like “Yo-Yo” were written as two rows.

SELECT GAME_DATE
     , CATEGORY
     , COUNT(1) / 4::FLOAT AS CNT -- BECASUE THERE WILL BE 4 ROWS FOR EACH INSTANCE
FROM CONNECTIONS_DATA
GROUP BY 1,2
HAVING CNT % 1 <> 0
ORDER BY CNT DESC

Alright, probably good but let’s run some summary stats to be sure:

SELECT COUNT(1) AS TOTAL_WORDS
     , COUNT(1) / 4::FLOAT AS TOTAL_DISTINCT_CONNECTIONS
     , COUNT(1) / 16::FLOAT AS TOTAL_DISTINCT_DAYS
FROM CONNECTIONS_DATA

Oops, why is TOTAL_DISTINCT_DAYS 334.5?

I assumed every game was unique
There was some boiler plate text in the text file from the web scrape that was duplicated
So… why not remove rows in the text file that were duplicates of a previously seen row to clean it up?
Wrong! Connections reuses categories with identical words:
- August 30th, 2023, and May 30th, 2024: “Influence” category.
- January 11th, 2024, July 25th, 2023, and August 15th, 2023: “States of matter” category.

Instead of fixing this in the extract code, I just stripped them out of the table and called the clean_table funtion at the end of pipeline.py. Sweet, clean CSV, no quotes this time.

I also removed 4/1/2024, April Fools day, when all words were Emojis ¯\_(ツ)_/¯

Analytics (insights):

I hacked together a few queries in a Jupyter notebook style file, printing data frames to the console. Over the +/- 335 games in this dataset there were:

~3500 distinct words, not many words have spaces (fewer than 30 occurences), and most words are 4 to 5 letters.
Word length is consistent across difficulties on average 5.1 letters
Most commonly reused words: {RING:11, BALL:10, COPY:9, LEAD:9, WING:8, JACK:8, CUT:8, HEART:8}
Shortest words are individial letters like “X”, occuring only twelve times
Longest words include Peppermint Patty, Concentration, Mashed Potato, and Horsefeathers. Horsefeathers is synonymous with nonsense or rubbish (I could see myself yelling this in an argument)

Pivoting out difficulty by category type:

Unpacking a few things:

This is biased towards the “Hard” categories, describing ~70% of them. This is due to this category being pattern focused and based on challenging word play each day. The only way I’m going to be able to solve these is to reduce the degrees of freedom in selection or determine the pattern with high confidence.

Dunderscores are extremely common, describing ~40% of hard categories:
- ROCK ___: (Star, Candy, Bottom, Garden)
- ___ MOON: (Sailor, New, Blue, Harvest)

Homophones show up a few times, and I should probably just do a static check
for these against a database of English language homophones.

Anagrams should be checked similarly.

Prefix_String, Suffix_String, Minus, Added_Plus all denote occasions where the words are similar due to having some character(s) appended or removed.

WITH categories are a significant challenge. I’m not sure how to approach them yet.

Things With Trunks: (Cars, Elephants, Swimmers, Trees)
Tarot Carods With “THE”: (Fool, Lovers, Magician, Tower)

THINGS overlaps WITH 11 times in the data, so I could have included the intersection of the two: THINGS WITH ___. Instead, I’ve just pulled the intersections into the WITH category.
```
Things People Shake: (Hands, Maraca, Polaroid, Snowglobe)
Things to crack: (Egg, Knuckle, Smile, Window)
```
Medium is typically some niche knowledge like countries with red and white flags, weapons in the game Clue, or British cuisine. To some, this is akin to “Easy” if they are familiar with the terms. For the less familiar, these can be confounding and lead to misses.
Easiest/Easy difficulties are generally tightly related words without requiring a layer of context. However, they aren’t always so straightforward.
```
Connect: (Couple, Link, Tie, Join)
Attach with adhesive: (Adhere, Glue, Paste, Stick)
```

Filtering for the triple dunderscore categories was a bit tricky, with _ being reserved. In Redshift, I was able to do LIKE ‘%\\\\\_\_\_%’ , but this doesn’t seem to work in duckDB. Instead, I went with some length based logic checking for at least a difference of 2 characters once stripping them out:

LENGTH(CATEGORY) - LENGTH(REPLACE(CATEGORY, '\_', '')) >= 2

Final thoughts on a solution:

One advantage of this problem is that its surface area decreases exponentially with the reduction in the size of the initial set. Something I never explicitly provided Chat-GPT was information regarding which categories it had gotten right and what was wrong. Allowing it to build on top of past decisions, and reducing the size of the problem.

For combinations, the formula: C(n,r)=n!/r!(n-r)!

Evaluates the number of ways a sample of “r” elements can be obtained from a larger set of “n” distinguishable objects where order does not matter (unlike permutations) and repetitions are not allowed.

At the beginning of any Connections game, a player will choose 4 words from a set of 16 yielding 1820 possible combinations.

C(16,4)=16!/4!(16-4) = 1820

As categories are found and labeled as correct the set is reduced: 16->12->8->4

C(12,4)=12!/4!(12-4) = 495
C(8,4)=8!/4!(8-4) = 70
C(4,4)=4!/4!(4-4) = 1

Like I mentioned in the intro, I find NLP interesting, but I am by no means an expert. After my manual checks to catch some of the tricky categories in Connections, I’m not entirely sure what an optimal framework looks like for grouping four words and iterating over a solution:

Maybe I don’t even need to perform these manual checks if I provide the right context to the correct algorithm?
I’ve noted “confidence” several times in the diagram, but I’m not sure how to objectively arrive at this or if I’ll need to normalize “confidence” across different categories/difficulties?
I’ll want to operate on a reduced set of words given that a category has correctly been identified, but I’m not sure if I should throw out previous work to avoid any noise created by the larger set of words?
I’ll need to figure out how to handle Nouns, single letters, and words with spaces (Maybe this requires injecting a word classification layer between the raw set of words and the cached check for specific categories)
Not sure I’ll be able to get this category: Reads the Same Rotated 180 degrees: 96, MOW, NOON, SIS

Anyway, that’s part one. Who knows what part two will be, stay greasy!