Sam's Blog

A Review of Hex

Sat, 04 Apr 2026 00:00:00 +0000

Hex Review

It’s Easter Sunday. After a long day of chocolate mogging and stealing eggs from small children, I’ve decided to check out Hex which calls itself the “The AI Analytics Platform for your whole team”. I’m going to toss a small csv into it, if it does well I might hit it with a 20GB Parquet file.

I’m not going to connect a data warehouse to it (This is wrong, I ended up connecting to MotherDuck), I’m going to treat this as the classic “There’s a dashboard, but I need to download the underlying data and re-manipulate”

I signed up for a free trial and started off with a prompt and some data.

I have not read a single guide, looked at any template or read any docs
I have no idea how Hex works on the backend
I’m just going to figure it out as I go and bumble my way through

The data

I gave it a ~740 line csv with historical March Madness data

Example:

Season	Round	Team	Odds	Result
2026	1	TCU	120	1
2026	1	Ohio State	-142	0
2026	1	Troy	650	0
2026	1	Nebraska	-1000	1

The Prompt

I’ve added the prompt below but the TLDR is we want to evaluate the “accuracy” of pre-game odds for March Madness from a CSV containing 5 years of historical data. The prompt isn’t great, and I didn’t even provide extra contents on what I was working on.

Click to view the entire prompt word for word

I have captured historical odds on NCAA march madness tournaments. The CSV contains data with the year, round, team, odds (in american odds) and result (1 for win 0 for loss). The data is not perfectly clean, some of the odds formatting may be wrong or missing, not all years have the correct number of games.

I’d like to visualize the accuracy of these predictions on a discrete vent over the years have been. Use something like brier scores and/or log loss to do this. If you have a better way feel free to do that as well. Split this by round as well as there are many blowouts or “easy” to call games in the beginning of the tournament.

I’d also like to measure the competitiveness of these games over the years. Have the odds gotten closer together on avg over the years, how is this changes if we look by round?

Dropping in

After issuing the prompt and attaching the csv, I immediately found myself in a “project” or, more descriptively, a “notebook style environment”. I’m prompted to give my project a name and watch as some LLM gets to work on the right side of my screen. I can give the project a status and category, which is nice as I could see these things getting unwieldy pretty fast if you had 20-30 people doing “analysis”.

The first cell

There’s a collapsible cell with a SELECT * of my data displayed as a table. The column structure is there, and I can actually create a new column with a formula that can reference existing columns if I want. If you’ve read my blog you’ll know I don’t like SELECT *, but this doesn’t seem to be auto-materialized for all new projects and this is a csv so no harm no foul.

I can re-write the SQL in the cell to produce a new data-frame, but I can also manually filter it via the UI. Applying the filter via the UI updates the “relation” (I don’t really know what to call this yet), and it also looks like it updates visualizations/cells that depend on the relation (The LLM seems has been hacking away below).

Importantly, comments can be added, responded, reacted, and resolved in a cell. Maybe more importantly? Cells can be triggered in DAG fashion with downstream cells taking dependencies. This is good. It’s kind of feeling like Marimo which was recently acquired by CoreWeave. Although Hex is not open source, and you don’t really own your notebooks they live in cloud land… this is actually wrong, you can import/export notebooks as .ipynb or their Hex yaml situation.

Speaking of DAGs there’s this graph view which is nice. the graph shows me that Hex has decided to SELECT * from my csv, written some pandas transformations, and then created three charts. I can’t figure out how to pop the graph view out (Full screen) or change the size… thus the blurry image. In general, every square inch of my screen is packed with features.

The LLM & Context

Moving over to the right pane there’s a chat window with an LLM. The chat window can host multiple threads with conversations >90 days being deleted. Unlike the graph view, I can make this pane basically eat the entire screen.

I quickly find out who’s baking this cookie (claude), but it won’t share system prompts/instructions. After a bit of back and forth, Claude tells me I should be using Guides which exists inside “context studio”, which seems to be exactly how context is managed in Hex. This made me a little nervous at first as it looked as though you’d need to embed/maintain context in Hex, but they allow connections from outside sources (i.e. context in github updated via github actions) If relevant context is haphazardly stored in a bunch of people’s brains this is still of little use though.

But wtf is the compute here

So now I’m curious how this SQL is getting executed. I ask in my little Claude thread and the weights and biases tell me “it’s querying the uploaded OddsMM.csv file directly using the embedded DuckDB engine”. DuckDB… tight. I can confirm this going into the SQL Cells Introduction. It also seems that you can parameterize SQL using Jinja which is a huge win in my book. It also looks like it integrates well with dbt Cloud which is solid. DBT or something like it is pretty simply how modern data teams should be authoring transformations, there’s just not much of a question at this point. I wouldn’t want to be orchestrating transforms inside of Hex though. I’d be doing this in a data warehouse, orchestrating with Dagster and managing transformations with DBT and only after that connecting to Hex.

Opening the “Environment” on the left tool box shows I’m using a Medium instance with 8GB of memory and 4CPUs. They have a beefy 128GB instance, but I’m getting the feeling that if you need this inside of Hex, you may also need a better data model (Yes, dude who’s visualizing some ridiculous telematics data, feel free to rip into me for that comment). I’ve also got access to python 3.9 -> 3.12 and a bunch of python packages come pre-installed (no polars… which is big sad), but it seems you can just add to your venv with uv.

Other staff in the left most toolbox

There’s a full revision history with version checkpoints and a large space dedicated to a search feature which seems to be able to grep through your current workspace. The variables section stores all the built in variables as well as anything we create in our python code and dataframes. There’s also a data browser which acts as a functional data catalogue for assets inside the workspace. Opening up the column types inferred from my csv: Team & Odds are labelled as having type “object”… which is a bit weird, but I guess they are strings… so arrays… so… objects? Idk, I’m stretching here for why that doesn’t say “varchar()”.

Back in the middle

Coming back to center stage with our data/assets, there’s a nice little chart telling us what is taking the longest inside of our “DAG”.

Cool feature to have rather than setting up timing, %%time on some sprawling notebook.

Continuing on, Claude has written a little pandas to create some new data-frames based on the aggregations I requested. These data-frames are available for view in the data browser. These have become materialized relations (I’m pretty sure) that I can query with SQL in a new cell and are also referenced in the visualizations.

Visualizations

Speaking of visualizations, Hex seems to be using Vega after they performed some sort of acqui-hire of the VegaFusion maintainer. The visualizations seem fine and responsive. There’s a suite of UI tools to edit them a bit (both style and substance). You get access to the generated SQL, which seems to build the relation required for each one.

From the screenshot you can see accuracy_by_season_round is referenced which was created by some pandas in the cell above and exposed in the data browser. I’d like to have full control of my charts as code… and it seems like I can easily do that. I created a cell that references that same data-frame, but creates a basic chart with plotly and it works.

Make an… App?

So it seems like the main other thing you can do is switch out of notebook mode and make an “app”. I turned my analysis into an app (If you read this a while from now this will most likely not be available… sorry not sorry) which seems to expose the visualizations at an endpoint, and they can refresh on a schedule. You can export it, take snapshots, turn it into a presentation. So the apps are basically interactive dashboards that you might expose to an end user rather than tossing them a notebook and saying “run this and look at cell 8”.

These apps can be added to a “library” that contains published knowledge that can be “endorsed”. I don’t know how this solves for people just endorsing a bunch of stuff. I don’t think there’s a re-endorsement system where you have to be like “yeah this is still good”… because all of this manual tagging typically loses meaning. It’s like people flagging workloads as “business critical” to get access to resources faster… it’s a short time before everything is “business critical”. Some kind of continuous mechanism to confirm endorsement might be nice (Or just annoying enough that people would disregard it… its a hard problem admittedly).

Competitors

It looks like DeepNote offers something similar, but I’ve never used it. It seems like every data warehouse in the sky has something like this. Posthog’s platform is similar but more focused on web analytics. ClickHouse’s cloud dw offers a SQL notebook and dashboard experience. Marimo is a great notebook based analytics tool that’s open source (recently CoreWeave is in the picture).

This is in NO way an exhaustive list of the space, but I have to say… Hex is very complete. They are clearly thinking hard about how to nail “last mile” analytics and make integration upstream simple. If I was sitting in a small shop and we needed analysts/data science to produce some standard reports, version them over time, and sprinkle LLMs into the workflow I’m probably reaching for Hex.

I’m seriously trying to figure out the value proposition of something like Looker, PowerBI, Tableau, etc… as opposed to Hex.

People either play around with the filters a bit or they actually just want the underlying data for the dashboard… which Hex quickly provides access to and lineage for.

Increasing the data size

I had some larger parquet files from my Prediction Markets article that I thought it’s be a good test to see if Hex could quickly reproduce a few of the charts. They are big enough that I can’t just drop them into the UI.

The Kalshi Markets dataset is ~2 GB & 30 million rows with data on all historical markets
The Kalshi Trades dataset is ~11 GB & 203 million rows with data on every single trade to ever occur on Kalshi

I loaded my local copies to MotherDuck by establishing a connection to a new database I set up.

Then it took about 30 seconds to give Hex a Read/Write access token, and I could start writing SQL against my tables.

I gave it a pitiful prompt/context and in under 5 minutes it produced some charts that you can view here. Again, to do this well with the data in under 5 minutes with limited context is quite impressive. You can directly compare these to a few charts in the actual article… and they get the point across, I’m sure that with a bit of effort I could get these one’s inside Hex to look quite nice.

The Prompt

Please use the Kalshi data assets (markets & trades) to derive a series of visualizations.

Show me weekly trade volume in a bar chart
Show me contracts per trade month over month avg and median
Show me a stacked bar chart displaying total trade volume by month with the bars segmented into the implied probability of each trade in cohorts of 0-20%, 20-40%, 40-60%, 60-80%, and 80-100%

Overall Thoughts

I’m impressed. It’s a well thought out product. It’s definitely much more than just a “Claude wrapper”, and they are clearly trying to integrate things that modern data teams need. For a quick proof of concept on a personal project I could easily see myself tossing some data in here with some prompting/context as long as I had access to the elevated features of a “Team Plan”. I do think this is geared more toward enterprise, as it’s way overkill for a hobby project, and I can pretty much get all the features I’d want out of this with Claude Code, Marimo, and a little know-how. But they know this and can cater to the “Yeah we don’t want to maintain that but want that functionality” crowd.

I’m always a little weary of notebook development for large teams. How do you stop people from building data assets in parallel that describe the “same” thing but end up with a different result. This is a perennial problem that Hex seems to be great at reducing in comparison to people hand waiving Jupyter notebooks and pivot tables. And holy FUCK can we please realize that there are better things out their than just isolated local notebooks performing data wrangling, visualization, and presentation… they do NOT scale… use Hex.

I’m trying to think of other headwinds here. I mean, this is a lot of product. I’m familiar with these concepts, but if the ideal state is to drop this in front of 20 PMs and say “Hey you can make your own dashboards and data assets now” there would be a significant amount of churn/up-skilling where as Data Science could probably get on board with this very quickly. You still need to solve for upstream, Hex isn’t going to fix your broken medallion architecture and redundant/stale tables. Notebooks are a classic foot gun, if people don’t use the tools that Hex provides (DBT integration, jinja templating, version control, data catalogue, etc…) you could easily run into spaghetti mode just as anything else. Hex can lead your team to the data lake, but they can’t make you…

What I learned this week 04-04-2026

Sat, 04 Apr 2026 00:00:00 +0000

What I learned this week

Software

Python cad library

Build a GPU

Business/Finance

Interesting Links

Pallet camper

Rescue fraud

Fun fusion overview

Poker lecture at Harvard

Math/Stats

Travel

Other

What I learned this week 03-28-2026

Sat, 28 Mar 2026 00:00:00 +0000

What I learned this week

Software

Alan M. Turing allowed himself to be drawn into the discussion of the question whether computers can think. The question is just as relevant and just as meaningful as the question whether submarines can swim.

The goat

Copy all rendered html copy(document.documentElement.outerHTML)

Common Design Diagram Mistakes

Eggs… and high token cost

Business/Finance

Interesting Links

John Rawls

18p hard lemonade

Hotels but only during the day

Building a garden

Stop over paying for vanilla extract

Math/Stats

Travel

Other

Successfully dehydrated some fruit this week.

https://www.freshoffthegrid.com/diy-fruit-leathers/
https://eatsleepwild.com/homemade-dehydrated-backpacking-meals/
- https://www.amazon.com/mylar-bags/s?k=mylar+bags
- 100PCS Silica Gel Packets

What I learned this week 03-21-2026

Sat, 21 Mar 2026 00:00:00 +0000

What I learned this week

Software

Voltage glitching

7x better because “composition”

Bellman equation

Business/Finance

Suno AI music…

Autonomous driving might have the biggest positive impact on the world this decade

Interesting Links

Plant watering:

Making things shitty

Billiken

One Trillion brackets

Math/Stats

Travel

Other

“the innovator has for enemies all those who have done well under the old conditions, and lukewarm defenders in those who may do well under the new.”

What I learned this week 03-14-2026

Sat, 14 Mar 2026 00:00:00 +0000

What I learned this week

Software

SQLite skip scan

Dead internet arrival

Namespace prevents bucket squatting

Business/Finance

Interesting Links

Wow, 21 years off grid

Clacker news

Math/Stats

Travel

Other

Doom

What I learned this week 03-07-2026

Sat, 07 Mar 2026 00:00:00 +0000

What I learned this week

Software

I did not know that appending a shell command with & runs it asynchronously. You can view running things with jobs.

nc ip port to send a message to a port, nc -lv port to listen to it.

yes just pipes y to stdout over and over

pstree for viewing process tree on Linux (all starts from systemd)

powermetrics is useful for cpu/process info.

TinyPNG’s

Conflict free replicated data types

Visual walkthrough of diffusion modelsxs

Good Postgres blog

Business/Finance

Rent seeking in science

“AI Pays for itself” first prompt: connect wallet…

Price controls suppress the mechanism that prevents chaos, and the resulting chaos invites still more control, creating an escalating cycle the price system would have prevented from starting.

Culture is those behaviors that employees believe will be rewarded.

Interesting Links

Spinnaker Cool place ot visit and make chocolate

Cig app with a great UI

GK chesterton

The whispering earring

Math/Stats

Travel

Other

Wild Shia LaBeouf Interview

The 2026 Barkley

Last day at AMXL

Fri, 06 Mar 2026 00:00:00 +0000

Leaving AMXL after 5 years

What I learned from ~5 years at Amazon all of which were spent at an organization called AMXL. I’m moving to a new organization within Amazon, and I felt like it was a good time to reflect.

Reflecting on people

Being negative and short with people is almost always detrimental to both you and them. If this is occurring figure out why and fix it asap.

Most requests will be labeled as “urgent”, very few will actually be urgent.

One of the best things a manager can do is help allocate your time well. There will be meetings you never sit in that have an outsized impact on what is worked on and how things are measured, especially long term.

General Reflections

Good documentation is gold. Building in systems to keep it updated over time is gold x2.

Be cautious of your time… 5 years went very quickly, and I don’t see the train slowing down. You also have no idea where the end of the tracks are, enjoy it.

Clarifying questions are powerful & necessary, make sure as much context/information is made available before leaping into action.

It’s very important to understand inputs/outputs. People like to drop you into the middle of a complex problem/workflow which quickly becomes an unsolvable context trap if you don’t understand the start and the end. Also be careful of the XY problem.

Don’t allow my self worth to be entirely defined by work, I need other outlets.

A good metaphor is underrated. The ability to produce a good one quickly is even more underrated.

You should be failing 1-2x per year. This does not need to be out right crash and burn, but it should be something like “yeah we hit the limit of that system, looking back we should have done X, but now we NEED to do X”. This is how you learn, there is no book for this.

Data Reflections

Under the right stressors, people will treat a data orchestration framework as cron.

People prefer a wide table rather than creating a data model. It’s important to be careful about this, See post on wide tables

Always assume that people have no understanding of statistics. Always assume your chart/graphic/UI is too complicated. This is NOT a knock on people. You are inherently waist deep in a problem that most others are only weakly aware of, if you can’t boil it down to something graspable that they can care about, that is your own fault.

What I learned this week 02-28-2026

Sat, 28 Feb 2026 00:00:00 +0000

What I learned this week

Software

One step from magic

the bottleneck in AI-assisted development isn’t the quality of your ideas - it’s the quality of your feedback loops.

Business/Finance

Why isn’t GDP growing faster?????

Insider trading pt 1000

Insider trading part 1001

Levered single stock etfs

Interesting Links

The best college hockey player

My new favorite hickey team name

Obree

Math/Stats

Travel

Other

Distill scientific papers

wild way to pay rent

What I learned this week 02-21-2026

Sat, 21 Feb 2026 00:00:00 +0000

What I learned this week

Software

your org rarely has good ideas. ideas being expensive to implement was actually helping

Sony pictures hack

The “AI makes people boring.” Thesis

Business/Finance

Interesting Links

Belarus likes to drink

A new word I learned

Ski ballet

AI proof business? 2 Rubiks Cubes 2 Cig 2 Beer

Lasers

Math/Stats

Jensen’s inequality, I’d like to write an article about this one

Travel

Other

Penis injections…

What I learned this week 02-14-2026

Sat, 14 Feb 2026 00:00:00 +0000

What I learned this week

Software

Nice overview of PolyMarket activity

Another interesting “deploy software here” business

Amazing website, good content as well - how is data stored

Fast LLM inference, we’re going to get model on a chip soon

Business/Finance

The Economic Impact of Mass Deportations

Interesting Links

Paddy lost

DIY noise canceling headphones

Math/Stats

Travel

Other

I feel like long Fishing line makes for a good analogy for complexity as project grows in size (both scope # or people). If you have a foot of fishing line it’s easy to deal with you know where both ends are at all times. If you have 250 yards, even simple issues can become infinitely complex and you just end up cutting line and starting over.

What I learned this week 02-07-2026

Sat, 07 Feb 2026 00:00:00 +0000

What I learned this week

Software

3D engine

The Website is down

Nobody knows how the whole system works

Business/Finance

Good overview of sub penny rule

Physical products are… difficult

Oops

Interesting Links

Vacant land fraud

Math/Stats

Travel

Other

The Tenkara bum blog

Places to fish:

Willow springs creek hike behind the dam
Canyon creek, passed woods canyon
Silver creek (upper) show low

Scaling markets with non-human operators

Sun, 01 Feb 2026 00:00:00 +0000

I’m jotting this down quick so apologies for run-ons and shit that doesn’t read well.

Scaling markets with non-human operators

There’s this common adage that you should do things that scale beyond yourself.

Doing things that are limited by your capacity to complete them and garner little long term value moving forward don’t “scale”.

Uber, autonomous trucking, and scaling

If you drive for uber your earning potential depends on your willingness to take more rides and how you act during those rides. There’s probably some momentum/multiplicative factor where if you have close to a 5 star rating and a ton of rides you become more desirable in the market and you can derive outsize returns. However, this is probably limited by the fact that most people doing uber are also aiming for this status that typically just requires time and a decent personality to get.

So NO, Uber driving doesn’t scale. What might scale is if you have a fleet of autonomous vehicles that can constantly give rides for you.

You pay a large fixed cost for the vehicles and the software
You pay a variable cost for maintenance, software updates, etc…

This begins to feel like a vending machine type business. Something similar could be said for the trucking industry where you could direct your autonomous fleet by winning bids for loads in the market and then sending a truck to execute them. Which is kind of how it already works except people drive the trucks and they bring added complications. Although to be fair, autonomous systems also bring added complications but they are usually preferable to the ones people bring.

Today I learned that the FMCSA established that the legal BAC for a commercial motor vehicle operator is .04%… which is fucking insane. It should be 0.00000000%. Human beings, distracted humans beings, in anyway inebriated while they drive a ~20 Ton vehicle is INSANE.

I see a few major problems with this.

As autonomous vehicles become more pervasive what’s stopping anyone from giving up the excess capacity of their vehicle to a market of people willing to pay for it. You could say “I am not using this car from 8pm to 8am” and then allow users to bid for its use all night… this is literally Uber but without an individual sacrificing their time driving (still sacrificing the maintenance cost of the miles and the vomit in the back seat).
The other thing is that large companies like google will control fleets of these vehicles. It may become so cheap to rent an autonomous vehicle that vehicle ownership becomes irrational financially. Cars that don’t have to worry about humans driving them can be completely retrofitted for leisure, camping, etc… and you can leverage some pool of supply for a trip at such a low cost and such high liquidity (no latency in acquiring capacity) that it would be silly to buy a car that is only good at one particular thing. In this way, you could push the burden of insurance, maintenance, liability, etc… to some larger supplier (although they’d probably try and pass that cost back to you, I don’t know exactly how that works out). I hae no idea if it’s right to push transportation supply to be under control of a single entity… that also sounds… bad?

Humans in an LLM inference market

Ok that was a little nuts but let me reel this back in to the central thesis of “things that scale beyond yourself”. Something that struck me recently was the clawd bot buys a car thing. This is a rough example, but we are early days. What this shows as that traditional markets are going to break down under the pressure of non-human relations. As an individual, negotiating a car takes time or you can just have a system negotiate for you. We have companies that try to do scale this like Carvana or Cars.com, but they’ll need to pivot or risk being entirely exhumed by the ability for a single person to mass produce enquiries/negotiations and get the “best price”. Same ting goes for travel sites.

You could never compete against a high frequency trading firm by tapping away from Robinhood on your phone. In that instant that you bought $BBW is it possible that there may have been a slightly better price available… yes, but you paid some tiny tax to some firm that helps make the market for $BBW for the convenience of the liquidity and your sanity. I don’t know if I’m illustrating this well, but I wonder if humans attempting to participate in less liquid, higher friction markets online like ebay, facebook marketplace, car shopping, travel, etc… will struggle to compete for goods/services amongst bots. This markets were gate kept because you kind of needed to be a person to navigate them, they are built for people and used by people… but what if they aren’t or some fraction of the market becomes no longer human.

Scaling oneself against a larger organization/entity

On a slightly different note, think about this when it comes to battling insurance. You can literally attack this seemingly unassailable behemoth that relies on your inability to take the time, read the fine print, and fight to make money. Something as arcane as this will fold under the constant/immense pressure of the arguments/enquiries that can be automated against it.

HOWEVER, this may be prevented in the short term by these companies lobbying that automated communication for XYZ not be allowed, in which case there will be massive gray areas and a whole host of new problems/new inventions.

The other thing that could happen is that they build an automated defensive shield against these enquiries. In which case we have LLM on LLM inference to the bottom and the winner is whoever you are paying for inference. This is worrisome for MANY reasons, its probably bad for the climate to be doing all this inference for no fucking reason, and it will be nearly impossible to sift through it all and paralyze decision making (very bad).

I’m racking my brain for some middle ground. Maybe it’s higher cost for communication to make mass spam prohibitive? Maybe there’s human in the loop reality check points?

This is going to be so crazy to watch play out.

What I learned this week 02-01-2026

Sun, 01 Feb 2026 00:00:00 +0000

What I learned this week

Software

SQL >

Business/Finance

SUbstack tv

Gambling on…. wait… gambling

TSMC pushing risk to hyper scalers

Interesting Links

On the saw, this dude is the CEO of Obsidian

First make me care

Removal of security theater

Slop Markets in everything

Math/Stats

Travel

Other

What I learned this week 01-24-2026

Sat, 24 Jan 2026 00:00:00 +0000

What I learned this week

Software

Being useful

Business/Finance

Stupid title but interesting

Interesting Links

“Put differently, there were fewer births in China in 2025 than in 1776”

I got some of these spicy peppers from my neighbor

Evil exists primarily in the eye of the beholder, especially in the eye of the victim.

Math/Stats

Travel

Other

What I learned this week 01-17-2026

Sat, 17 Jan 2026 00:00:00 +0000

What I learned this week

Software

Need to try out sprites

Tailscale remote into device

Prepared statement, the anti-sql injection

The over engineered data pipeline

Business/Finance

The Blog that made me start writing

Pair trade

Iron Condor

Interesting Links

Existential choice paralysis

Math/Stats

Travel

Other

What I learned this week 01-10-2026

Sat, 10 Jan 2026 00:00:00 +0000

What I learned this week

Software

Planet scale

Andy’s annual review

Modern column storage

Business/Finance

Interesting thoughts about modern work

Interesting Links

FogCam

18000 water cups

Math/Stats

Travel

Traveling »»>

Other

The Crazy Alberto

The real issue people miss is that code isn’t an asset, it’s a liability. Every line of code is a potential bug or something that will slow down your future development. Experienced devs know to weight trade offs of more code vs maintainability. Whereas just using LLMs to shovel middling quality code as fast as you can misses those nuances and will lead to huge problems in the long run.

What I learned this week 01-03-2026

Sat, 03 Jan 2026 00:00:00 +0000

What I learned this week

Software

Setuid

Business/Finance

Accounting explained in graphs

Interesting Links

A Modern Movie Ranking

Cool site for learning knots

Ghost guns

Math/Stats

Travel

Other

New Years Resolutions:

Become conversationally proficient in Spanish
Be able to fish for myself both shore fishing and stream fishing (not fly fishing)
Continue to Travel as much as possible

Puerto Peñasco

Thu, 01 Jan 2026 00:00:00 +0000

About

Trip to Puerto Peñasco with Kenta, Kenta’s dad (John) and Tristan.

12/26:

Played some poker the night before and got up around 7:20. Ally woke me up. Packed up the last bit of my stuff and had Ally drive me over to Kenta’s. Met up with Kenta, his dad John, and Tristan. Piled in the four-runner and headed to Mexico.

It was about 4 hours to get to rocky point. We got the rods set up and went to the pier. The pier was right next to wrecked on the reef. It was basically a ton of massive concrete blocks haphazardly stacked on top of each other.

We cast for a bit, I lost my lure and we didn’t catch anything.

We drove over to el pollo Lucas and had amazing roasted chicken with tortillas.

After that, we drove through the touristy part of rocky point which I remembered from when I was a little kid (have a photo with Alec there).

We drove to our resort and brought our stuff in. We grabbed the rods and headed down to fish directly in front of our place which was right on the beach.

We didn’t catch anything but Kenta caught a pelican. I practiced casting, lost a lure, and successfully tied a new one back on. So that was good. I’m definitely a bit embarrassed at being so bad at fishing.

As the sun started setting we went back up stairs and then headed out for dinner. We went to some taco place and got aguachiles, quesadilla, and tacos. After that, we want to Al capones and watched a crazy 49ers vs bears game and I had a guava beer.

We went home, chilled, watched despicable me 3 in Spanish and went to bed.

12/27:

We got up around 7:30 and got out of the place around 8:30. The water wasn’t working in our resort which delayed us a bit.

We went to Reggie’s and got some breakfast burritos which were a little dry but pretty good if you added some salsa.

We took the highway three, passed las conchas to the second estuary “la pinta”. The highway is a tiny two lane road that follows the hills underneath it. We were cruising along and it kind of felt like a roller coaster at 70/80 mph.

To get to the second estuary we turned off the highway onto a dirt road marked by an abandoned house and mile marker 32. Taking the road for five minutes we drove passed another shack looking house with a green tractor. We were also met by a dog that got up real close to our car and barked as we entered.

John drove up the beach a bit until he felt the sand was a bit loose and then parked the car to avoid getting stuck. This would have been a problem as we were completely alone. This was a popular spot where normally you could see other people fishing, but today being a Monday morning close to new years it was completely empty.

The morning was a blast. I got a handle on casting the spin rod and closing the bail fast and keeping the tip up to avoid getting my lure stuck. I did get it stuck early on and John and I made almost every mistake possible getting a new one on there it was hilarious (not really) because I was useless and keeping John away from catching fish.

We then instantly started catching fish. John caught a corbina, a small grouper, and a leather jack. Tristan caught similar fish and so did Kenta. I caught one good size corbina when John gave me his rod that had some fly fish feathers obscuring a hook that the fish seemed to really like.

The tide was going out all morning so while we were able to cast farther out the fish stopped biting. Seems like knowing where the fish are going to be is a complicated ordeal… but we had a better chance as the tide was coming in and just after it started going out.

We also saw a dorsal fin pop through the water which seemed to be a dolphin but kind of looked like a shark. This was a problem as it was eating all our fish!

After about 1 o’clock, I was basically just practicing casting, nothing was biting. John even brought out his fly rod but it was a bit too windy. Around 1 we called it and headed to Tortas Siria.

We walked in with our water shoes on and just ruined their floor with mud hahaha. We asked for a mop to clean it but the lady told us it was no problem. We got tortas and John accidentally got something off the breakfast menu. I got a torta millenesa which was good. Kenta got some prepared drink which we think was made with local water but he seems fine, it was kind of funny because he got a bit nervous at the time.

After lunch we drove over to malecon or the tourist area that I remember as a kid and walked around. It’s kind of annoying just a bunch of people asking you to buy useless shit. A bunch of guys asked if we wanted to go fishing. I told them we had been fishing all day. He then said “what about finger fishing!?”. After walking around a bit we got some ice cream. And drove back to the resort.

We turned on Scarface. Tristan and Kenta went down to the hot tub which was apparently not hot, and I talked with John.

We decided to fish/walk down the beach from where we were staying and go to a bar for some beers. It was rocky and difficult to catch much with the large waves, but we cast out a bit and got to the bar around 6. We watched this guy at the beach get his stupid jeep unstuck and the cops ran out to give him a ticket. We had a few beers and Kenta, Tristan, and I had mango margaritas.

We walked back up the road to the car to get dinner.

For dinner we drove to Diego’s tiki bar. Which was back up toward la cholla. We got shrimp tacos, coconut shrimp, bacon wrapped shrimp, etc… there was a theme! Our waiter was nice and Tristan is basically fluent at Spanish at this point. We got his backstory and he told us about puerto Peñasco and Mexico, how most places are safe, that he plays soccer and the restaurants all play soccer against each other. He also let us know how many ladies he was getting.

We were super full after dinner. We headed over to a super market to get some breakfast for tomorrow as we plan on getting an early start. We got some beef jerky and conchas… breakfast of champions.

Watched a bit of Scarface and then showered/headed to bed.

12/28:

Got up around 7, a bit before Kenta/tristan but after Kenta’s dad who had already gone down and prepared the gear.

Had some old coffee and a bit of a concha for breakfast. Started having a headache about an hour after I woke up. It was the usual sort, pain in the temple and lasts for 4-5 hours. I think due to all the salt/dehydration from the day before.

Got packed up and then we drove over to the second estuary. Started fishing and I caught one fish. Wasn’t feeling great and my stomach was starting to churn.

I saw the splashing of some fish about 25 yards off shore and attempted to cast my lure out right in the middle of that area. I cast shittly and left it about 10 yards short of where I wanted it. I quickly reeled it in and sent it out again and hit the spot perfectly… almost too far. I started reeling it in and immediately got a fish on. It was a pretty good sized corbina, I didn’t have the tension right so I was struggling to bring it in. Tristan came over and increased the tension a bit, and I managed to bring the fish in and gently drag it ashore.

The fish was hooked twice in the mouth and once through its right fin. They were pretty easy to get out and I quickly let it back into the water.

At this point, I gave John my rod (it was better than the one he was using as he wanted us to catch fish) and watched for an hour or so. Around 11:30 the wind picked up and we called it a day. We got the gear packed up and drove back to el pollo Lucas for lunch.

I wasn’t feeling great so I didn’t eat, but got some tortillas to bring home. After lunch, we got some gas and headed back to the border. We drove passed all the people trying wash our car and sell us puppies to have our passports checked. Tristan had an emergency passport and the agent said at first he came up as an interpol person of interest before he manually typed in his info and found that to be wrong. We drove through why and ajo before one more border patrol checkpoint look at us and let us through.

Driving back from Gila bend, John told us about his skateboarding and snowboarding days. As well as Living in Hawaii and Japan. He also talked a lot about his work at perforce, enforcing abuse of rogue wave’s software licenses (ALWAYS BE CAREFUL WHAT SOFTWARE YOU DEPEND ON AND READ LICENSE AGREEMENTS)

Summary:

Amazing trip, I need to get better at fishing and knot tying. Puerto peñasco is an amazing resource within 4 hours drive of where I live that I had underrated before. Spending time in nature with friends is one of my favorite things. I really enjoy speaking Spanish and generally like speaking local languages it’s so much fun.

Whiskey that John likes: Weller, Eagle rare

Rod: Phenix redeye

Value at the limit

Thu, 01 Jan 2026 00:00:00 +0000

An unbounded supply of Chess

I used to play chess only on lichess everyday. Lichess is great, it’s free, open source, and I’ve been donating to the site monthly for years.

I prefer Lichess to Chess.com in nearly everyway, but, recently, I’ve been playing/practicing on chess.com. Why? Because it offers a stopping point.

Chess offers me a slippery slope. On one hand it’s an outlet for deep thought and creativity and on the other its an aggravating time suck that leaves me infuriated with little improvement in my abilities to show for it. Often times, I’ll play blitz or bullet chess for hours. I play these time controls poorly and find myself sliding into the “just one more game mentality”.

Lichess offers me unlimited puzzles, variants, and studies. I can play and practice as much as I want without ever having an advertisement or banner to “subscribe” thrown in my face.

On Chess.com I only get to solve 4-5 puzzles a day before reaching a limit and being told I need to “upgrade to premium”. Unless I create a new account, I wont be able to solve another puzzle on the site for 24 hours… this is exactly what I need.

I need a stopping point. I need someone to tell me that I’ve done enough. Scarce supply incentivizes me to take my time and leverage each puzzle to the maximum.

Being restricted is a feature

Unbounded supply is dangerous… it feels valuable, but requires a high level of self discipline. The same self discipline that is chipped away at from the time you wake up to the time you go to bed.

Restriction is a feature. I’m enabled to gamble on nearly anything, eat/smoke/drink as much as I want, I can watch sports for hours a day on all of my devices, I can scroll short form videos until my brain is mush, play video games until my fingers are numb, and read information (both true and false) until I’ve developed myopia… in the kind of abundance we live in today, restriction is a feature. Being told to STOP is a feature.

Though not realizing it, Chess.com’s subscription model has offered me a small respite from the onslaught of abundance. The restriction of my actions keeps me coming back to solve my four puzzles everyday. For that, I’m grateful.

What I learned this week 12-27-2025

Sat, 27 Dec 2025 00:00:00 +0000

What I learned this week

Software

Livecoding melodic DNB music in Strudel

Business/Finance

Selling onions > Selling software Sell your… voice

Interesting Links

Nonce

When a measure become a target it ceases to become a good measure… goof background on Goodhart’s law

sepulchral

Math/Stats

Great Transformer explainer

Travel

Other

Chicken Stock

What I learned this week 12-20-2025

Sat, 20 Dec 2025 00:00:00 +0000

What I learned this week

Software

readability, hackability and abstraction

Reverse a hexdump cat some_file.txt | xxd -r >reversed_hex_dump.txt

Databento looks great

Start a dev container

Business/Finance

Insider trading?

Interesting Links

Cool rugs

Think before grabbing

Why are so many peds killed by cars

Math/Stats

Travel

Other

Need to build this custom big mouth bass

LLM Weaponized Spam Circle

Sat, 13 Dec 2025 00:00:00 +0000

Right now, circular financing and AI bubble takes are a dime a dozen. These may prove true, but be warned:

“Markets can remain irrational longer than you can remain solvent.”
― John Maynard Keynes

I think another kind of circle is forming: The Weaponized Spam Circle.

Thesis

LLMs are great at commoditized tasks because it lowers the cost to produce them. People love this so much they leverage them for non-commoditized tasks, in doing so, they shift the burden of actually thinking to the reader. A reader will be overwhelmed by the quantity of generated text and require LLM assistance to ingest/summarize it, resorting to LLM assistance to respond.

It’s Brandolini’s Law but instead of refuting the bullshit, the bullshit is shot back at the bullshitter and the cycle begins anew.

In adding another layer to how we communicate with one another, we shell out big $$$ to afford the matrix multiplication to do so.

But what is a commodity?

Let’s do the wikipedia cliche:

“In economics, a commodity is an economic good, usually a resource, that specifically has full or substantial fungibility: that is, the market treats instances of the good as equivalent or nearly so with no regard to who produced them.”

“With no regard for who produced them…” will be the thread I hold throughout the rest of this piece.

Computers (neural networks trained on millions of positions) are better than humans at Chess. Humans prefer to watch humans play chess and regard using a computer as cheating.

This means playing chess IS NOT a commodity, we CARE who plays the chess

Do people check where the gas is coming from… do they deep dive the exact source of the energy being used to charge their Tesla? Pretty much no… if you want to get pissed here and tell me how you’d never use Saudi oil and you check at every gas station be my guest.

This means filling up/charging our cars IS a commodity, we DON’T CARE how it happens… we want it as cheaply/quickly as possible.

But writing... is it a commodity?

When Characters are a Commodity

Not all writing is thoughtful. There’s tedious writing marked by “character limits”, extrinsically imposed deadlines, random boiler plate, documenting a new feature, emails to insurance companies, and the gut wrenching feeling of having to start putting words down for a book report.

In these moments, an LLM can shoot through character limits and fake decorum to produce words that felt far away. With a bit of editing, large writing burdens are reduced to a slight inconvenience.

But we’ve already made a big mistake… if you’re nodding your head at the two paragraphs above, read them again.

Expand when you've formed an opinion

I threw “book report” in there with the rest of those “examples”. A book report is not a commodity. If you’re writing a book report you’re most likely learning the basics of writing. You’re learning the fundamentals of how to take a position on something and defend it. If that is a commodity, then I don’t know what is. The thinking is the important piece and the author wielding their brain is what matters, the physical words are a by-product used for other humans to interpret your thoughts.

Also what about: “documentation” and “extrinsically imposed deadlines”. I can spot issues there as well.

Sometimes it’s not easy to determine what IS and what IS NOT a commodity when it comes to writing.

Sometimes it’s too easy to label some writing as a commodity and shovel it off to an LLM.

Sometimes it’s not easy to think, thinking is hard, new ideas are hard.

When Characters are NOT a commodity

“Writing is thinking. To write well is to think clearly. That’s why it’s so hard." ― David McCullough

At Amazon, there is the notorious six pager.

“When you have to write your ideas out in complete sentences… in complete paragraphs it forces a deeper clarity of thinking.”
“The author who has put a tremendous amount of work into writing the memo gets to see everyone read the memo.”

A six pager is NOT a commodity. The moment it becomes a commodity is the moment it becomes no better than a powerpoint. The people who wrote the document should care about it, should have thought deeply about its contents, and those reading the document digest it out of that same thoughtfulness.

Yet LLMs have quickly crept into this space, because writing a six pager is hard, it requires thinking, and plopping that burden on something else is just a few key strokes away. The writing is on the wall:

Python 3.7 on Lambda (LLM Knowledge cut off)
“its not just a ___ its a ___”
cascades of bullet points
Hallucinated information that looks ok but is wrong after a bit of thought

Wait, are they treating this document like a commodity?

Did they take the time think through the key ideas?

Did this person even read what they wrote?

Do they even care about my time?

…I’ll respond in kind.

That “six pager” can be done so FAST because nobody is THINKING ABOUT IT. The LLM takes a unique piece of articulate thinking and turns it into a commodity. How do we respond to a commodity? Drive down the price! If they chose not to think why should I chose to think. And……… we have our circle.

Output as a result of input

I could care less about the output of the hallucinated + verbose document. Show me the input, show me the meat of what you are trying to convey. If there was no or sparse input to begin with… then there was no or sparse thought.

Is adding fluff to things holding us back? Is this why people reach to LLMs to puff up their ideas? If there is an important point to make then lets start there, but we don’t need to play telephone back and forth via LLM.

Why do we teach assembly

Many undergraduate curriculums teach assembly. Most undergraduate students will never earn a $ or spend a minute of their free time writing assembly. Why teach it? Because it helps provide insight into how computers work.

Why take the time to develop an idea and write it down? Because it forces you to think critically about the opinions and about the facts.

Doing this because we can’t help ourselves

There are many practical use cases for LLMs. It is 100% clear that they are beneficial, especially if you take the time to give them the tools, context, and guidelines they need to succeed. Just like a human, if you give an LLM limited context and tell it to do something it will probably suck at it. LLMs are NOT humans, it’s math, and if the necessary inputs aren’t provided the equation won’t be solvable.

Just like with any beneficial thing, humans are going to exploit it for their gain. It’s fundamental to how we operate. This massive boon in LLMs feels like a free lunch… it feels like something to be exploited, yet it remains a balancing act.

Sure you can generate a bunch of text and throw at someone, but they can do the same to you… we are back to square one.
Sure you can generate 100,000 lines of code, but now you need to maintain, refactor, and change one hundred thousand lines of code.
Sure you can automate that PMs flow where they aggregate a bunch of stuff from Asana, Excel and your ticketing system into a general summary, but why the fuck are we tracking things across 6 different application in the first place.
Sure you can automate your Linkedin post, but it’s just getting gobbled up by bots rolling around in the absolute slop trough of their own creation. If you want a peek behind the curtain and see what the dead internet will look like, scroll around there for 5 minutes.

Other Random Thoughts

Do we really want people growing up and learning how to read/write via LLM? The pattern’s they’ll learn will converge us towards the style of LLM writing that people find so obviously annoying today. Will the humans have the ability to recognize that and AVOID that style?

You have a brain, you can still think, brains are powerful… use it.

Writing without thought distills an entire method of thinking into something transactional, input output. Don’t ask how it works it just does.

For kids, writing is the new math.

In regard to math:

When will I actually use this when I grow up?
I have a calculator I don’t need to learn this.

Now, in regard to reading/writing:

I have an LLM I don’t need to write this.
What’s CliffsNotes? I just use an LLM

For adults, the same super powers you can use to generate a lot of text can be used against you. Most adults will be using their single LLM chat session to compete against an entire layer of corporate LLMs. Let the words be fired back and forth and to the datacenter owner’s, chip manufactures, and model trainers go the spoils.

Whenever I leverage an LLM to write I ask a flavor of these questions:

Would the reader or any of the reader(s) consuming this content care if it was produced by an LLM?
Would I personally care if the content under my name was authored by an LLM?
Have I provided the proper steering, guidelines, and context for the model to be successful?
Have I read what the model has produced?

What I learned this week 12-13-2025

Sat, 13 Dec 2025 00:00:00 +0000

What I learned this week

Software

Cool designs

Interesting points

That’s also why I see no point in using AI to, say, write an essay, just like I see no point in bringing a forklift to the gym. Sure, it can lift the weights, but I’m not trying to suspend a barbell above the floor for the hell of it. I lift it because I want to become the kind of person who can lift it. Similarly, I write because I want to become the kind of person who can think.
The original sin of artificial intelligence was, of course, calling it artificial intelligence. Those two words have lured us into making man the measure of machine: “Now it’s as smart as an undergraduate…now it’s as smart as a PhD!” These comparisons only give us the illusion of understanding AI’s capabilities and limitations, as well as our own, because we don’t actually know what it means to be smart in the first place

ROT13

misaligned LLMs

Business/Finance

R.I.P ESPN bet

Is this the dumbest ticket buying experience of all time. The amount of rent seeking in this industry is disgusting

Interesting Links

Brandolinis Law a.k.a bullshit asymmetry principle

Lebron played against 35% of all NBA players

Put your teeth in milk

Math/Stats

The Fourier Transform explained via smoothies

Top P Sampling

Travel

Other

variegated: exhibiting different colors, especially as irregular patches or streaks.

The Count of Monte Cristo

Sun, 07 Dec 2025 00:00:00 +0000

The book

This will serve to store notes/insights/thoughts about The Count of Monte Cristo

Quotes

Page 34 - A curse on those who fear wine: it’s because they have evil thoughts and they are afraid that wine will loosen their tongues.

Page 54-55 - God can change the future, He can not alter even an instant of the past.

Page 90 - He looked up with the satisified air of a man who thinks he has made a discovery when he has commented on someone else’s idea.

Page 107 - …in polotics there are no people, only ideas; no feelings, only interests.

Page 109 - You people who hold power have only what can be bought for money; we, who are waiting to gain power, have what is given out of devotion.

Page 121 - what can any prisoner have to ask for, apart from his freedom?

Page 132 - To a happy man, a prayer is a monotonous composition, void of meaning, until the day when suffering deciphers the sublime language through which the poor victim addresses God.

Page 154 - Because you have an instinctive horror at the idea of such a crime, to the point where it has never even entered your head, the old man continued. For, in simple and permitted matters, our natural appetites warn us not to exceed the boundaries of what is permissible for us. The tiger, which spills blood in the natural course of things, because this is its state of being, its destiny, needs only for its sense of smell to inform it that a prey is within reach; immediately it leaps towards this prey, falls on it and tears it apart. That is its instinct, which it obeys. But mankind, on the contrary, is repelled by blood. It is not the laws of society that condemn murder, but the laws of nature? Dantes was struck dumb: this was indeed the explanation of what had gone on, without him knowing it, in his mind - or, rather, in his soul: some thoughts come from the head, others from the heart.

Page 161 - unless an evil thought is born in a twisted mind, human nature is repelled by crime. However, civilization has given us needs, vices and artificial appetites which sometimes cause us to repress our good instincts and lead us to wrongdoing.’ Hence the maxim: if you wish to find the guilty party, first discover whose interests the crime serves! Whose interests might be served by your disappearance?'

Page 168 - Learning does not make one learned: there are those who have knowledge and those who have understanding. The first requires memory, the second philosophy.

Page 339 - In every country where independence takes the place of liberty, the first need felt by any strong mind and powerful constitution is to possess a weapon which can serve both for attack and defense; and which, by making its bearer formidable, will mean that he often inspires dread.

Page 502 - more than once, proved to me that our excessive concern with the welfare of our bodies is almost the only obstacle to the success of any of our plans, when these demand rapid decisions and vigorous and determined execution. In reality, once you have made the sacrifice of your life, you are no longer the equal of other men; or, rather, they are no longer your equal, because whoever has taken such a resolution instantly feels his strength increase ten times and his outlook vastly extended.

Page 552 - What I am saying, Monsieur, is that your eyes are fixed on the social organization of nations, which means that you only see the mechanism and not the sublime worker who operates it. I am saying that you only recognize in front of you and around you those office-holders whose accreditation has been signed by a minister or by the king and that your short-sightedness leads you to ignore those men whom God has set above office-holders, ministers and kings, by giving them a mission to pursue instead of a position to fill.

Page 552 - You know that all human inventions progress from the complex to the simple and that perfection is always simplicity.

Page 727 - With one of those fatuous smiles which had the same effect on Monte Cristo as the pallid moons that inferior painters plant in the sky above their ruins.

Page 766 - ‘His experiments have greatly advanced science, I presume?’ ‘No, but he writes them up in a very fine style.”

Page 850 - When I close my eyes, I can again see everything that I used to see. There are two ways of seeing: with the body and with the soul. The body’s sight can sometimes forget, but the soul remembers for ever.

Page 872 - Because, you must understand, my dear friend, one should never be exclusive. When one lives among madmen, one should train as a maniac.

Page 952 - Moral wounds have the peculiarity that they are invisible, but do not close: always painful, always ready to bleed when touched, they remain tender and open in the heart.

Page 953 - Truly generous men are always ready to feel compassion when their enemy’s misfortune exceeds the bounds of their hatred.

Page 1121 - There are some situations which men instinctively comprehend but are unable to comment on intellectually. In such cases, the greatest poet is the one who emits the most powerful and the most natural cry. The crowd takes this cry for a complete story, and it is right to be satisfied with that, and still more so to find it sublime when it is truthful.

Page 1242 - There is neither happiness nor misfortune in this world, there is merely the comparison between one state and another, nothing more.

Notes

Amazing story… took me over 3 months to finish, but it was highly worth it. Up there with my favorite books of all time.

What I learned this week 12-06-2025

Sat, 06 Dec 2025 00:00:00 +0000

What I learned this week

Software

Write stderr to dev/null: 2>/dev/null

To find a file based on user+group+size: find / -type f -user bandit7 -group bandit6 -size 33c

Only unique lines in a text file: sort some_file.txt | uniq -u

strings command in Linux is a utility used to extract printable character sequences.

Business/Finance

Yikes, but not surprising… insider trading applies only applies to some of us

A bit negative but thought provoking

If no one asked and no one is on the hook to change anything: Stop talking.

Interesting Links

Living in Casinos full time

To each their own, but this seems like a full time job which you’d still need to support yourself. I guess you just have to really enjoy vegas and sleeping in hotels

1120 foot wingspan ft wingspan

Unlocking wheels

Math/Stats

Travel

Other

The name “marimo” is a reference to a type of algae that, under the right conditions, clumps together to form a small sphere called a “marimo moss ball”. Made of just strands of algae, these beloved assemblages are greater than the sum of their parts.

What I learned this week 11-29-2025

Sat, 29 Nov 2025 00:00:00 +0000

What I learned this week

Software

Business/Finance

Interesting Links

Transfer back to VHS

Math/Stats

Travel

Other

Jason Williams was teammates with Randy Moss

I think I deleted all my saved links this week so only these this week.

Korea

Mon, 24 Nov 2025 00:00:00 +0000

About

Trip is South Korea (Seoul and Jeju) from 2025-11-07 -> 2025-11-21.

Below is my daily notes from the trip, lightly edited post trip. (Edits are bolded).

I’m picking one photo from each day to include.

Summary Thoughts

Korea Travel Log

Korea journal

2025-11-07:

Currently waiting to take off for SF. Today was very chill compared to yesterday where I got a flat tire on my way to present at our summit. (This was a work summit where I was presenting on Q CLI).

Feeling pretty good, back is still a little tight, I kind of tweaked it riding some gravel with Kaiser a week or so ago.

Anyway, feeling pretty relaxed overall, ready to sleep on some flights. Landed in SF, walked around and talked to my girlfriend. Got some ramen and sat next to the cofounder of some startup talk about how his life is designed around revenue, dreams about his customers, doesn’t know what holidays are all he does is work, and how he’s slept with his gay banker but he’s not gay… going to keep tabs on this one and see how it does lmao!! Welcome to San Francisco. Preferred meeting Charles Barkley last time when I was flying to Japan hahaha.

2025-11-08: 3

I guess it’s the 8th now although it’ll be a short day for me. I’m like 9-10 hours into the flight to Korea. Slept ok for first 8 hours. Almost 18 hours since being driven to the airport, I can’t decide if it’s felt like a long 18 or not. Anyway, going to wrap up the last few hours of this flight and then on to the 9th.

2025-11-09:

Landed, struggled to figure out T money but eventually got it and got in the right train hahaha (This involved me realizing I needed to get cash to load the things and going back and forth between thinking I needed cash or not… then figuring out how to get cash). Got picked up from very nice Airbnb lady who brought me up to the room. I made coffee, cooked breakfast (homemade jam) and talked to pal on the beautiful porch. Sent the airbnb lady a picture of my breakfast and she gave me more eggs and tomato’s… so nice.

Took the bike they have out for a nice ride on their river path. It was awesome, so many riders out, great weather, and nice pavement. (This didn’t feel grerat on my back but was a beautiful windy day for a ride)

Got back and then walked over to the noodle place that our host recommended. (This place was extremely surprised to see me and was located in what seemed like a neighborhood surrounded by small statues). The walk was nice it was almost in a neighborhood and then I came to a place with a ton of statues, walking to the end of them lead me to the noodle place. I took my shoes off and sat by myself and had a massive bowl of buckwheat noodles. Had a nice walk back checking out the town a bit before heading home.

The bathroom was so cramped I had to sit in the tub to shower hahahaha. Zach arrived around 8, I let him up and went to bed.

Noodles: 인천광역시 계양구 학고개길 15-29 Sufia : 인천광역시 계양구 정서진로 1247

2025-11-10:

Woke up at 3:45 so almost got 8 hours. Made coffee/toast with Zach, cleaned up and headed out (watched a little football too). (This required some finagling with a VPN/location settings).

Took an uber to GMP where we got dropped off at the international terminal so had to walk over to the domestic one. I needed a new boarding pass so it would match my passport so we had to wait and got Jamba Juice. Then at security at turned out Zach needed the same thing but they didn’t do it for him so he had to go all the way back and redo it.

Anyway hope we make this flight to Jeju hahaha.

Flight got delayed and had plenty of time. Met some nice people from Sevilla if I’m there I need to visit Bar baratillo.

We landed in Jeju and Zach forgot his water bottle on the plane. We got that, picked up Ray and then drove to some random noodle place to get lunch. It was good and then I had Zach drive to the Airbnb without routing it and he did pretty well. We also saw an f-150 on the drive over, America!

Stopped at an emart for groceries and went to the new Airbnb. This place is ok, good views though. Went for a walk and the surrounding area is beautiful. The JW Marriott we are next to is crazy as well. There’s some good restaurants around us.

Anyway feeling a bit tired, going to try and stay up until 8pm and call it a night.

Also lemon sour hi-chew are good and strawberry snickers are bad.

2025-11-11:

Slept for like 11 hours and got up at 7. Cooked up coffee and oatmeal which took forever in this tiny little kitchen. Of course we don’t have salt, pepper or laundry detergent.

Went out for a quick walk along the coast which was beautiful. Then face timed Ally and watched the packers vs eagles a bit (won fantasy football this week).

We decided to go check out a waterfall which wasn’t really happening because it only happens when it rains, but saw some big spiders and oranges. On the way there we stopped at this cute coffee place (I got orange juice and a slice of cake) that had an amazing logo. I think I’ll use it to make ally a shirt.

Then we drove to a forest loop hike where the entire path was paved with this braided rope that was very soft to walk on (about 5k) (This hike smelled pretty bad as we’re pretty sure we were surrounded by manure from the thousands of farms on the island)

When then headed towards a running track on the north western part of the island so Zach could run. It was by Donatos a place we wanted try for pizza. While Zach was running I walked over to hanmart and bought detergent. Then we decided we’d go to some little Thai food place for dinner.

We walk into this Thai food place (Lam Taetae) and there was nobody there. In fact we’d be the only people to enter the entire time we were there. It was just a guy and, I assume, his wife and we had amazing fried rice, pad Thai and he had us try soy bean soup. He was very nice and we chatted a bit with him, apparently he had been a chef at a thai restaurant for a decade. Amazing meal, just another example of randomly walking into a great experience while traveling.

We drove back home (1 hour ) in the dark, hitting speed bumps and making a wrong turn or two. What an interesting place/culture.

2025-11-12:

Woke up and was a little quicker at making coffee/oatmeal today. Heading to hike Hallasan (the mountain in the middle of the island). Didn’t realize we had to make a reservation so did that really quick and headed over there before our 10am cutoff time.

Parking lot was full and we missed the cutoff time. So then we decided to do a different hike that climbed another side of the mountain, while that one didn’t meed a reservation it still had a cutoff time to start for some reason and the guy wouldn’t let us go through hahahah. So we ended up looping around this cool cemetery and we’ll try hallasan again tomorrow. (We did not realize that we needed to be a bit more prepared to do this hike, they seem very regimented about it)

Our new plan was to drive back to Cheonjiyeonpokpo Falls and have our sandwiches. We parked kind of close which led us to walk around for a few hours before finally getting to the park and checking out the falls. (We did not park close, we parked on the absolute wrong side and walked all the way around the park to get to the falls haha)

We took photos, ate sandwiches and started back towards the car. We got some snacks from CU (I like seaweed lays chips) and watched old people play put-put golf. We both kind of wanted coffee so we ended walking back down these stairs we had just gone up and into the town.

The coffee place we wanted to go was closed so we walked past another that looked cool. Had coffee and I bought some beans for my mom. We then went up this small street that seemed to be dedicated for an artist and get gifts for people from a small book store and another shop.

We walked back to the car and drove home. Out on the roof, I was helping Zach improve on chess while we ate yogurt and hatched a new plan for the Hallasan. After the sunset, we needed to get dinner. I found this random place called inupaen that was incredible. Some of the best steak I’ve had in a very cute place that was well decorated.

After getting back we decided we wanted to sleep in tomorrow. We’ll drive up the eastern side of the island tomorrow and do Hallasan on Friday before our omakse experience.

2025-11-13:

Woke up and I burned some avocado toast. We charted a few places to stop on the way to the eastern most part of the island where they have this cool glass house cafe.

We headed out around ten and went to seosokkak. It was this little inlet where freshwater meets the sea. We rented a small boat and paddled up it. We walked the beach a bit and then headed towards our next stop. (I found a small heart shaped rock for my mom on the beach)

After driving past a place that said Texas bbq the next place was closed, opening in late November. So we decided to go to a burger place instead. It’s called fantastic burger. With a view of the sea we had large quantities of curry and onion sauces poured on top of burgers with milkshake and fries. Suffice to say we had to go for a walk after. (Zach did NOT enjoy this meal)

Leaving the car at the burger place we walked down to some tide pools to help digest the slop. We then headed towards an herb farm.

The farm was pretty cool, extremely well manicured. There were tons of cool looking plants and we got some grapefruit tea which was pretty nice to walk around with. We couldn’t figure out how to play the put put golf so after a bit ended up leaving (think it was more for couples at night).

We continued on to the glass house. We parked our car by the ocean on this peninsula and walked to this massive glass building. Inside I bought shoes and Zach bought more gifts. We then went to the bakery and got some baked goods/coffee. We sat up on the second level overlooking the sea and played chess. (This place and the surroundings were highly aesthetic, would recommend)

After this we walked around outside to a small lighthouse before turning around and heading back to the car. We drove to a supermarket to get supplies for tomorrow. After wards, we went to a nice Korea bbq place for black pork. It was excellent we had pork, egg dish, rice, and tons of sides.

Driving home I booked our tickets for the hike tomorrow (I think they were literally the last two available). Got back and made some sandwiches and am going to bed now. Going to be a great day tomorrow but we are getting an early start.

2025-11-14

Woke up at 4:30 had coffee and oatmeal and then we were out the door around 5:30 headed to the parking lot. The drive up was cool in the dark the rods are tight and it was a little hard for Zach to see.

When we got to the parking lot it was already full at 6:30. Luckily we had a backup plan to park at the university and grab a bus back to the parking lot (comprehensive shooting range west was the bus stop name). We parked and got a taxi instead which worked out fine.

Back at the parking lot we showed them our QR codes and were hiking by 7:00am. The hike started off pretty chill then we hit a middle section which was steeper/rockier and the last bit was extremely steep pretty much just stairs. The weather was great, there was no water but there were rest stops/bathrooms. People kept giving us oranges and on the way down we must have said hello in Korean 1000x.

The top of the mountain was pretty cool, great views. We had some snack and then hiked down, we didn’t wait too long as we didn’t have much water. The way down was rough, the rocks we had hiked up now required constant attention/balance while dodging people going the other direction. At least passing was easier as we had nobody to pass.

It took us about 6 hours total to do Hallasan about 5 hours of hiking. We took a bus back to the car (the driver just waved us on as we had no idea how to pay). Driving home we made it back by like 3PM after getting some gas that we thought we didn’t pay for but turns out we did. Zach and I had a bet about how many “dings” we’d get which occur when Ray hit a bump too hard.

We showered quickly and then hung out a 60 beans, Z got another flat white and I got a hot chocolate. Even though the Airbnb wasn’t great it was in a good location and we just sat and watch the ocean.

Afterwards we walked back up the hill a bit to our dinner at Hachi. This was an amazing omakse experience. I got to butcher a bit Japanese, eat some amazing pieces of fish, and ended the night with a little green dessert ball that was amazing.

This was one of the best meals I had, and for only $100 per person it was a bargain. If I end up in Jeju again I’ll have to go back. This was an amazing day to end an insane stay. I was telling Zach that I didn’t think I would be able to be as cool as Japan and this week has been! It’s been great in pretty much every way.

2025-11-15:

Alright our big travel day begins. Calves are a little sore but we’ve got nearly every possible form of transportation to take.

We cleaned up the Airbnb and hit the road at like 7:50. Apple Maps fucked up and we ended up taking a scenic route on the other side of Hallasan we had been the day before. We dealt with some slow/dumb drivers. Before retuning the rental car (not full). We took a shuttle to the airport where we checked our bags and then headed through foreigners security.

Through security I got some snacks at 711 before our flight was delayed 15 mins and our gate was changed. We got on a bus which took us out to the tarmac where we boarded our flight. On the plane now hopefully no issues as we need to take the subway over to Busan Station to catch our high speed train to Seoul.

Took the subway to Busan Station. No problems and we got lunch at the station (Ramen + Kimbap). high speed train rolled up on time and we headed to Seoul. There wasn’t much to see and our ears kept popping. I ended up watching Seinfeld and playing chess poorly. We both agreed we were glad to say we’ve taken the KTX but it was NOT worth the hastle.

After we got to Seoul station around 6pm and we were both tired. Butttt we had more subways to take. Lugging our bags we took several crowded Friday night subways to Walkerhill. At our final station, we had another 1km walk ahead of us UP THE HILL to the hotel. We did all the travel but wow it was not worth it haahahaha.

We checked in. We were going to get an extra bed but decided against it because the room was small and our single bed was big. The shower is great and the hotel it really nice but everything is overpriced. I was so tired I just skipped dinner and went to bed.

2025-11-16:

Got up early. There was some coffee I could make in the room but I thought it was instant coffee so I poured it straight into the cup but you were supposed to poor water through it via a filter lol. So I went downstairs to the buffet and got a fucking $8 Americano. I chugged that and then got my stuff together for my hike.

Ready to go I set off for Baegundae peak in Bukhansan National park. My right calve is pretty sore but overall I feel ok and today is the best weather for it. I took the subways without issue. Grabbed a snickers bar at the base and started up.

The first bit was basically on a road before the actual hike. Some taxi crashed his car on the road oof. I made it up to the start but was a bit low on water. I went downstairs to this small shop that this lady was trying to open. I helped her get the lock open and then got some water from her shop. I threw a liquid IV in one of bottles and started off.

This thing was steep my legs were still tired from Hallasan. While this was shorter it was tough. Reaching the top, I threw my gloves on for the final scramble to the peak. It was kind of like half dome but with less ropes and more metal railing. It was very crowded and I don’t hang out long at the top. Took a few photos and heard back.

Rather than heading down the same way I decided to descend the other side and cross the park. This side may have been steeper than the other and there was no stairs whatsoever just straight rocks. I got down to this road and then it was easy from there. The road was beautiful with fall colors and I was having a great time listening to 80s music. I wasn’t super hungry but decided I needed lunch. I needed up at a small Korean restaurant where I ate a bunch of rice tofu and fermented vegetables. This white fig and some Koreans sat at the table next to me and I think they worked for black diamond. I was going to show them my carabeaner but decided not too.

I walked to a bus and then took subways back to walkerhill. Shower felt great and than Z got back from exploring the city. We waited a bit for his lab mate Skylar to show up. We had to go down and get Skylar his room because his card was getting declined.

All three of us went and got dinner, I randomly found some place called Bao Seoul that was amazing. We had Bao, Fried chicken, and noodle… insanely good. After walking out it started raining so we waited underneath a convenience store and I debated purchasing these little nipple patches that looked hilarious.

The rain stopped and we walked over to this place called nowhere which was an art installation that also seemed to be selling glasses and perfume (Gentle monster was the name I believe). I talked to this lady about what her favorite weird glasses were and then we Ubered back to the hotel. we made sure to clear out the complimentary minibar before hitting the sack.

2025-11-17:

Going around and exploring the city today. Got up around 7 and Z went to the buffet. I had coffee in the room, watched football and called my parents. my dad has his surgery scheduled for 12/11 so that was good to hear.

I headed downstairs to go explore the city and was hit with a blast of cold air… went back up and threw my thermals on. Another breakfast at Starbucks for 1/8 the price of the hotel and then I was off to Gyeongbokgung Palace. It was pretty cool but my legs are shot from the last few days of hiking/travel and the amount of tourists here is nauseating. It’s hard to do anything without getting in the way of somebodies photo. I walked around for a bit before taking shelter in the modern art museum across the street. This place is pretty weird but it’s nice to sit inside and eat a little pastry and type this out.

I think I’m going to go to the national museum of Korea next and look for lunch.

Started toward the museum and immediately found a cute lunch spot by the palace. More Korean food but it was really good and had a little purple flower on top. Finished up and took the subway to the museum.

Zach sad he would come but ended up getting hung up so I walked through the museum by myself. It was pretty cool but my legs were hurting from all the walking and a left around 4. It was freezing outside and the sushi place I wanted to try didn’t open until 5pm so I ate so MnMs in a 7-11.

I pulled up to Sushi Tei right at 5 and was the first one sat for dinner. I got the sushi set which was a bunch of sashimi (tuna, white fish, squid,etc) as well as some udon. It was excellent. Took the subway back to the hotel and read a book about hotels in the library. After that I helped Zach practice his presentation and called it a night.

2025-11-18:

Got up around 8:30 and went to Starbucks again to get a little breakfast. It’s just convenient and cheap. I’m taking the subway to go make Kimchi with a nice lady named Grace.

At my final station there was a little flower shop so I could buy grace flowers I don’t know how these things work out so perfectly this happened the last time I did one of these Airbnb experiences as well.

Two other people showed up and we made/learned about kimchi, made kimchi pancakes, and ate pork with kimchi and drank tea. It was amazing and grace was so nice. She even made us homemade dumplings and we got a nice jar of kimchi we made to take home. Amazing experience, and I’ll have to go back and do her dumpling class!

I’m now sitting at bean brothers with a cup of coffee it’s quite cold. Not sure where the rest of the day will take me but I need to get this kimchi back to the hotel hahaha. Crazy I’m taking this home.

Went back to the hotel with the intention of leaving but ended up staying at the poster session for a few hours. It was fun to talk with various people about their work and I could understand enough that it made for good conversation. I worked up an appetite so I headed to a restaurant that grace recommended called namdosikgark. I’m sitting here now waiting for my food to come out, I’m the only one here.

Food was excellent. I got a large vegetable pancake and spicy beef stew. I learned how to say radish in Korean today so I asked the lady if the kimchi was radish in Korean even though I already knew it was radish.

I was kind of in some sort of shopping district so I walked around after eating and got a few things for ally. This one artist had drawn John wick on a scroll so I bought that for Louis. There is an endless supply of cute tiny things here

Took the subway back uneventfully and then hiked back up to the hotel. Showered and then hung out in Zach’s lab mates room (Skyler) who has a great view of the river/bridges. Went to bed around midnight.

2025-11-19:

Woke up to all the lights coming on in the room randomly around 9:15. Called ally and talked with her for a bit. Got out of the hotel around 11 and headed toward inwangsan which is a mountain in the middle of the city. Took subway over and stopped at Starbucks for a hot chocolate.

The inwangsan hike was pretty cool. My legs are tired it was steep as hell but the view of the city at the top was unbeatable. The Seoul wall is also pretty cool. Hiked back down while eating some wasabi covered almonds which are excellent. Decided to take the subway over to the buk-seoul art museum. I’m sitting on the roof now and it appears to be a bit of a mistake to have come here, there’s only like one exhibit. I think my plan now is to head back to the hotel and meet up with Zach and Skyler for dinner. I do need to go to bed at a reasonable hour as the DMZ tour is early tomorrow and I don’t want to be wrecked for it.

Went back to the hotel and met up with Z and Skyler we took the subway to some chicken place where they put bunch of stuff in a big pan right in front of us and we ate it. The guy that worked there was nice and showed us a little donut shop so we got donuts after. While leaving the donut shop i used my new ability to say “see you later” which was a hit.

We went back to the hotel and a realized that I didn’t want to get up at 5am for the tour. So I booked a 10am tour and wrote the 7am one off as a loss.

2025-11-20:

Woke up at 7:30 and had coffee/yogurt in the room. For some reason I checked Viator and noticed a message saying meet at 9:30 instead of 10 so I hustled over to the station and made it there on time.

We hopped on the bus and our tour guide Lina talked us through some basic instructions as we took the highway towards the DMZ.

We got to our first stop after about an hour which was just a little tourist park with a train going over a river. I think it was sentimental about a particular battle there and there were memorials. I walked around, got some snacks and bought some North Korean currency which is kind of cool to have, I think I’m going to give it to Jeff.

We boarded the bus and headed to the tunnel. This was the point where we actually crossed into the DMZ. There was a little checkpoint and some 20 year olds on military gear hopped on the bus to check our passports. Anyways, Apparently North Korea has been digging a ton of these tunnels under the DMZ toward South Korea. Every once in a while SK finds one, and this one they turned into a tourist attraction. We watched a 8 minute SK propaganda video which was pretty funny and then got to go into the tunnel.

We were given a hard hat before walking down a 300m long very steep tunnel. This one was well built. At the bottom we hit the North Korean made tunnel. It was quite small, hence the hard hats, and you kind of walked bent over to the end of it where the blocked it off. Then you turn around and huff it all the way back up. Lina gave me a sticker for being the first to make it back up. It was pretty cool to be there.

Next we want up to the Dora observatory where you could look at but not take pictures of North Korea. There was a tit-for-tat flag battle between the two nations that was finally needed when North Korea built a 160m tall flag and SK decided to stop playing the silly game. That flag is pretty cool to see it’s very large and high up, even from 5k away and a bit foggy it was very visible. Otherwise there were just abandoned factories and mountains you could kind of make out in the distance.

I went to this little cafe and had a chocolate muffin while looking out at North Korea. It was a cool experience, they also sold shitty hotdogs and a had to listen some lady bitch out her sister next to me while I enjoyed the view.

We went back to the bus for our last stop. This stop was through a village of farmers that were still allowed to live in the DMZ before stopping at a glorified convenience store/bathroom to allow us to buy things before heading back to Seoul. I bought a water and met some nice guys from London who K chatted with for a bit outside.

Back on the bus we headed back through a military checkpoint to make sure that everyone that was in the bus on the way in was on the bus on the way out. I chatted with a nice guy named David from Charleston as we drove on the highway back into the city.

I asked Lina for some food recommendations, she pointed out a few places and then one of the lady’s on the tour mentioned she was going to this dumpling place that was on the Michelin guide (for value restaurants). We got off at myeongdong station and decided to get dumplings with her.

The tour made it difficult to remain sincere about the whole thing and the tragedy that is the war and the current state of North Korea with how curated/touristy each of the sites was.

We walked up to the third floor of this restaurant and they sat us and brought out noodles and dumplings almost immediately. They were pretty good, the kimchi was very spicy. She was nice, she was traveling for several months after being laid off due to trump cutting all the funds of USaid. We talked about politics a bit and she’s also going to stay in some Buddhist place for meditation which sounded cool.

We parted ways after our dumpling snack and I headed back to the station to go have dinner with Zach. Dropped stuff at the hotel and we headed towards this place by the lotte world tower. That place closed at 9 so we were cutting it pretty close. We ended up walking around and randomly found a small restaurant called dokidoki. We got fried chicken, a tepenyacki noodle dish which were both amazing. Then we got one more dish which was some sort of fish roe pasta that was great. An amazing dinner to cap off the trip.

We headed back to the hotel and Zach packed a bit and I got a coffee from the CU on the basement for tomorrow and headed to bed.

2025-11-21:

Woke up around 7:30. Packed up my stuff and dropped the bags off at the concierge. We checked around for anyway to load our T Money Cards but realized we’d need to leave the hotel. We walked over to a bank and pulled out 10k krw for a fee of 4200krw hahaha.

I got a banana from Starbucks and we waked over to a little park. Zach walked barefoot on some barefoot walking trails and I chatted with a lady about Achason mountain which I didn’t get to hike but was right next to Walkerhill.

We headed back to the hotel, grabbed our bags and took the bus to the station. Two hours of subway later. We made it to the airport. We got Chinese food and then I said goodbye to Z and we parted ways as his flight was in Terminal 2.

Flight was delayed a bit, so I walked 5-6 miles around the airport which was mostly just souvenirs shops and extremely high end fashion. I wonder if there’s an arbitrage opportunity for buying things here and selling them back in the US. I didn’t go in to check any of the prices but would’ve been interesting to know if they were priced differently. I also wonder how often they update the price with given exchange rate changes.

On the flight we had a massive tail wind which saved us probably over an hour of flying time. I chatted with some of the flight attendants I had met in the airport after I found an AirPod and gave it to them. One guy was 51 and had fun out clubbing in Seoul the night before, he told me to visit Singapore and Greece.

Once we landed we got stuck on the runway and lost all the time we had gained in there. I hustled through customs, re checked my bag and easily made the next flight, so the rush wasn’t as warranted as I thought.

On the flight from SFO to PHX I sat next to a guy who works for openAI we played chess and had a good conversation. He’s had dinner with Tyler Cowen before (no wonder he’s so bullish on “AI”).

Mom picked me up and then I showered and had dinner at home. I picked up Zach from the airport and brought him home. I wasn’t really tired, so I decided to drive down to Tucson to see ally and pick up my bike. I made it down to Tucson in about 90 minutes getting there before midnight. And that closed out an extremely long day (21 hours of travel to Phoenix + another 2 to Tucson).

What I learned this week 11-22-2025

Sat, 22 Nov 2025 00:00:00 +0000

What I learned this week

Software

Business/Finance

Overregulation, where’s the happyMiddle ground?

Interesting Links

Telezapper

Math/Stats

Travel

Other

What I learned this week 11-15-2025

Sat, 15 Nov 2025 00:00:00 +0000

What I learned this week

Software

Wifi location

Wifi aware

Business/Finance

Interesting Links

Korean maps

Math/Stats

Travel

Other

YMCA and low income housing

What I learned this week 11-08-2025

Sat, 08 Nov 2025 00:00:00 +0000

What I learned this week

Software

Running your own ISP

Business/Finance

10% of meta revenue is from scams

Lets go, no more driving

Interesting Links

Truck desk

Biggest piece of glass

Math/Stats

Travel

Other

4/19 revolution

Steinbeck I’d like to read

Cool place to stay

What I learned this week 11-01-2025

Sat, 01 Nov 2025 00:00:00 +0000

What I learned this week

Software

How do I get in this tournament

oof

Business/Finance

South Korean stock market going crazy

rob granieri

Interesting Links

True spirit of gravel

The gambling state

Watching a video about the harms of gambling that we preempted by a Kalshi ad telling me to bet on things

Math/Stats

Travel

Other

What I learned this week 10-25-2025

Sat, 25 Oct 2025 00:00:00 +0000

What I learned this week

Software

Worked through these, pretty fun!

SSH hardening

TV signals

Business/Finance

Kind of accurate?

Interesting Links

Everything is gambling

Math/Stats

Travel

Other

What I learned this week 10-18-2025

Sat, 18 Oct 2025 00:00:00 +0000

What I learned this week

Software

Redshift observatory

Foundational mental model to get the hang of ever “new” LLM conceptual churn:

An LLM
…called in a loop
…maintaining a history of stuff it’s done in the session (the “context”)
…with access to tool calls to do things. Like, read files, write files, call bash, etc.

Business/Finance

A breakdown of “if I call myself ‘x’ or manager of ‘x’ I make more Money”

How does a single person have the authority to carry this out

LINQ is cool

Interesting Links

Motorcycles

I hate Turbo tax so much, please use this

AI is commoditizing code. Technical execution is table stakes rather than a competitive advantage. AI can generate code, but it can’t refactor the organizational reality customers have built around your architecture—the workflows, integrations, and institutional muscle memory that compound over time. * Is code really comoditized? Or is just boiler plate comoditized? But… wasn’t it already comoditized?

Math/Stats

Travel

Other

Could be fun to create a visual graph of all money-stuff posts

Stop putting screens on things that don’t need screens

What I learned this week 10-11-2025

Sat, 11 Oct 2025 00:00:00 +0000

What I learned this week

Software

bashrc vs bash_profile

A Linux contribution

Business/Finance

Starbucks protein

Back to supersonic flight

Interesting Links

Procrastination

Ultra processed ok in moderation

Hanja vs Hangul

Math/Stats

Travel

One bag travel

Other

A State of Inevitable Gambling

Mon, 06 Oct 2025 00:00:00 +0000

This is a collection of semi-coherent, random thoughts I have on “gambling”.

The inevitable gambling state

My brokerage wants me to gamble on sporting events
When I watch sporting events I’m bombarded with ads to gamble at {insert sportsbook}
My credit card wants me to gamble
McDonald’s wants me to play Monopoly and hope for a lottery like payoff if I buy their food
Go hang out at a gas station for a while and see how many people go in and buy scratch offs
Prediction markets want me to gamble
There’s someone out there looking to take the other side of my bets in the stock market
Crypto… do I even need to?
I’m getting ads on Venmo to join DraftKings and buy lottery tickets

There are so many people in the business of flipping a coin, heads they win 52 cents tails you win 48 cents. They’ll disguise this bet as a really fun thing, and they’re going to socially engineer you to make this bet as many times as possible. Because the MORE they can scale this bet the more money they win, they want you to put this bet on every second of every hour of every day to speed up the rate of wealth transfer.

The business model of a casino continues to march into consumer products.

Sign up to get emails and we’ll let you spin a wheel for a prize (some of which are great)
Buy our food and you could win a fucking car
You should pick who’s going to win the next presidential election!
You should buy this stock, it’s going to 10x in the next 3 months

To be fair, with enough hype and social lubrication people are willing to buy anything.

Random aside on NFTs

Side note, why the fuck would anyone rugpull NFTs? Like if you get a bunch of people to buy your NFTs isn’t the scam already complete? Just give them their stupid internet pictures, collect your money and allow them to trade on the marketplace. The valuation is now utterly useless to you, you’ve already perpetuated your scam? The hard part is already done, you’ve already convinced people that something useless is valuable and that you can generate the useless thing…

These people know that in aggregate they WILL make money off these promotions. Just like sportsbooks have all the little tools to keep you coming back, making it habit… the same tools surround us everywhere.

Is it possible to go a day without being bombarded by gambling?

The problem

Here’s the hard part… gambling is fun and, at the tail, people make money.

If you can refrain from all the garbage on Sportbooks you can take advantage of all the free money they offer
It’s clear that sharp bettors can make money from sportsbooks or, at least, perform arbitrage across books
Similarly, sharp people trading equities, commodities, etc… can make money over time (a lot of money) a.k.a Rentec
You can get really good at poker and… make money.
You can go to Kalshi with insider information, trade on it, and make a ton of $ while making markets more efficient

But now that you’ve gotten a taste you realize what you’ve become… you are now the one making 52 cents every time the coin flips.

THe problem? You exist on a PLATFORM. Just because you want to flip the coin… doesn’t mean you’re going to be able to anymore.

You will get banned/limited by Casinos/Sportsbooks
Is insider trading welcomed or lambasted in prediciton markets?
People don’t have to play poker with you or take the other side of your bets (I massive part of poker is playing at tables where you have an edge)
You may also fully exploit your advantage and not be able to continue to make money from it (Renaissance Technologies’ flagship Medallion fund does not accept outside money)

So you win 52 cents on borrowed time, or you continue to exploit people who don’t understand what kind of bets they are putting on.

I guess taking advantage of misinformed people/organizations is part of operating in a free market. In order to properly allocate capital there must be some winners and losers and in the long run we (hope to) lift wealth, life expectancy, and improve our existence.

If you’re a non-gambler you’re also kind of tied to the gambling ship. As much as you want to say, “Oh I’m VT and forget, I max out my retirement accounts, I never even look at my portfolio, etc…” Yet, if the a bubble pops your downturn will be large and hard to stomach just as those who are leveraged out of their minds in risky assets. You could make an argument that something like VT would be less destroyed than a concentrated bet in an devastated asset class and you’d be 100% correct (I hope), but you’re still on the same ship… you will feel the pain.

Personal thoughts

I like gambling. I know enough to understand when I’m making a terribly negative expected value decision and essentially just handing cash over to somebody… and I avoid that at all costs.

You can derive the sense of winning, the sense that I “knew” something that somebody else didn’t and while mindlessly, clicking, odds and offerings and spinning wheels certainly provides dopamine. There’s another level of dopamine from being right.

What are you going to compare this to?

Mon, 06 Oct 2025 00:00:00 +0000

A new project

You’re setting off on a journey to build a new {table, whole series of tables, dashboard, predictive model, pick your adventure book, etc…}. The initial request from THE BUSINESS lacks any semblance of detail required for immediate execution. You’ve done this before, people love quick calls to discuss the minutiae (a.k.a any level of detail) that was left out of the request. Lets hop on a quick call.

Things to remember when beginning “$project”:

Entitlement:

There will probably be no specific “entitlement”, “$ benefit”, etc… associated with the request, if you ask for this you will get:

“I don’t know”
“this will save x hours per week”
a round number in the $ millions
“let me circle back with you on that” (You will not hear about it again)

Context:

The person you are talking with may have less context then you. If this is true, you will most likely need to approach the most tenured member of their team to understand what is going on/why this is being requested. This is important if you can find the person who actually understands the broader picture, you will be able to sort out hours of hand waving in minutes.

Urgency:

Remember, everything is high priority and needed to be done yesterday. Wether the request “saves $100MM” or “saves me 15 minutes per week” it will be something urgent for the requestor. This is because:

In a race for constrained bandwidth, teams will continue to increase the unfounded urgency of requests to get things into a sprint, this causes the “urgency” to lose its meaning
People default to a general sense of what I’m working on is dire, important, and most be completed… this can be internally driven or imparted by point 1. above
Information is siloed causing large changes to remain unknown until just before/after implementation. This inflates a sense of urgency as things literally needed to be done yesterday

Create a graph:

A graph can be quite helpful in determining what you’ll actually need to do to complete this. I typically use Obsidian and jot down notes in markdown files and then link them as needed to create my graphs.

These notes are my scaffolding for how I’d go about executing the request. What information, alignment, data, sign off, etc… will I need to do this? if I had all of that right now, how would I execute this, what tools would I use, etc… What does the end result look like, how does it operate, whats the UX like?

This will NOT cover everything that comes up… it serves as a base to begin to cover edge cases and open questions that you’d like to establish an answer to now rather than weeks into this thing.

The Questions you ask:

What are you going to compare this to?

One of mt favorites gets to the heart of what you need to do.

If you’ve created a graph above there is a good chance you may have put more thought into the request than the requestor has, thus these questions may be just as important for them as it is for you.

SQL Buckets

Sun, 05 Oct 2025 00:00:00 +0000

Case Statements:

A common occurrence when writing SQL is needing to bucket some data and look at how it is distributed across said buckets.

Typically I’d write (have an LLM spit out buckets) something like this:

SELECT CASE WHEN COLUMN_TO_BUCKET < 5 THEN 'a. < 5'
            WHEN COLUMN_TO_BUCKET < 10 THEN 'b. >=5 - <10'
            WHEN COLUMN_TO_BUCKET < 15 THEN 'c. >=10 - <15'
            ETC...
            ELSE 'u. >100' END AS COLUMN_TO_BUCKET_DIST
      , SUM(THIS_IS_ANNOYING) AS ANNOYED
FROM TABLE
GROUP BY 1

I have to add a character/number in each bucket to allow for sorting, making writing/updating the lengthy case statement that much more annoying. I have to write all this because unlike other OLAP DBMS, Redshift doesn’t have a bucketing function that I’m aware of.

Or do I…

Solution:

Let’s say I have a table that records when a record was created and when it was last updated. I’ve never used this table before, and I’m interested in the distribution of the number of days between created and updated to see how common it is for a record to have an update n days after it was created.

CREATE TEMP TABLE #DELTA_DIST AS (
SELECT A_UNIQUE_ID_OF_A_ROW
     , DATE_DIFF('DAY', CREATION_DATE, LAST_UPDATED) AS CD_LD
FROM MY_TABLE
WHERE A_FILTER >= '2025-01-01'::DATE
  AND ANOTHER_FILTER = 1
);

Cool, so we want to know how these records look like in some buckets!

We could do this:

SELECT CASE WHEN CD_LD < 5 THEN 'a. < 5 Day'
            WHEN CD_LD < 10 THEN 'b. 5-10 Day'
            WHEN CD_LD < 15 THEN 'c. 10-15 Day'
            WHEN CD_LD < 20 THEN 'd. 15-20 Day'
            WHEN CD_LD < 25 THEN 'e. 20-25 Day'
            WHEN CD_LD < 30 THEN 'f. 25-30 Day'
            WHEN CD_LD < 35 THEN 'g. 30-35 Day'
            WHEN CD_LD < 40 THEN 'h. 35-40 Day'
            WHEN CD_LD < 45 THEN 'i. 40-45 Day'
            WHEN CD_LD < 50 THEN 'j. 45-50 Day'
            WHEN CD_LD < 55 THEN 'k. 50-55 Day'
            WHEN CD_LD < 60 THEN 'l. 55-60 Day'
            WHEN CD_LD < 65 THEN 'm. 60-65 Day'
            WHEN CD_LD < 70 THEN 'n. 65-70 Day'
            WHEN CD_LD < 75 THEN 'o. 70-75 Day'
            WHEN CD_LD < 80 THEN 'p. 75-80 Day'
            WHEN CD_LD < 85 THEN 'q. 80-85 Day'
            WHEN CD_LD < 90 THEN 'r. 85-90 Day'
            WHEN CD_LD < 95 THEN 's. 90-95 Day'
            WHEN CD_LD < 100 THEN 't. 95-100 Day'
            ELSE 'u. >100' END AS CD_LD_DIST
, COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

That is pretty annoying… it is verbose and, if you’ve done this before, you’ll know that catch all bucket at the end will have some spike in records in it that you’ll want to further investigate. So I came up with the following:

SELECT LPAD(FLOOR(COLUMN_TO_BUCKET / 5::FLOAT)::TEXT, 2, '0') -- Creates the sortable digits at the beginning
            + '. ' +
       FLOOR(COLUMN_TO_BUCKET / 5::FLOAT) * 5 -- Generates the first day in the range
            + '-' +
       (FLOOR(COLUMN_TO_BUCKET / 5::FLOAT) * 5 + 4) -- Generates the last day in the range  (+4 as I'm bucketing by 5 days at a time)
            +
       ' Days' AS COLUMN_TO_BUCKET_DIST -- Adds the Days string at the end for clarity
     , COUNT(DISTINCT THING_TO_AGGREGATE) AS AGGREGATED_THING
FROM TABLE_I_WANT_TO_DISTRIBUTE
GROUP BY 1

We can pass in the column to bucket, adjust the bucket size, and also handle the number of buckets. This does use LPAD but FLOOR() is standard. Regardless, lets break it down in detail if you need to rewrite for a different SQL dialect.

FLOOR: rounds a number down to the next whole number
LPAD: prepends characters to a string

On an input of “1” this is what the flow would look like to go from 1 -> 00. 0-4 Days

-- FLOOR of 1/5 = 0 and cast to text
SELECT FLOOR(1 / 5::FLOAT)::TEXT -> '0'
-- LPAD adds a leading zero to single digits
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') -> '00'
-- Just cosmetic, appending a "."
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. '  -> '00.'
-- Now we know that this will iteratively return 0,1,2,3,n... as our input increases
-- And if we multiple by 5 we get the beginning of our bucket
       -- 0 * 5 = 0 (for 0-4)
       -- 1 * 5 = 5 (for 5-9)
       -- etc...
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 -> '00. 0'
-- Now we add another cosmetic piece ( a dash to separate start/end of bucket)
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 + '-' -> '00. 0-'
-- Remember how we generated the start of the bucket 
-- If we know the bucket is going to span 5 days (inclusive of start and end) we just add 4 to the start
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 + '-' + (FLOOR(1 / 5::FLOAT) * 5 + 4) -> '00. 0-4'
--- Add final cosmetic piece ' Days'
SELECT LPAD(FLOOR(1 / 5)::TEXT, 2, '0') + '. ' + FLOOR(1 / 5::FLOAT) * 5 + '-' + (FLOOR(1 / 5::FLOAT) * 5 + 4) + ' Days' -> '00. 0-4 Days'

We’ve transformed from an input of CD_LD = 1 and assigned it a sortable, bucketable (probably not a word) value -> ‘00. 0-4 Days’

I’m worried about an unconstrained series of buckets… how do I cap it?

SELECT  -- Define our buckets when we are below our threshold
       CASE WHEN CD_LD < 100 THEN LPAD(FLOOR(CD_LD / 5::FLOAT)::TEXT, 2, '0')
            || '. ' ||
       FLOOR(CD_LD / 5::FLOAT) * 5
            || '-' ||
       (FLOOR(CD_LD / 5::FLOAT) * 5 + 4)
            ||
       ' Days'
       
       -- Define final bucket when threshold exceeded
       ELSE LPAD(FLOOR(100 / 5)::TEXT, 2, '0')
          || '. >100' END AS CD_LD_DIST
     , COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

I want to change the increment size

-- Change increment size from 5 days to 7 days
-- 5 -> 7
-- 4 -> 6
SELECT LPAD(FLOOR(CD_LD / 7::FLOAT)::TEXT, 2, '0')
            || '. ' ||
       FLOOR(CD_LD / 7::FLOAT) * 7 --
            || '-' ||
       (FLOOR(CD_LD / 7::FLOAT) * 7 + 6) -- Generates the last day in the range  (+6 as I'm bucketing by 7 days at a time)
            ||
       ' Days' AS CD_LD_DIST -- Adds the Days string at the end for clarity
      , COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

I have more than 99 buckets, the two digit solution wont work:

-- Pad an additional 0 with LPAD
-- 
SELECT LPAD(FLOOR(CD_LD / 5::FLOAT)::TEXT, 3, '0') -- 2 -> 3
            || '. ' ||
       FLOOR(CD_LD / 5::FLOAT) * 5 
            || '-' ||
       (FLOOR(CD_LD / 5::FLOAT) * 5 + 4) 
            ||
       ' Days' AS CD_LD_DIST -- Adds the Days string at the end for clarity
      , COUNT(DISTINCT A_UNIQUE_ID_OF_A_ROW) AS NUM_ID
FROM DELTA_DIST
GROUP BY 1

Hope that’s helpful! Maybe there’s a better solution, I didn’t do an exhaustive search!

What I learned this week 10-04-2025

Sat, 04 Oct 2025 00:00:00 +0000

What I learned this week

Software

HTML Regex parsing, at least an LLM would have said “great idea”

Future proof file format

Tiger beetle DB

Business/Finance

Hmmmmmm

Interesting Links

Our efforts define us

Volcano

Math/Stats

Travel

Greenland

Other

CFB predictions

What I learned this week 09-27-2025

Sat, 27 Sep 2025 00:00:00 +0000

What I learned this week

Software

Shaper looks interesting

Homomorphic Encryption

Fully Homomorphic Encryption (FHE) is a cryptographic scheme that enables computations to be performed directly on encrypted data, as if the data were in plaintext. After all computations are performed on the encrypted data, it can be decrypted to reveal the result.

Business/Finance

Interesting Links

Haribo power bank

Automaticity “Highway hypnosis”

iPhone air looks… bendable

Math/Stats

Travel

Other

Jazz

The Goal

Sat, 20 Sep 2025 00:00:00 +0000

The book

This will serve to store notes/insights/thoughts about The Goal by Eliyahu M. Goldratt

Quotes

“But after a while, I learned not to waste my time, checking the numbers because the numbers were almost always right. However, if I checked the assumptions, they were almost always wrong.”

Notes

Overall reminded me of Supply Chain class.

I like the process evaluation, it was funny that you can actually draw parallels between manufacturing and software development at large organizations.

Bits of the story were a little cheesy, but I guess that’s to be expected.

What I learned this week 09-20-2025

Sat, 20 Sep 2025 00:00:00 +0000

What I learned this week

Software

Business/Finance

TDF rankings

Dominated ETFs:

“As a case in point, the iShares US Technology ETF (IYW) is consistently dominated by the Technology Select Sector SPDR Fund (XLK) and the Vanguard Information Technology Index Fund (VGT). At the end of our sample, IYW generates an annualized $18 million in fees with a 0.43% expense ratio on $4 billion in assets. XLK and VGT generate $28 million and $20 million in fees, respectively, despite their much larger sizes of $22 billion and $20 billion in assets. Even though SPDR and Vanguard attract five times the assets with significantly lower expense ratios of 0.13% and 0.10%, the three funds are each profitable and generate similar fee income.”

Rug pullling…. Yourself:

Interesting Links

Grapevines = plastic

Deal making at its finest

Math/Stats

Travel

Other

Sodium citrate helps prevent the formation of kidney stones

“Humor can be dissected as a frog can, but the thing dies in the process and the innards are discouraging to any but the pure scientific mind.”

― E. B. White

What I learned this week 09-13-2025

Sat, 13 Sep 2025 00:00:00 +0000

What I learned this week

Software

10/10 fishing

Highest Paid Person’s Opinion

Business/Finance

Spotify/Robinhood want to control more social real estate

Health care arb because stupidity:

Interesting Links

NFL helmet deep dive

Security Theater

Math/Stats

Travel

Other

Steamed brisket

What I learned this week 09-06-2025

Sat, 06 Sep 2025 00:00:00 +0000

What I learned this week

Software

md files to make llm go brrrr

Business/Finance

Interesting Links

Ablution

Math/Stats

Travel

Other

Baked ribs

What I learned this week 08-30-2025

Sat, 30 Aug 2025 00:00:00 +0000

What I learned this week

Software

Nimble and Lance vs parquet

One of the most well designed blog

Interesting thoughts on booleans

Business/Finance

An argument for the US stake in intel

RIP llm subscription mode

Full circle back to physical currency

Interesting Links

What a cool sport

Strawberry TIL

Got to experience this haboob

Red queen’s race

Laser peening

Longest name

Math/Stats

Travel

Other

Because transparency is of prime importance, the healthy cornea does not have or need blood vessels within it. Instead, oxygen dissolves in tears and then diffuses throughout the cornea to keep it healthy: https://en.m.wikipedia.org/wiki/Cornea

A VM makes an OS believe that it has the machine to itself; a container makes a process believe that it has the OS to itself.

What I learned this week 08-23-2025

Sat, 23 Aug 2025 00:00:00 +0000

What I learned this week

Officially one year of doing this!

Software

OLAP ORMS

Whitespace could be the language of the future

Semantic layers… if your org can stomach it

Filter first

Business/Finance

$enterprise

Yes, tracking down owners of things IS that hard

America!, hell yeah!

Interesting Links

“As real returns compress, risk increases to compensate.”

Bases in Obsidian

Math/Stats

A little further into derivatives

Travel

Other

Gigantic jet

Robinhood Gambling

Fri, 22 Aug 2025 00:00:00 +0000

The Robinhood gambling factory

Robinhood has a nice mobile experience, they have good products/deals, and they also have a credit card.

There has been a lot of talk about Robinhood engaging with human “gambling instincts”.

Just search “how has Robinhood gamified investing” and you can watch confetti rain down as someone yolos their life savings into GME 0 DTE calls

Again, I’m not saying the product is all bad, but there is one piece which is astonishingly ridiculous to me.

The Credit Card

I’m not going to write about the Casino-like aspects of the brokerage product… but there is also the Robinhood CREDIT CARD!

I opened the card a few months ago, payed the $50 Gold annual subscription, applied, and got the card. The card is kind of nice… the gold is a little gimmicky, but the 3% back on everything and 5% on travel is good for someone that doesn’t want to swap cards around for each kind of purchase.

I can just nail down 3%, convert those points to cash that lands straight back into Robinhood, and earns 4% APY. Am I leaving a few bps on the table… sure, but everything is contained in the same ecosystem, transfers are fast, and 4% APY on my uninvested cash is fine.

The Mystery Box

For the first time, I go to exchange my points for cash, 1 point = 1 cent. I’m not going to fuss around with gift cards/shopping stuff, I want my cash, and I want to go make 4% on it.

After opening up the rewards tab, I’m forced to scroll ALL THE WAY TO THE BOTTOM to get to the “Cash Back” section. I’m scrolling through shop with points, travel with points, buy Robinhood merch with points and Low and Behold I see “The Mystery Box”… what the fuck is “The Mystery Box”.

Well, I’m glad you asked, and you can retrieve the full details here… but, in short, the Mystery box is LITERALLY an online casino.

That’s right, You can shuttle 1000 points (~$10) into this thing and come out with… less than $10! Amazing, lets break it down.

A few things to start, the odds do sum to 100%, so you’re guaranteed to win one of these things. Each win is independent and the odds don’t change as far as I can tell. So lets look at expected value:

“Cash” Prizes:

10,000 Points @ 0.50%: 10,000 Points = ~$100, $100 * 0.50% = 50 cents
7,500 Points @ 0.75%: 7,500 Points = ~$75, $75 * 0.75% = 56.25 cents
5,000 Points @ 1.50%: 5,000 Points = ~$50, $50 * 1.50% = 75 cents
2,500 Points @ 2.50%: 2,500 Points = ~$25, $25 * 2.50% = 62.5 cents
1,500 Points @ 4.00%: 1,500 Points = ~$15, $15 * 4.00% = 60 cents
1,250 Points @ 5.00%: 1,250 Points = ~$12.50, $12.50 * 5.00% = 62.5 cents

Total EV: $.50 + $.5625 + $.75 + $62.5 + $.60 + $.625 = $3.6625

Hmmm… I hope we catch up a little bit on the cashback prizes.

Cashback Prizes:

A few notes on cashback:

We must factor in the fact that we would have earned 3% on these purchases anyway, thus, we have to take cashback% minus 3% to calculate the true net benefit
To maximize the value you must spend up to the cap. This means you must WANT to purchase from where you win and be LIQUID/WILLING enough to make said purchase e.g. spend $3500 in the next 24 hours.

Assumptions:

I’m going to assume the opportunity cost of capital is $0, this is especially generous as the $500 you spend at Target could be used for investing in 0 DTE options on… Robinhood!…
For prizes with no Cap, I’m going to assume average cost (if I can find it) + 25%, otherwise just going to insert a reasonable guess
For prizes with a Cap, I’m going to assume the reward is maximized
I’m going to assume the above conditions hold for 33% of the items below. This is a big assumption, but it is fair given the variety of options and the previous conditions. This means we’ll cut our EV calculation for this section in 1/3 when it’s all said and done.

Disagree with my assumptions? I simulated the whole thing so you can test for yourself and see if it’s worth it for you. This is actually ripping the rewards straight from api.robinhood.com (and doing so poorly), so it will inevitably break at some point, but it’s fun for now!

10% cashback on next grocery purchase ($500 cap) @ 5.00%: $500 * (10-3)% = $35 * 5% = $1.75
10% cashback on next automotive gas purchase ($300 cap) @ 6.00%: $300 * (10-3)% = $21 * 6% = $1.26
5% cashback for 24 hours ($3,500 cap) @ 4.00%: $3500 * (5-3)% = $70 * 4% = $2.8
10% cashback on next Uber or Lyft ($300 cap) @ 7.00%: $300 * (10-3)% = $21 * 7% = $1.47
15% cashback on next Starbucks purchase @ 8.00%: $25 * (15-3)% = $3 * 8% = $.24
4% cashback for 24 hours ($3,000 cap) @ 4.50%: $3000 * (4-3)% = $30 * 4.5% = $1.35
5% cashback on next Amazon.com purchase ($500 cap) @ 8.00%: $500 * (5-3)% = $10 * 8% = $.80
5% cashback on next restaurant order ($500 cap) @ 8.00%: $500 * (5-3)% = $10 * 8% = $.80
5% cashback on next DoorDash or Uber Eats ($300 cap) @ 8.00%: $300 * (5-3)% = $6 * 8% = $.48
5% cashback on next Target purchase ($500 cap) @ 6.00%: $500 * (5-3)% = $10 * 6% = $.60
5% cashback on your next phone bill @ 6.00%: ($141 * 1.25) * (5-3)% = $3.525 * 6% = $.2115
5% cashback on your next Lululemon, Alo Yoga or Vuori purchase @ 8.00%: $300 * (5-3)% = $6 * 8% = $.48
4% cashback on next spa purchase @ 7.00%: $300 * (4-3)% = $3 * 7% = $.21
5% cashback for 30 days ($10,000 cap) @ 0.25%: $10000 * (5-3)% = $200 * .25% = $.5

Starbucks, Lululemon, Alo Yoga or Vuori, and Spa purchases I think are fair estimates. The phone bill at least has a source. Feel free to throw some different estimates in there and test it out for yourself.

To find the total EV, we will sum the individual values:

$1.75 (Grocery)
$1.26 (Gas)
$2.80 (5% for 24h)
$1.47 (Uber/Lyft)
$0.24 (Starbucks)
$1.35 (4% for 24h)
$0.80 (Amazon)
$0.80 (Restaurant)
$0.48 (DoorDash/Uber Eats)
$0.60 (Target)
$0.2115 (Phone Bill)
$0.48 (Lululemon etc.)
$0.21 (Spa)
$0.50 (5% for 30 days)

And get: $12.9515 which we can divide by 3 to get $4.3172.

Adding this to our cash prizes, we expect an EV of $3.6625 + $4.3172 = $7.9797!

$10 in… $7.9797 out. You can play around with the assumptions but unless you’re ready to maximize every single one of the Cashback prizes… every single time you roll the dice on this Mystery Box you’re handing $2 back to Robinhood in the form of them not having to pay out the true value on their 3% rewards program.

This is the definition of a Casino. Robinhood will argue that the program is more than fair as it’s positive EV for someone willing to maximize every single one of these rewards. We can go around in circles about this, but if you land on the 5% cashback on next DoorDash or Uber Eats ($300 cap) @ 8.00% the maximum value one can accrue is .02*300 = $6…

Funny Thoughts (This is not legal or financial advice)

Ok so there are three uncapped offerings:

4% cashback on next spa purchase
15% cashback on next Starbucks purchase
5% cashback on your next Lululemon, Alo Yoga or Vuori purchase

Now if you haven’t head of Konstantin Anikeev let me introduce you. Konstantin curated this beauty:

Purchasing Gift Cards: Using his American Express 5% ‘cash back’ Rewards Program, he Purchased Visa gift cards at grocery stores or pharmacies where the 5%Rewards applied.
Although Visa cards cannot be converted to cash, they can be used to purchase Money Orders. He would purchase the Money Orders with the Visa gift cards.
Deposit the Money Orders to his Bank Account.
Pay the Amex Balance.
Repeat to the tune of $6.4MM in purchases and $320,000 in ‘profits’ by converting has 5% cashback in excess of fees

There’s also an interesting tax ruling in Konstantin Anikeev v. Commissioner, but this should now draw your eyes to that uncapped 15% starbucks purchase. The 5% and 4% uncapped offerings are interesting, but 15%…

Lets say we call up Starbucks and tell them we have a $20,000 limit on a card paying 15% on our next purchase, how could they turn down this opportunity. Better yet, what if Starbucks wanted to outfit a few stores in Lululemon and Lululemon wanted to buy starbucks for its employees… we could facilitate this trade, in which case I’d be hitting the mystery box like crazy.

Finger Under the Page

Sat, 16 Aug 2025 00:00:00 +0000

Reading

I tend to put my fingers under the right page of a book in a rush to get to the next page. As soon as I do it, I’m already thinking about the next page and struggle to focus. Other times I’m not even thinking about the book… just reading on auto pilot, paragraphs slipping by.

I’ve been catching myself doing this more. I can’t decide if it’s

a sign to move on from the book or
a sign that my mind can’t (doesn’t want to) focus on reading.

As a catch myself doing this I re-read the last few paragraphs. If it keeps happening, I then have to decide between 1 & 2. I don’t want to force myself to read a good book when I don’t feel like reading, and I don’t want to force myself to read a book cover to cover just because “I started it”.

Driving

I drive between Phoenix, AZ and Tucson, AZ a lot. 100 miles on the I-10, pretty much straight, set the cruise control and never touch the brake kind of drive.

My Aunt came over for her birthday, and we talked about driving the speed limit. I recalled my false sense of urgency on the I-10 in a rush to get from one city to another. Someday I’ll reminisce about the drive. Maybe I’ll wish I hadn’t allowed it to blend into a meaningless annoyance only prolonged by slow drivers taking up the left lane.

I’m going to try and drive the speed limit. I’m going to do the drive no podcasts or music. Time to mix it up.

School

The first week of school was always really slow. New classes, new teachers, new peers, etc… then everything normalizes and it’s like someone takes the speed slider and scrubs it to the max and you’re in the rinse cycle of the core of the semester.

Then you open your eyes and its done.

Months of tests, games, events, lectures, etc… can now be boiled down to a pin head in your brain, something that had previously fully occupied your schedule/life and contained nearly all forward looking events can be distilled down to “Fall Semester Sophomore year”… a time period you can only recall memorable events from, the mundane washed over.

Life

Then you hit this post-school period. There’s no arbitrary delineation of time periods like semesters. Sure a company will post quarterly earnings, but it doesn’t mark a time where you’re faced with entirely new course work with a new group of peers.

There’s not really anyone challenging you to try new things or join a club, your friends are further away and it’s a little more difficult to make new ones. It’s crazy how fast you can slide from Monday to Sunday if you pay attention (or don’t).

Result

Approach the mundane with a renewed sense of observation in the minutia or put your own spin in on it.

Try new things.

Don’t try and preemptively turn the page, enjoy the page. If you can’t enjoy the page, do something else you can focus on now.

Stop wasting time doing trivial bullshit that accelerates the coagulation of life into a blurry sense of “past”.

What I learned this week 08-16-2025

Sat, 16 Aug 2025 00:00:00 +0000

What I learned this week

Software

Business/Finance

Poly Market Arbing

Uh Oh Kodak

Leslie’s has some… uh… debt obligations

Interesting Links

Clams in poland

Math/Stats

Travel

Other

Alcohol

Creatine in cycling

Letters to a Young Poet

Sun, 10 Aug 2025 00:00:00 +0000

The book

This will serve to store notes/insights/thoughts about Letters to a Young Poet

Quotes

Xix- solitude is not merely a matter of being alone. It is a territory to be entered and occupied.

XX – when doubts arise, simply “school them”: “Instead of being demolishers they will be among your best workers”

Page 5 – things are not all as graspable and sayable as on the whole we are led to believe; most events are unsayable, occur in a space that no word has ever penetrated, and most unsayable of all are works of art

Page 13 – allow your verdict, their own quiet untroubled development, which like all progress must come from deep within and cannot be forced or accelerated

Page 14 – to be an artist means: not to calculate and count; to grow and ripen like a tree, which does not hurry the flow of sap and stands at ease in the spring gales without fearing that no summer may follow.

Page 22 – but your solitude, even in the midst of quite foreign circumstances, will be a hold and a home for you, and leading from it you will find all the paths you need

Page 32 - we know little but that we must hold fast to what is difficult is a certainty that will never forsake us. It is good to be alone, for solitude is difficult; that something is difficult should be one more reason to do it.

Page 40 – just as for a long time people were deceived about the movement of the sun, so we are still deceived about the movement of what is to come. The future is fixed, dear Mr. Kappus, but we move around in infinite space.

Page 42: perhaps everything terrifying is deep down a helpless thing that needs our help.

Notes

I liked it, some went over my head and some of it didn’t.

What I learned this week 08-09-2025

Sat, 09 Aug 2025 00:00:00 +0000

What I learned this week

Software

tor history

Business/Finance

Germany exit taxes…incentives…

Interesting Links

Oops

a little transparency goes a long way

Math/Stats

Travel

Other

Waist to height ratio better than bmi?

MOGENS by Jens Peter Jacobsen, I’d like to read

Don’t start with the thank you.

Equivalent speed

Reese + Oreo, incredible

What I learned this week 08-02-2025

Sat, 02 Aug 2025 00:00:00 +0000

What I learned this week

Software

Ubuntu is an ancient African word meaning ‘humanity to others’

“With a sufficient number of users of an API, it does not matter what you promise in the contract: all observable behaviors of your system will be depended on by somebody.”

Business/Finance

Interesting Links

Taking a break

To make a prairie it takes a clover and one bee, One clover, and a bee. And revery. The revery alone will do, If bees are few.

Emily Dickinson:

Math/Stats

Travel

Other

Nan-in (南隠), a Japanese master during the Meiji era (1868-1912), received a university professor who came to inquire about Zen. Nan-in served tea. He poured his visitor’s cup full, and then kept on pouring. The professor watched the overflow until he no longer could restrain himself. “It is overfull. No more will go in!” “Like this cup,” Nan-in said, “you are full of your own opinions and speculations. How can I show you Zen unless you first empty your cup?”

“Expression, carrying the elegance that comes from compressed energy, is like a perfectly tuned stringed instrument: the strings are taut to exact degrees to correspond to exact pitches that stand in exact relations to one another. The pitches and their relations exist before the strings are tuned. Each string is tuned to a pitch and the results are judged by comparison to this pre-existing reality. A musician tuning an instrument is not finished until everything is exactly right, but once it is exactly right, there is simply nothing left to do.” – Clear and Simple as the Truth: Writing Classic Prose, Thomas, Turner.

What I learned this week 07-26-2025

Sat, 26 Jul 2025 00:00:00 +0000

What I learned this week

Software

Setting up a new mac

Addded Raycast to my Mac and it’s quite nice

Quick start mcp

Zip Bomb server protection

don’t just red docs, also talk to them

Interesting, pretty undeniable use case for LLMs that’s effective

zprof

Letting Q run wild

Business/Finance

Interesting Links

Math/Stats

Travel

Other

What I learned this week 07-19-2025

Sat, 19 Jul 2025 00:00:00 +0000

What I learned this week

Software

Cloudflare Tunnels interesting to check out

How a screen works

Local LLM

Business/Finance

Is this a “we are not a monopoly” play?

AI Capex

Interesting Links

New Anti-biotic

Math/Stats

Travel

Other

What I learned this week 07-12-2025

Sat, 12 Jul 2025 00:00:00 +0000

What I learned this week

Software

Put Doom on a Peloton:

Found the Wifi Debugging screen
Downloaded Android SDK Platform-Tools and used adb to connect and inspect the android tablet. ./adp pair and then ./adp connect and ./adb shell to get a shell
Can even take a screenshot and pull it back onto my computer screencap /sdcard/myscreen.png & ./adb pull /sdcard/myscreen.png
I side loaded Nova Launcher by downloading the .apk and using ./adb install
Then for doom, I found another .apk for Doom and grabbed doom1.wad from the Doom wiki. Pushed the .wad to a new doom directory I created on the tablet ./adb push doom1.wad /sdcard/Doom/ and then installed the doom apk.

And then we had doom:

Everything curl!

Dig

uname -a to get operating system info

Business/Finance

Interesting Links

When the Credit Cards start rolling out you know Crypto is BACK!

Math/Stats

Travel

Other

What I learned this week 07-05-2025

Sat, 05 Jul 2025 00:00:00 +0000

What I learned this week

Software

What’s security for anyway?

How NAT Traversal Works - Let’s say you’re making your own protocol and that you want NAT traversal. The protocol should be based on UDP. - For UDP, the rule is very simple: the firewall allows an inbound UDP packet if it previously saw a matching outbound packet. - The problems start when two of our “clients” want to talk directly. Now the firewalls are facing each other. According to the rule we established above, this means both sides must go first, but also that neither can go first. - So, to traverse these multiple stateful firewalls, we need to share some information to get underway: the peers have to know in advance the ip:port their counterpart is using. Tailscale has set up a coordination server for this purpose - Stateful firewalls have limited memory, meaning that we need periodic communication to keep connections alive. If no packets are seen for a while (a common value for UDP is 30 seconds), the firewall forgets about the session, and we have to start over. To avoid this, we use a timer and must either send packets regularly to reset the timers, or have some out-of-band way of restarting the connection on demand. - The most common use of SNAT is to connect many devices to the internet, using fewer IP addresses than the number of devices. In the case of consumer-grade routers, we map all devices onto a single public-facing IP address. - the same principle applies on corporate networks. The usual difference there is that the NAT layer consists of multiple machines (for high availability or capacity reasons), and they can have more than one public IP address, so that they have more public ip:port combinations to choose from and can sustain more active clients at once. - STUN relies on a simple observation: when you talk to a server on the internet from a NATed client, the server sees the public ip:port that your NAT device created for you, not your LAN ip:port. So, the server can tell you what ip:port it saw. That way, you know what traffic from your LAN ip:port looks like on the internet

IPv5

Pros/Cons of QUIC

Wireguard uses only UDP due in part to the TCP Meltdown Problem. Creator of Wireguard’s website is pretty wild

Port Forwarding

UNIX password manager

Sketch starts a docker container and outputs its work onto a branch in your host git repository.

Business/Finance

enshittification

But network effects cut both ways. If a system gets more valuable as it attracts more users, it also gets less valuable as it sheds users. The less valuable a system is to you, the easier it is to leave.
When switching costs are high, services can be changed in ways that you dislike without losing your business. The higher the switching costs, the more a company can abuse you, because it knows that as bad as they’ve made things for you, you’d have to endure worse if you left.

Interesting Links

IPv6 Adoption

Third, I need to help new programmers who never got to experience simple, pleasurable programming in a safe environment understand that programming can be fun.

Math/Stats

Travel

Other

What I learned this week 06-28-2025

Sat, 28 Jun 2025 00:00:00 +0000

What I learned this week

Software

Bad measures

What NPM Fund means

View ipv4/v6 Route tables: netstat -r

Typically for ISPs give out more than a /64 for ipv6 so people can have multiple networks (/60 or /56 possibly) /64 only allows for one ipv6 network so if you were to have multiple vlans at home you’d have some issues with a /64

IPv4

Business/Finance

Interesting Links

Dropping thousands of dollars on peoples head from a helicopter

Simpsons Paradox

Facts about hummingbirds

Math/Stats

Tetration! && Pentation!

Fun logic puzzles

Travel

Other

Pink fairy armadillo

What I learned this week 06-21-2025

Sat, 21 Jun 2025 00:00:00 +0000

What I learned this week

Software

HyperLogLog in Redshift

OLTP Snowflake

In Clickhouse primary keys do not represent a unique row, it represents the sort order of how data is stored in the file system for merge tree tables

Cool Animations

Business/Finance

A touch of bribery

Interesting Links

Rattlesnakes of Arizona

Bob is hard to come by_

Math/Stats

Geometric distribution: If you’ve flipped 10 tails in a row, you’re NOT “due” for heads. The next flip is still 50/50.

Exponential distribution : If you’ve waited 20 minutes for the bus, the next minute is just as likely to bring the bus as the first minute was.

Binomial Number of successes in n trials How many heads in 10 coin flips? Geometric Number of trials until first success

Travel

Other

Your Data Warehouse Ignores Your Primary Key

Thu, 19 Jun 2025 00:00:00 +0000

The agenda…

Today we’re digging into the curious case of “Primary Keys” in modern data warehouses (think Redshift, BigQuery, Snowflake). On the menu:

A high-stakes request from our VP
What happens when a PRIMARY KEY doesn’t do what you think it does
A hands-on experiment in Postgres to see why OLAP systems choose not to enforce a PK
A dash of data quality, some opinions, and a tangent or two

A little background

Lets take Redshift for example. In Redshift, primary keys are informational only; they are not enforced. However, the query optimizer will trust our declaration and use it to create more efficient execution plans. How silly of Redshift to trust us like that.

This is not shocking to the seasoned data warehouse connoisseur, but to the uninitiated this may look unnerving. So for anyone unfamiliar here’s what I mean.

Our VP needs to know

Our VP needs to know if Bob is hungry or not to understand whether our start up is doing good or bad.

Yikes, we deleted all our data when the U.S. Space Command was investigating us for fraud! We can’t tell our VP that we don’t know anything about Bob so lets insert some random records into our datawarehouse. We’ll need to start from scratch!

First lets make a new table and define a primary key.

-- REMEMBER! When Storing strings make sure to take up as much space as possible 
-- Defining these columns as CHAR pads strings with blanks to ensure we waste as much space as we can 
-- Everyone will love you for this, and you'll probably get promoted!
CREATE TABLE CHECK123 (
SOME_ID CHAR(4096) PRIMARY KEY, <-- Ahh, a nice PK
AN_ATTRIBUTE CHAR(4096),
ANOTHER_ATTRIBUTE CHAR(4096),
SOME_NUMBER INT4
);

Alright lets manually insert our correct data!

INSERT INTO CHECK123
VALUES ('abc123', 'Bob', 'Hungry', 2) <-- Hmmm...
     , ('wow', 'Big Tom', 'Fast', 86)
     , ('this', 'Dale', 'Thoughtful', 12)
     , ('is', 'Doug', 'Tired', 9)
     , ('pretty', 'Hammer time', 'Happy', 6)
     , ('annoying', 'The bug unit', 'Angry', 6)
     , ('abc123', 'Whopper', 'Sad', 6) <-- Hmmm...

And run our important query that our VP needs. We don’t have time for data quality checks so lets throw it into Excel and send it everywhere.

SELECT SOME_ID
     , COUNT(1)
FROM CHECK123
GROUP BY SOME_ID;

abc123,2 -- UH OH!!!!
wow,1
this,1
is,1
pretty,1
annoying,1

SELECT DISTINCT SOME_ID FROM CHECK123;

abc123 -- UH OH!!!!
wow
this
is
pretty
annoying
abc123  -- UH OH!!!!

Oh no… we have duplicate records even after writing SELECT DISTINCT. But I thought SELECT DISTINCT would remove duplicates? Well, yes, but… we declared SOME_ID as a PRIMARY KEY, effectively telling Redshift, “I solemnly swear this column contains no duplicates.” The query optimizer, trusting us completely, chooses the most efficient execution plan: it just scans the column and hands it back without performing a costly deduplication step. Why would it? We already promised it was unique.

Imagine someone gives you a book with 1,000,000 unsorted names in it and tells you they are unique. Then they say, “cross out any duplicate names and hand the book back to me”. You’re initial reaction is to immediately throw that book back as checking for uniqueness of one million names sounds… well… awful.

In this case, the query optimizer is just handing us the book back. If we were to run:

EXPLAIN SELECT DISTINCT SOME_ID FROM CHECK123

We’d get something back like:

XN SEQ SCAN on check123 (cost=0.00..0.07 rows=7 width-304)

That’s just a SEQ SCAN handing us back the entire column because we’ve already told Redshift these values are unique. In contrast, what if we scan a non-PK column for uniqueness:

SELECT DISTINCT AN_ATTRIBUTE FROM CHECK123;

And and there you have it, now we get:

XN Unique  (cost=0.00..0.09 rows=7 width-304)
    -> XN SEQ SCAN on check123 (cost=0.00..0.07 rows=7 width-304)

You’ll find XN Unique in query plans when DISTINCT or UNION (which implies distinctness, unlike UNION ALL) are used. Our DBMS now returns a unique result set from values it does not know are unique. I go on a tangent about this below if you’re interested.

Click to view a tangent on XN Unique & XN HashAggregate

If we have an index on the column deduplication can be fast, I’ll link the Postgres docs on B-TREE Indexes specifically the section on deduplication. But what if we don’t have an index… Redshift doesn’t even have indexes… Redshift just has metadata (min-max values) for blocks written to disk called Zone Maps.

Redshift docs notes that XN Unique: “Removes duplicates for SELECT DISTINCT queries and UNION queries.” This gets weird because I’d assume with this operator that we are sorting and plucking unique values from this column. However, there is no XN Sort in that plan. XN Sort:“Evaluates the ORDER BY clause and other sort operations, such as sorts required by SELECT DISTINCT queries. Our SELECT DISTINCT query above doesn’t have an XN Sort it only contains an XN Unique, maybe XN Unique implies an XN Sort? I generated an explain plan for a relation whose rows were 100% unsorted according to STV_TBL_PER and that plan did NOT include an XN Sort either…

I think a couple things could be happening:

The optimizer can choose between different strategies to perform our SELECT DISTINCT and in these cases it’s actually choosing a hashing strategy but just labeling it as XN Unique.
XN Unique implies a sorting operation, and we’re just not being shown that in the plan

For example if we take:

SELECT DISTINCT AN_ATTRIBUTE FROM CHECK123;

And re-write it as:

SELECT AN_ATTRIBUTE FROM CHECK123 GROUP BY AN_ATTRIBUTE;

We get:

XN HashAggregate  (cost=0.09..0.09 rows=7 width-304)
    -> XN SEQ SCAN on check123 (cost=0.00..0.07 rows=7 width-304)

This is similar to XN Unique but the costs are different cost=0.00..0.09 & cost=0.09..0.09 respectively. The first number is the relative cost of returning the first row for this operation. The second value, in this case 0.09, provides the relative cost of completing the operation. So I’d surmise that depending on how we write this query:

XN HashAggregate is a blocking operator designed for GROUP BY clauses; it must consume its entire input to correctly compute aggregate functions like COUNT() before returning any rows.
XN Unique is a specialized operator for SELECT DISTINCT that can implement a streaming (non-blocking) hash algorithm, allowing it to output unique rows as soon as they are encountered.

This seems to explain why Unique can have a zero startup cost while HashAggregate’s startup cost reflects processing its entire input.

On another note, I’m only able to get an XN Sort operator via explicitly using ORDER BY:

SELECT DISTINCT AN_ATTRIBUTE FROM CHECK123 ORDER BY AN_ATTRIBUTE;

XN Merge  (cost=1000000000000.19..1000000000000.20 rows=7 width=304)               
  Merge Key: an_attribute                                                          
  ->  XN Network  (cost=1000000000000.19..1000000000000.20 rows=7 width=304)       
        Send to leader                                                             
        ->  XN Sort  (cost=1000000000000.19..1000000000000.20 rows=7 width=304)    
              Sort Key: an_attribute                                               
              ->  XN Unique  (cost=0.00..0.09 rows=7 width=304)                    
                    ->  XN Seq Scan on check123  (cost=0.00..0.07 rows=7 width=304)

Anyway, our VP is pissed because we sent trash, but why would an OLAP DBMS do something like this?

PK in name only

We can see that our primary key’s exist in name only. The DBMS is not going to enforce them, its going to assume they are right, and if you want to enforce them then you’ll need to write that logic yourself.

So what could be the reasons why thoughtful and well payed, Engineers, TPMs, Directors, and VPs building these cloud data warehouses decided to allow people to designate primary keys that aren’t enforced? I’ll boil the arguments down to the following:

Write performance

What do we mean by write performance, and can we actually take a look at a DBMS and see this impact on performance?

Yes, we can:

spin up Postgres inside of a container
create tables with/without PK
write data to said tables
measure performance
and even look a little deeper at what Postgres is doing

A couple caveats before, so I don’t get dunked on…

Single Node vs. MPP: I’m running Postgres inside a container on my ~~[kindle](Insert sponsored link)~~ Mac, which is completely different than the orchestration/networking effort that is required for a cloud data warehouse with dozens of compute nodes
Row-Oriented vs. Columnar: Postgres is row oriented, an entire row (SOME_ID, AN_ATTRIBUTE, ANOTHER_ATTRIBUTE, SOME_NUMBER) is stored together. Redshift utilizes columnar storage, each column is stored in a separate block(s). This is fantastic for aggregates SELECT SUM(SOME_NUMBER) but makes single-row operations inefficient. If we wanted to update or insert a new row in Redshift while enforcing the primary key we’d have to:

Search for the existence of/find the id of the record we are updating in its block (If it doesn’t exist then insert)
Then find the corresponding attributes in their blocks, so on and so forth for every column
Just to reconstruct one tuple for an update… the opposite of what columnar stores are designed for

This leads into the second point people will make… that this isn’t what the DW is for. You can now see that these inserts are going to be gnarly, why not have some sort of slowly changing dimension that would allow for faster writes and control changes to facts in an OLTP DBMS better suited to handle transactional workloads.

Insert with & w/o a PK

Anyway… lets do our little test:

First we pull the docker image and run a container in detached mode:

docker run --name postgres-test -e POSTGRES_PASSWORD=mysecretpassword -p 5432:5432 -d postgres

We can check the status of our new container with docker ps -a then we can run docker exec -it postgres-test psql -U postgres to get a psql shell. We’ll create two tables, one with a PK and one without, and use generate_series() to curate some faux data for us to write to these tables, something like:

SELECT i, 'some payload data for row ' || i PAYLOAD FROM generate_series(1, 10) s(i);

-- Which will produce the following ID + PAYLOAD 
 i  |           PAYLOAD          
----+------------------------------
  1 | some payload data for row 1
  2 | some payload data for row 2
  3 | some payload data for row 3
  4 | some payload data for row 4
  5 | some payload data for row 5
  6 | some payload data for row 6
  7 | some payload data for row 7
  8 | some payload data for row 8
  9 | some payload data for row 9
 10 | some payload data for row 10
(10 rows)

So lets create our two tables, one with a PK one without:

CREATE TABLE no_pk_test (
    id  INTEGER,
    payload TEXT
);

CREATE TABLE with_pk_test (
    id  INTEGER PRIMARY KEY,
    payload TEXT
);

Click to view a deep dive into Postgres internals

Now we have two tables and with_pk_test has a primary key. We can see it has an associated index:

SELECT oid, relname, relkind FROM pg_class WHERE relname IN ('with_pk_test', 'with_pk_test_pkey');

-- Returns:
  oid  |      relname      | relkind 
-------+-------------------+---------
 16393 | with_pk_test      | r <-- Our Relation
 16398 | with_pk_test_pkey | i <-- Our index
(2 rows)

Each table and index is stored in a separate file. We can also see the initial size of this index, Postgres has allocated the first 8kB page for our B tree:

-- 8192 bytes
SELECT pg_size_pretty(pg_relation_size('with_pk_test_pkey')) AS index_size;

We can get the OID of all databases with:

SELECT * FROM pg_database;

-- Returns (truncated Columns):
 oid |  datname  
-----+-----------
   5 | postgres
   1 | template1
   4 | template0
(3 rows)

-- We can see our current database with: 
SELECT oid FROM pg_database WHERE datname = current_database();

-- Returns:
 oid 
-----
   5
(1 row)

Now we can exit psql (\q) and use bash instead: docker exec -it postgres-test bash. We can cd to the base directory and list the directories.

cd /var/lib/postgresql/data/base && ls` # shows 1  4  5

Nice… 1, 4, and 5 match the OID for the databases we listed above, and now if we cd to 5 and ls -lh we’ll find:

-rw------- 1 postgres postgres    0 Jun 23 16:30 16388 <-- OID of our relation no_pk_test
-rw------- 1 postgres postgres    0 Jun 23 16:30 16391
-rw------- 1 postgres postgres 8.0K Jun 23 16:30 16392
-rw------- 1 postgres postgres    0 Jun 23 16:31 16393 <-- OID of our relation with_pk_test
-rw------- 1 postgres postgres    0 Jun 23 16:31 16396
-rw------- 1 postgres postgres 8.0K Jun 23 16:31 16397
-rw------- 1 postgres postgres 8.0K Jun 23 16:31 16398 <-- OID of our index with_pk_test_pkey

Alright, with that deep dive out of the way lets insert 10MM records using generate_series() into these empty relations:

-- INSERT 0 10000000
-- Time: 3846.560 ms (00:03.847)
\timing on
INSERT INTO no_pk_test SELECT i, 'some payload data for row ' || i PAYLOAD FROM generate_series(1, 10000000) s(i);

-- INSERT 0 10000000
-- Time: 6038.888 ms (00:06.039)
INSERT INTO with_pk_test SELECT i, 'some payload data for row ' || i PAYLOAD FROM generate_series(1, 10000000) s(i);

Yep, closing in on twice as long (3846.560 ms vs 6038.888 ms). And just for fun we can check on the size of our tables/index.

table_name	table_size	indexes_size	total_size
“public”.“with_pk_test”	651 MB	214 MB	865 MB
“public”.“no_pk_test”	651 MB	0 bytes	651 MB

And for even more fun we can see this time to insert explode further after adding a secondary Index.

CREATE TABLE multi_index_test (
  id INTEGER PRIMARY KEY,
  name VARCHAR(256) NOT NULL, -- Our new table has a new column (name)
  payload TEXT
);
CREATE INDEX idx_name ON multi_index_test (name); -- Add a secondary B+Tree index

INSERT INTO multi_index_test
SELECT
  i,
  'user_name_' || (i % 100000),
  'some payload data for row ' || i
FROM generate_series(1, 10000000) s(i);
INSERT 0 10000000
Time: 30859.168 ms (00:30.859) <-- *cough*

There is NO free lunch, sure you can make your reads faster, but at what cost?

Are people even using the index?
Whats the ratio of writes to reads on this table anyway?
I wonder how many B+ Trees I can fit in my 2012 Chevy impala LT?
These are the questions we need to answer

Merge Upsert

What happens if we insert additional records into our tables that already have 10,000,000 records in them. We could compare the data warehouse style insert where we insert everything vs a merge upsert where we update records that already exist and insert new records that don’t.

Let’s regenerate IDs spanning 9,000,001 -> 11,000,000 so we get a mix of new and already existing records and perform our two writes.

-- Creating a new table with the records as discussed above 
CREATE TEMP TABLE new_batch AS
SELECT i, 'new payload for ' || i FROM generate_series(9000001, 11000000) s(i);

We can perform our basic insert into no_pk_test:

INSERT INTO no_pk_test
SELECT * FROM new_batch;
INSERT 0 2000000
Time: 598.475 ms 

-- 12MM Records now, 10MM old and 2MM new
SELECT LEFT(payload, 10), COUNT(1) FROM no_pk_test group by 1;
    left    |  count   
------------+----------
 new payloa |  2000000
 some paylo | 10000000
(2 rows)

Now what about our upsert? We can use On Conflict courtesy of Peter Geoghegan, Heikki Linnakangas, Andres Freund and Jeff Janes.

INSERT INTO with_pk_test
SELECT * FROM new_batch
ON CONFLICT (id) DO UPDATE SET
    payload = payload;
INSERT 0 2000000
Time: 3287.132 ms (00:03.287) 

                                QUERY PLAN                                
--------------------------------------------------------------------------
 Insert on with_pk_test  (cost=0.00..33414.40 rows=0 width=0)
   Conflict Resolution: UPDATE
   Conflict Arbiter Indexes: with_pk_test_pkey
   ->  Seq Scan on new_batch  (cost=0.00..33414.40 rows=1869440 width=36)

-- 11MM Records now, 9MM old and 2MM new
    left    |  count  
------------+---------
 new payloa | 2000000
 some paylo | 9000000
(2 rows)

-- We can also use merge, slightly slower 
-- Time: 4593.521 ms (00:04.594)
MERGE INTO with_pk_test pk
USING new_batch n
ON n.id = pk.id
WHEN MATCHED THEN
  UPDATE SET payload = n.payload
WHEN NOT MATCHED THEN
  INSERT (id, payload)
  VALUES (id, payload);
MERGE 2000000
Time: 4593.521 ms (00:04.594)
                                          QUERY PLAN                                          
----------------------------------------------------------------------------------------------
 Merge on with_pk_test pk  (cost=398687.23..523481.91 rows=0 width=0)
   ->  Hash Left Join  (cost=398687.23..523481.91 rows=1869440 width=48)
         Hash Cond: (n.id = pk.id)
         ->  Seq Scan on new_batch n  (cost=0.00..33414.40 rows=1869440 width=42)
         ->  Hash  (cost=207833.88..207833.88 rows=10979388 width=10)
               ->  Seq Scan on with_pk_test pk  (cost=0.00..207833.88 rows=10979388 width=10)

The merge upsert takes much longer (598.475 ms -> 3287.132 ms). These are two different kinds of writes, but its generally indicative of what you might be trying to accomplish in an OLAP system vs OLTP. Hopefully this sheds some light on what people mean when they bring up the “performance” argument.

These constraints should be enforced upstream

This is idea #2.

And this is opening a can of opinions, hot takes and worms. I think in theory maybe this works, but in the real world there are certainly people with relations sitting in their datawarehouse that could benefit from the safety of PK enforcement or, at least, have that option. So on one hand we have quick writes/optimized plans, and on the other hand all those performance gains are a wash because we have to write those deduping checks anyway and people are confused that their PRIMARY KEY has duplicates in it.

Opinion Time! I don’t understand why the syntax in Redshift couldn’t have been changed to fully remove confusion. If you read the documentation its clear, but why call it a PRIMARY KEY when it could have been PRIMARY KEY NOT ENFORCED… a little clarity over brevity. Databricks has the RELY option. Snowflake doesn’t enforce them in their standard table offering but do enforce them for hybrid tables. BigQuery uses NOT ENFORCED.

(1048064189, Have fun out there!), (1048064189, Hope you enjoyed the blog!), (1048064189, Take Care!)

Oracle Performance

Sun, 15 Jun 2025 00:00:00 +0000

The book

This will serve to store notes/insights/thoughts about Optimizing Oracle Performance book written around Oracle Version 9

Quotes

Preface xix- In my experience, diagnosis is where people usually mess up. There are lots of experts out there responding sensibly to poorly collected diagnostic data in poorly specified projects. It is usually easy to solve a problem when it is presented to you correctly. It can be impossible to solve the right problems when you’re focused on the wrong problem

Page 3- The severity of a problem is proportional to the number of people who show up in meetings to talk about it.

Page 11- Amdahl’s law

“the overall performance improvement gained by optimizing a single part of a system is limited by the fraction of time that the improved part is actually used”.

Notes

Arbitrage

Sat, 14 Jun 2025 00:00:00 +0000

Winning Every Time

I’m going to summarize arbitrage sports betting. I think I’m going to add this to “projects” although the artifacts/results of my time spent in this space aren’t really relevant to this post. Also anytime I reference myself, me, or I in this post I’m actually talking about Jonathan

Is that click bait? I’d argue its not because its true, and I don’t make any money or care who reads this so… no I don’t think so.

The scene

In 2021 or 2022 as sports books were rolling out across the U.S. they all had large promotions “Bet $5 get $200 in free bets” kind of thing. This was essentially just free money as there are many places that will freely show you how to optimally liquidate your free bets into cash. If you read through that link you’ll see that what you use is a hedge against your free bet… so

if your free bet wins you get a nice payout minus your hedge.
if your free bet looses you get a nice payout from your hedge minus nothing… the bet was free… or the cost of becoming eligible to receive the promotion.

This is all fun and stuff, but I quickly liquidated all the cash out of these opportunities and was left with a crippling addiction… just kidding… but gambling addiction is no joke and extremely sad. I was curious to know if sports books ever mis-priced lines, how lines were created, or if you could pit two sports books against each other…

Background on the books

Online Sports books are no better than playing roulette at a brick and mortar casino. They play on human biases and tendencies toward lottery like payoffs to extract value from players over time. Those who consistently win are banned or limited as value extraction is a one way street.

Sportsbooks typically have well informed lines. A betting line refers to the odds that a sportsbook lists for a wager. There are different ways to display odds. I’m going to specifically talk in American odds just because that’s what I’m used to, not because they are superior in anyway (I think most professionals use decimal odds, but I could be wrong).

In American odds, the favorite is shown with a (-) and denotes the amount of $ you’d have to bet to win $100, if the odds are -400 you’d have to bet $400 just to win $100 meaning that “thing” you’re betting on is likely to happen. If the odds are +400 we are betting on an underdog, in this case, if we bet $100 we’d win $400. We can also determine the implied odds underlying the odds we are being shown.

For example:

+400 could be re-written to note the we are betting $100 to win $500 (We get our $100 bet back + $400 in winnings) -> 100/500 or 20% implied odds. For our favorite at -400 we are betting $400 to win 500$ (We get our $400 bet back + $100 in winnings) -> 400/500 or 80% implied odds.

These lines are constantly updated with the latest information and these books are also keenly aware of the action that is occurring on a particular line. I won’t go into detail here but this article has info on how books manage risk on specific lines.

So are we just going to look for lines that are “off”: This team should only be a +200 underdog but the book has them at +500? Given this information above, we are unlikely to catch sports books utterly mispricing a particular line… but maybe we can catch them being just so slightly off. We also have to remember that because these books are just extracting money from people there is a lot of competition to be an extractor. What I mean is that there are a lot of ways to gamble on sports, even Robinhood has begun adding basic predictions markets on sporting events. When all of these different places offers odds on an outcome; sometimes, they begin to disagree with one another. This is what we can take advantage of.

Taking both sides

What do I mean by “pit books against each other”. To explain this I want to walk through how these books function.

We are going to be a sports book for a second. Lets say sports team A is playing sports team B. At our book we think this is going to be a close match so we offer team A at +110 (slight underdog) and team B at -110 (slight favorite). I play the scenarios out in the below drawing.

Two players one on either side of the line place bets so that their total payout is equivalent ($210):

You can see our sports book makes no money regardless of the out come (It’s pot of $ generated by the players must be payed out in full to satisfy the bets regardless of the outcome), its just facilitating transactions, and it’ll be in the red after it has to pay its variable/fixed costs. This is why you’d never see these odds at a sports book outside of gimmicks like the Super Bowl coin toss, promos, etc… You’ll see lines like this:

What changed? The odds on Team B went from -110 to -130 (It became more of a favorite, a.k.a the sports book pays out less money on the same amount wagered). Previously, player 2 only needed to bet $110 to win $100 now they have to bet $130 to win $100. I added the implied odds of the specific outcome in the first diagram with a 47.62% chance our underdog wins and a 52.38% chance our favorite wins. These sum to 100%.

Now lets check out the implied odds in the +110 / -130 scenario… 47.62% and 56.52% respectively… these don’t sum to 100% anymore, now we get (104.14%) uh oh. NOW our book is making money:

if team A wins the books makes $20
if team B wins the book breaks even.
We calc the EV here assuming the true odds are -110/+110 as (.4762 * $20) + (.5238 * $0) = $9.52

Wow so our book has an EV of $9.52 every time two people put on this bet! And looking at little further if we divide our EV by the Pot of money: $9.52/$230 we get ~4% remember our new totaled implied odds 104.14%… 104% - 100% = 4% this is the books “hold”. Hold in sports betting refers to the percentage of money that sportsbooks keep for every dollar wagered. The percentage will vary by wager type. Sportsbooks can’t just maximize hold because they have… competition. If I can get +130 in book A where as book B offers +150 on the same bet… you’re going to bet with book B every time… sports books need volume they need your bets to make money.

And this is what these books want, they want as high volume of these transaction as they can get. That’s why they put on promotions and offer free bets and create Ads with tigers and famous people etc…

But do all books have the same odds? No. Just as books create lines that ensure they win, we can look for favorable lines as well. These books don’t exist in a vacuum they compete against each other for a pool of betters, many of which are sensitive to how great of a hold a book has on a particular line. As sports books update their lines, mis-price things, or disagree with each other we can take advantage. Lets look at this below.

In this scenario the same player is taking both sides of a line across two sports books. Sportsbook 1 offers +110 while sportsbook 2 offers the other side at -105. Lets look at what happens.

Bam and now we can see that no matter the outcome of this game, player 1 makes $5 or comes out even. Summing the implied odds 47.62% + 51.22% = 98.84% we have engineered an inverse hold (1.16%), where the player always comes out on top regardless of the outcome. Our EV on this bet, assuming true odds of +110/-110 is (47.62% * $5) + (52.38%* $0) = $2.38 and don’t forget $2.38/$205 = 1.16% (Our inverse hold).

Wow so if I have $205 I can instantly make 1.16% on it if I can find these lines?

And if you take that same $205 and use it again and again every night for a month I could make even more $?

And if I increased my bet sizes to $10,000 could I scale up the magnitude of my profit?

Yes, in our contrived +110/-105 example if we bet $10,000 on +110 and $10,500 on -105 we’d have an EV of (47.62% * $500) + (52.38%* $0) = $238… I like $238 better than $2.38.

What if we found something even better like +110/+110?

Yes that would be nice.

So what’s the catch?

Books aren’t dumb, they will see you are consistently hitting soft/slow lines. Lines with low hold or lines that weren’t updated fast enough to avoid mis-pricing across other books. They will ban you or limit you to only being allowed to bet a few dollars at a time.
It’s harder than you think to find these odds. Odds scraping is difficult. Scaling is hard. If you buy odds… that’s going to cost you $, and they are slightly delayed. You are going to struggle to place bets via an api, most books don’t have one, so you’re going to have to build something to place these bets via the UI or manually do it (Doesn’t scale well).
Odds can change as you take both sides as there will inherently be some latency between the two bets. If they change in the wrong direction and you already have exposure to one side… now you’re just trying to limit your losses.
I said before that books will limit you, they can also void your bet. That’s right, if the sportsbook realizes it gave you too good of odds it will just void the bet. Bad analogy alert: Imagine you open a 12 month CD at 4.5%, sign the paper, add the money done… you wake up the next day and the CD isn’t there any more. You call the bank… “Yeah umm we actually don’t want to give you 4.5% so we closed the account… how about 3%”. Sports books can and do just that… remember you are on their platform and this stuff is relatively “new” they get to do what they want (to a point) with little recourse. Its like if you build a YouTube channel and they strike down your video… you can try and appeal but at the end of the day you’re on their platform and play by their rules wether or not your video was justifiably taken down. And who is to determine justifiably? I don’t know that’s a topic for another day.
Books will ban you. Internally it’ll be called the fraud detection team or money laundering blah blah blah, and I’m sure they do a good amount of that… but in reality its the smart money detection team “don’t let us lose money at all costs if someone wants to win they can do it elsewhere.”
In the United States there is this thing called death, and there’s this other thing called taxes… both tend to happen to people. Speaking of taxes, there is another thing called taking the standard deduction, and there’s other things about qualifying as a “professional gambler”. Now what do all these “things” mean? I don’t know, I am not an expert but you should READ UP ON THIS before you start your lucrative arbing at scale.
Side note, Arbitrage exists in other markets as well, look at the trades SBF was putting on back in the day.

Here SBF is taking advantage of price differences between markets (buy low, sell high), his major challenges were essentially bridging the two markets. If people were willing to sell BTC for $10,000 and others willing to buy it for $15,000 in a single efficient market the prices would converge and the opportunity would be quickly eaten up. But because there was a lack of information, hurdles to cross in putting on the trade, and crypto being the wild west he was able to capitalize on this opportunity.

So if we could overcome all the issues above (I’m sure some people do) we could generate a nice profit. But it is NOT easy to do, NOT easy to remain undetected, and NOT something you could accomplish over night. You WILL need $ to get started, you WILL need to adapt to changes and you WILL run into issues as you scale.

Good luck.

What I learned this week 06-14-2025

Sat, 14 Jun 2025 00:00:00 +0000

What I learned this week

The for Loop

Software

Business/Finance

Free cash flow

Interesting Links

I just don’t get it

Politics!

Site for finding alternatives

Math/Stats

Travel

Other

What I learned this week 06-07-2025

Sat, 07 Jun 2025 00:00:00 +0000

What I learned this week

Software

Race Conditions

I like this definition of atomic: The capability of performing an uninterrupted update with valid data before and after the update and the data cannot be observed by another observer in any intermediate state it may take on during the update. It may be a single bit or it may be an entire database. S/O ACID!

Semaphore

I was unaware of SAVEPOINT and ROLLBACK TO

The reason for separating raw parsing from semantic analysis is that system catalog lookups can only be done within a transaction, and we do not wish to start a transaction immediately upon receiving a query string.

Greate overview on AWS networking, linking all sub-sections as well

Everyone gets their own private network inside AWS.
Solves IP address conflicts and security concerns of one big shared network
Mapping service connects instances in a single VPC across servers (kind of like a namespace) -The Mapping service can connect servers across AZs but not Regions, VPC span AZs not regions. To connect VPCs in different regions there’s VPC peering or AWS Transit Gateway

Subnets

An internet gateway is like a hole you poke in your VPC so you can talk to the internet.
Only a subnet can be connected to the internet gateway.
A subnet is a way to group your instances. It’s sort of like tagging them.
Private subnets can’t talk to the internet and are the default. Public subnets can talk to the internet.
Subnets are availability zone-scoped. To create an instance, we need to specify which availability zone we want to create the instance in.
A route is a connection from your subnet to the internet gateway so that your subnet can talk to the internet.

CIDR

Classless Inter-Domain Routing (IP address allocation)
Specifies a range of IP addresses by fixing a certain number of bits in a 32 bit IP Address 172.98.0.1/24 (/24 denote the # of bits that are fixed)
Each VPC will have a CIDR range, and each subnet within a VPC will have their own ranges within the VPC’s range

Route Tables

Route tables are mappings from destination to target, given a destination where to a go next (target)
Route tables will always try to match the most specific route.
Route tables are created at the VPC level. Every VPC comes with a route table, called the main route table. A VPC can have many route tables
all subnets are associated with the main route table by default

Security Groups

Security groups operate at the firewall (or EC2 instance) level (They have IB & OB rules), they are ALLOW only
IB traffic is allowed if its a response to your request (This is why SG are referred to as stateful)
NACLs operate at the network (or subnet) level (They are used to BLOCK an IP(s))

Netmask

Routers can assign private IPs from several private IP address ranges. They are addresses that start with 10.x.x.x, like 10.0.0.1; addresses that start with 192.168.x.x, like 192.168.10.1 and 192.168.12.1; and IPs in the 172.16.0.0 - 172.31.255.255 range.

RFC 6598 defines a Shared Address Space, specifically the 100.64.0.0/10 block, for use by Internet Service Providers (ISPs) in their Carrier-Grade NAT (CGNAT) deployments. This address space is reserved for connecting Customer Premises Equipment (CPE) to the ISP’s core routers.

Business/Finance

Interesting Links

Cool blog and public brain concept to explore

Math/Stats

Travel

Suggestion from my Korean Uber Driver: Seoraksan national park go in sept/oct

Spent the week in Seattle for our team Summit

Other

What I learned this week 05-31-2025

Sat, 31 May 2025 00:00:00 +0000

What I learned this week

Software

Duck Lake!

Dangers of a certain compiler flag

Complete CS Education, I wonder how many LLMs have trained on this

A progressive JPEG will actually show the entire image right away, the image will load in at full size looking pixelated and will become more clear as it loads.

O(n^2) is the sweet spot of badly scaling algorithms

Business/Finance

Interesting Links

Poland Economy larger than I thought

Math/Stats

Travel

Went to Olympic National park for 5 days for my birthday.

Other

What I learned this week 05-24-2025

Sat, 24 May 2025 00:00:00 +0000

What I learned this week

Software

Nice IDE for Postgres

FSRS, I want to do create flash cards from the section of the blog

Building on LLMs Handbook

Implementing Search

Business/Finance

Interesting Links

HDR

Math/Stats

Travel

Other

Time Enough for Love

Tue, 20 May 2025 00:00:00 +0000

The book

This will serve to store notes/insights/thoughts about Time Enough for Love

Quotes

8- A society that gets rid of all the trouble makers goes downhill

10- The most sophisticated machine the human mind can build has in it the limitations of the human mind

20- I have no beliefs. Beliefs get on the way of learning

24- never take anybody’s word about whether a gun is loaded

31- If a man is greedy you can cheat him every time

54- progress doesn’t come from early risers. Progress is made by lazy men looking for easier ways to do things

83- there doesn’t seem to be anything that a government can do to an economy that does not act as positive feedback, or as a break. Or both. Maybe somewhere someday someone will tinker with supply and demand to make it work better instead of letting it go its own cruel way.

113- don’t ever become a pessimist… a pessimist is right more often than an optimist but an optimist has more fun. And neither can stop the march of events.

114- whores perform the same functions as priests but far more thoroughly

129- The trouble with defining in words anything as basic as love is that the definition can’t be understood by anyone who has not experienced it

165- By the law of Supply and Demand a thing has value from where it is as much as from what it is.

229- Privacy is as necessary as company; you can drive a man cray by depriving him of either.

259 - Never underestimate the power of human stupidity

281- as money came in the bank I’ve been burning it. It’s waste paper. As long as we keep account

317- the centuries may not give a man wisdom, but he acquires patience or he doesn’t live through them

Notes

What I learned this week 05-17-2025

Sat, 17 May 2025 00:00:00 +0000

What I learned this week

Software

Business/Finance

Mcdonalds data

everything’s a $100

Interesting daily econ tidbits

364 Walmarts in China!?
Turns out there are Walmart’s in 19 countries, and 2.1MM associates wow

Interesting Links

Math/Stats

Travel

Other

What I learned this week 05-10-2025

Sat, 10 May 2025 00:00:00 +0000

What I learned this week

Software

Linux Tools

0 Deps

Can’t say I 100% agree but can’t say I 100% disagree, there’s probably some happy medium here on when to use deps.

JSON dictionary

Infra at Palantir

Business/Finance

Seagull management

Interesting Links

Appearing confidant Observations from watching people

I like how this is written but I don’t think we have some omniscient writer who can quickly define a person 100% correctly

Math/Stats

Travel

Other

What I learned this week 05-03-2025

Sat, 03 May 2025 00:00:00 +0000

What I learned this week

Software

Docker relies on Linux kernels, which means that macOS and Windows cannot run Docker natively without some additional steps. Each operating system has its own solution for running Docker. For example, Docker for Mac uses under the hood actually a virtual machine that runs a Linux instance, within which Docker operates.

It seems you can get a shell on this VM via: docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh
docker run -d -i -t --name {give it a name} {image} Finds/Pulls image, runs a container, -i for interactive, -t for a shell, -d for detached docker pull {image} to pull in an image
docker logs -f {name of container} to get container logs
docker pause {name of container} && docker unpause {name of container} pause/unpause container; there’s also docker container start and docker container stop
docker attach --no-stdin {name of container} attach your terminal’s stdout/stderr to a running container the –no-stdin flag meas we will not attach with std-in, so if you CTRL+C you’re only disconnected form STDOUT
docker exec {name of container} {command} Execute commands inside a running container
docker rm --force looper stops/removes a running container, basically: docker kill + docker rm
docker search {image} search images on docker’s servers. Images consist of: registry/organisation/image:tag. But may be as short as ubuntu, then the registry will default to Docker hub, organisation to library and tag to latest

ls containers even after they’ve exited: docker container ls -a or just docker ps -a

Remove all stopped containers with docker container prune and for images: docker rmi -f $(docker images -aq)

Flags: -i (interactive), -t (tty) and -d and What is TTY

A dockerfile contains the build instructions for an image

Quite a few popular images are so-called multi platform images which means that one image contains variations for different architectures, one example is Ubuntu.

Zip Bomb

Business/Finance

Remote work at its finest

Interesting Links

Block YT shorts

Math/Stats

Travel

Other

Porsche!

Picking Optimism

What I learned this week 04-26-2025

Sat, 26 Apr 2025 00:00:00 +0000

What I learned this week

Software

T-Strings

Business/Finance

On the ridiculousness of the tax code

Interesting Links

Math/Stats

Travel

Spent the week in Japan, had an amazing time. Will record what I did in the travel section.

Other

Do a trip with friends at least once per year. Biggest lesson from Rob who I rode with in Kiryu.

Japan

Sun, 20 Apr 2025 00:00:00 +0000

Japan Travel Log

This trip was the best/most impactful traveling experience I’ve had as an adult.

I’m grateful to have had the opportunity to do this.
I’m happy I spent a few months learning some basic Japanese.
I’ve drastically underrated traveling by myself
If I have children, I will highly encourage them to live abroad for an extended period of time as a young adult

Whats below?

Below is pretty much my stream of thought, recapping how each day went in the moment. I’ve lightly edited for clarity. My core reflections are above, although I have done some further reflection internally that won’t make it here. Any post-hoc edits will be highlighted like This.

Day by Day

4/14:

Mom dropped me off at airport. Was a bit busy for 5:30 am. Walked around terminal 3 and ran into Charles Barkley which was wild hahaha. Missed a great opportunity for a San Antonio joke. I’m on a hot streak for seeing famous people. I had seen David Goggins a few days before

Sat next to James on the plane to Hawaii. He’s in the army stationed in Honolulu and is a helicopter pilot/instructor. Good guy!

Flight went well beat all levels of angry birds while listening to young thug. Managed to drink about a liter and a half of water and ate ~700 cals… so feeling pretty good going into the second leg. Hopefully the WiFi works nicely.

Stretched my legs and walked to the garden at the Honolulu airport, it was really nice. The small garden is extremely nice and a great way to break up 16 hours of flying PHX -> HI -> JPN

4/15:

Ok so this is a little weird because of the time change but I’m still traveling to Japan (Honolulu to Japan leg). We’ve just changed days. Sitting next to another military guy had a good chat with him for 2 hours. Talking about if he likes the military, why do people sign up, kinds of people that are there etc… they are going to Japan for 1 month I guess for training or something. We also talked about voting, and how he thought that some sort of service to the country could be a requirement to vote and the pros/cons of that

The WiFi on the flight (Flight was equipped with free star link internet which was excellent) was nice but I wasn’t feeling all that great I toughed it out and got some sleep which made me feel better and played some chess.

Got off the flight and it I think a bunch of international flights all came in at the same time… so it took like another 2 hours to get through customs/duties with my bag. Met some guy from Finland who was cool.

Found the train station, eSIM doesn’t say it’s active but I still have service I guess from t-mobile. Can’t tell wtf train I’m getting on but I’m so tired I just bolt into the train I think is right and the doors close on me on the way in. I made a split second decision to get on which ended with the train doors closing on my backpack as I squeezed through the door

Turns out it was right! Got off at Kamata station and start to walk to the Airbnb. It’s wild like this place is straight out of a movie and I’m just in some tiny city outside of Tokyo. Make a few wrong turns walking but get to the place.

Do all their check in stuff and I get to their door and I can’t get this Lock Screen to come up so I can enter my code. So I just start tapping wildly and it works hahaha. The lock on the door was a little weird to use especially on basically no sleep

I had a little snack but I’m so tired I can’t go out to eat. I’m going to shower and go to bed hopefully by like 9:00pm. I’m going to set an alarm for 9:00am and force myself to get up after 12 hours of sleep. 12 hours of sleep was a PIPE dream.

4/16:

Woke up at 4am so got 6:30 hours plus the time in the plane… not bad. Showered, had a little food, did some laundry, made an instant coffee and organized a bit.

Worked on Ally’s website (slamming effect) Claude won’t do what I want so may have to make some tweaks. I’ll see what time a get back tonight and work on it then. I was making a website for my Girl Friend's birthday, but I know very little about web development and was running into issues

Current goal is to go to 7/11 get some cash, and maybe a snack. Then hit the grocery store, drop off what I get and then head out for the day and get some breakfast/go to the shrine. Let’s see how this goes. For some reason I was very nervous about going to stores... IDK

Ok 7/11 went smooth I got 20000 yen, a coffee and a snack! Payed with suica! Ate me little açaí cup in a park. There was a small park next to me airbnb

Grocery store went fine. I got bananas,yogurt, avocado, bread, juice, almond milk for like $10. Got change for one of my 10000 yen bills so that’s nice as well. I think the lady asked me if I wanted a bag, which I actually had studied, but she said it too fast so I just stood there looking like an idiot until she just got me a bag.

Went to kamata cafe which was good and then walked around sun road. Saw a couple restaurants that looked good. Now taking the train to the temple.

Walked through the cemetery and to the ikegami rooftop garden. Got a cool picture of mt fuji from far off in the distance. This would be the best view of Fuji I'd get even though we'd visit a city directly next to the Mtn it was covered by clouds Walked through the shrine. Inside didn’t allow photos but it had a cool golden display and large drums, very intricate/ornate. Continued walking around, bought a matchbox and sat in a park with a little pond and some trees.

Walked back to Airbnb from kamata station with no maps assistance! Bought some strawberry KitKat, eating them in the park by the Airbnb, they are pretty good!

Noticing a few things about the culture here:

No adults walking around wearing shorts
Nobody has sunglasses except some older people
Very quite in public places, no talking/eating on trains
Everything very clean
Everyone has their work uniform, seems most take pride in their work
There are 0 public trash cans

Took the train to go meet Chris for dinner. Sat by the meguro river and waited before dinner had some edamame snack. Another guy came to dinner with us named Ben. Good conversation. We talked about Japan a lot, what it means to live here the people/culture. Dinner was good, bunch of small plates, apparently fish semen is really bad, so is the raw chicken… horse is good though. We just ordered from our phones and they brought it. We did not get Fish semen, horse, or raw chicken we just talked about those things

We got booted from our table after 2 hours and went to a small pub. They played heavy metal and the bathroom had a chord attached to a door and a guitar, so when you opened the door a chord on the guitar was played… so cool. After a beer, we walked back to the train station and Ben and I took the ekigami line back, he got off before me.

Walked back to the Airbnb so tired! A couple of beers and lack of sleep will do it to you.

4/17:

Slept until 5:20 so got 7 hours!! Made some avocado toast but there’s no salt and pepper in the Airbnb but there WAS some in this little plastic utensil set bag a had… huge win.

Took train over to great cycling tours. Met Yuki-San and Takashi-San and chatted with them for a bit. Then met Eva and caitlin/rob (from Alaska). Was pretty hungry and only brought a couple strawberry Kit Kat.

We went to: - Nihonbashi Bridge - Kanda Jimbocho (book town) - Koishikawa Korakuen (Japanese Garden) - Sugamo shopping street - Gokokuji Temple - Zoshigaya Cemetery (Soseki Natsume’s grave) - Kishimojin Temple - St. Mary’s Cathedral (sorry, in Japanese) - Yasukuni Jinja shrine - Imperial Palace - and through lots of backstreets

Caitlin got a flat, so we stopped at a shrine and it was the tightest tire I’ve ever seen. Apparently it never happens because the roads are so smooth and clean, and these bikes aren’t the best maintained lol. After an hour and some bloody knuckles we got it fixed.

Lunch was really great had curry with udon noodles and tempura vegetables. Stopped for a while and got a matcha tea towards the end. After the tour we struggled to pay (Their device wasn't working quite right) and Caitlin forgot her phone and we had beers.

Took the train back but must have fucked up a bit and realized I started to go the wrong way about half way home (train started going west lol) got off and walked over to the right line and was fine.

Had a great dinner at Hung’s kitchen tiny Vietnamese restaurant and a nice conversation with the owner. Amazing experience, the epitome of why I enjoy traveling. Took some spring rolls to go and went to bed.

“It is by riding a bicycle that you learn the contours of a country best, since you have to sweat up the hills and coast down them. Thus you remember them as they actually are, while in a motor car only a high hill impresses you, and you have no such accurate remembrance of country you have driven through as you gain by riding a bicycle.” - Hemingway

4/18:

Went to bed at 9:40 and woke up at 5:40! Got a full 8 hours!! Called pal and ate some breakfast at home. I got the right amount of food, was really nice having some avocado/banana/yogurt for breakfast at home.

Called pal, sounds like she’s doing well. I struggled to articulate my experiences. Like I can talk about what I’m doing, but I’m struggling to capture how I’m feeling, how I think about Japan/the culture and the people/places I’ve been since I’ve been here. I just need some more time to process. The beau miles short film I watched this morning actually fits pretty well.

Took three trains to get to Mt Takao. Wasn’t bad at all and got to the Mt at like 10:00 am. Hiked up to the top there were a lot of schools with big groups of kids which clogged the trails a bit, I went up #6. The nature was beautiful and there were nice flowers.

I had a snack and some matcha ice cream at the top. Then I started to descend trail #4 and it spit me out directly at the soba noodle restaurant, I got the sansai since I didn’t want the raw egg and ate outside with an amazing view. I just wandered into probably the coolest meal so far this trip hahahaha… it just keeps getting better. Had a funny conversation with the waiter about a little jar she brought out which turned out to be spices. I tried to translate the writing on the jar but it was just the name of the merchant and the address not actually what it was and we both laughed. Randomly walking into this place and soaking it in epitomizes what made this trip specials

I kept walking down after the meal and asked some guy what time it was in Japanese which was cool. Ima nan ji de su ka? Then saw a waterfall/shrine where there was a monk washing himself and praying. Waterfall is Taki.

Kept walking down and saw a small animal, kind of looked like a badger or those things we have at Sabino canyon. Headed back down where all the shops were and bought Ally a cute owl and a card.

Now headed back to the Airbnb on the train going to get ready to hang out with Kenta tonight.

Took the train and met up with Kenta, train was packed to the brim and some lady was bringing her birds haha. Shibuya was busier and bigger than I could have imagined. I was disoriented just walking around. Bought a couple post cards for pal. and we walked to the place for dinner.

Kenta’s friends are nice guys, they all went to international school so they speak English fluently and they banter in some combination of English + Japanese. We had some nice sushi, little squid, octopus, chicken kutsu, seaweed soup and drinks for like 40$ per person. I don’t know if I was tired from hiking or just tired from the time change, but I felt like I had vertigo at dinner haha, it was wild.

Next we went to a pub and got a DK shot Pretty nasty. Some random guy asked to hang out with us and Kenta said sure lol. DK shot was just some nasty double shot, so did one of those. We went to another one of those British Pubs, I guess they have a bunch of British type pubs here it’s like a chain.

We went out side and I was a little buzzed and had 45 mins of trains to get home so ended up heading back around 9:45 and headed to bed.

4/19:

Actually slept in until 7:00am today, had some breakfast at home and started getting all packed up. I can’t believe it’s already time to leave this Airbnb, it was a good pick. Had everything I wanted, quiet location, and forced me to use the local trains to get around.

I got packed up and then got some cash from 7/11 and hunted around for some flowers to give Taka-San and Keiko-San, but couldn’t find any so headed to the station. Where I had a short ride over to Omori station only one stop away. I was at the station early and got some flowers from a little shop (just worked out perfect) fumbled all the Japanese haha.

Met Taka and this Croation couple and walked to Taka’s house about ten minutes away from the station. I had my full bag on my back the entire time at this point and we walked slowly, chatted and stopped at a shrine and by the time we had gotten to the house I must have been wearing it for an hour, so my traps/shoulders were completely shot.

Taka and Keiko are amazing people we did udon noodle making (danced on the dough), origami, made tempura vegetables, learned about our cultures, made sushi, were given gifts, and wrote our names in Katakana in calligraphy. Keiko spilled shochu on me and was very apologetic but it was totally fine. It was such a special experience, just a gem of Japanese culture from people who were caring/dedicated enough to spend so much time preparing for us and inviting us into their home. I can’t wait to return the favor/come back at some point and at the very least send them some photos of us making Soba/Tempura. I have since attempted to do tempura at home once and sent them photos, it turned out ok.

After that ended, went back o Omori station and headed towards the Airbnb, turns out the Airbnb in the google doc was not the right Airbnb as the address Tristan sent was completely different. Luckily they had landed, so I was able to clarify and get on the correct trains. I made it to the Airbnb first after walking through hordes of people with my bag and finding the door down a small path next to the building, it was a cool three story place above a kebab shop. But it was hot and I was a little nervous about how close we were to some of the busiest places in Tokyo.

Tristan and Matt showed up next and we walked around for a second and got a few things from a family mart. Then Kaiser showed up and we started ambling around to dinner. I ended up walking us into a Korean place a bit from where we were staying and we had a nice meal where I tried to navigate ordering and that went poorly. But it all worked out the meal was ok, but it was still just fun to hang out and go to a random spot half way around the world (literally).

I walked home and went to bed the rest of the guys went out for bit I think.

4/20:

On this day we were going to a baseball game and my friend's parents house, but I also needed to travel ~3 hours to a small town to go ride bikes the next day. I was pretty nervous about fitting all of this in and navigating the 4-5 trains I'd need to take to get out there.

Woke up at 6:00 today so I think I’m doing pretty good on my sleep schedule. Just wanted to get everything in order for heading out today as all I’m going to have on me is my backpack. I’m a little nervous for this trip, I think I’m going to skip dinner at Kenta’s parents house to give myself the best chance at making it to Kiryu and getting some sleep. I’m going to make it to Kiryu!!!

Ok so everyone finally woke up, I had gotten all ready early in the morning since I went to bed early. Grabbed a little pancake thing from Lawsons. Came back and Tristan and Mike were ready but Matt wasn’t so we went and got more pancakes and coffee. Then we came back and Matt still wasn’t ready, so we had to take a later train.

I got us on the right train and we met Kenta on the train. We got to the game it was dope. Lot of singing, but we lost, however it was fun and got some good food. Then I got some chopsticks from the shop. Some of Kenta’s friends showed up mid game and hung out with us.

Then we went to Kenta’s mom’s house and met Kenta’s Mom, Grandma, and sister. I’m so glad I decided to do this and not just go straight to Kiryu, I got to eat a little food and speak some japenese, I think I impressed Kenta’s grandma.

I’m now on the final train to Kiryu, I actually made a few mistakes closer to central Tokyo, but we freestyled and made possibly a better route. I’m a little tired and need to eat some of my food, but I’ve got everything I need and I made it to Kiryu!! (Fingers crossed).

Checked into hotel, shaved and called Al. Went to bed but couldn’t figure out how to close window so kept waking up as the wind would blow really hard haha. CLosing the window was not hard, but I spent probably 20 minutes attempting to do it.

4/21:

Got up after not great nights sleep but going to get after it. Suited up and walked over to Rob’s warehouse. Met Frances, had some coffee, and got set for the ride.

What a ride, I mean I can’t do it justice with words but I must go back the photos are such a sliver of a momento from such a vast and majestic place. Rob was such a nice guy! A few highlights:

started off riding on the river path
Roads so smooth few easy climbs
Massive perfect tunnel that nobody uses haha
Large climb can see volcanos, Tokyo, cherry blossoms and lakes
Stopped for lunch by the lake, chatted about friendships DO A BIG TRIP WITH FRIENDS 1/2 times per year
Frogs We saw two frogs on top of one another in the middle of the road
Flatted off a rock

Made it back to town and had lunch then got myself sorted at the shop. Walked back to the hotel and told the lady I was checking out early. Went up showered and found someone else’s shirts so I gave those to the lady and told her a spilled coffee on the sheet (it was blood from cutting myself shaving lol) and she was nice and charged me 1000 yen.

I then walked over to the other station in kiryu (shin-kiryu) and bought a limited express train ticket. This route is wayyy better than my previous one. Only need to switch trains once. Man I am a little fucked up and dehydrated. But was one of the coolest things I’ve ever done.

Got back at like 8:30 and the boys waited for me for dinner. We went to this little place and I met Walid, Julian and Jorge. After dinner we got a couple snacks and went to bed I was cooked.

4/22:

Woke up around 8am and got my shit organized didn’t get out of the house til ~9:30 but the boys picked me up and we walked over to Meiji Jingu. The gardens were really nice and peaceful. After the gardens we took a train to harajuku, and we shopped and looked at gaming cafes. Not really my thing

After a while I dipped out with Matt and went to this place for ramen which was gas. Had to buy a ticket at the machine and gave it to a little lady that sat us right in front of the chefs. They would take the noodles out of the boiling water and shake the boiling water onto the ground. Ramen was great.

Then matt and I took a train to meet up with Kenta who was going to take us around a museum. Tristan and them showed up a little after and he had crawfish so he was mucking crawfish with gloves on and got it all over his new bathing ape shirt and we were all laughing. I made friends with ladies in this little cafe and got an ice cream.

We went into the museum and looked at old Japanese stuff for a while. I got a postcard for ally from the gift shop. Then we walked to this large open air market. There was a ton of crazy stuff (fish, turtles, clothes,etc…) we stopped for fried octopus balls and beers.

Next we hopped on a train for a sushi dinner. Got to the place and they served out large trays of different sushi almost immediately it was really good and we got Matt an avocado only roll because he barely ate anything. I got us to stop at the hub (British pub) and we had some large beers and a DK shot. After that, we took a train back to the Airbnb, I took a separate train from the other guys but we met up in the walk back from the station.

4/23:

We had to get up early for a bus ride out to Mt Fuji at 10:15. Spent the morning packing up and then we went to the station to meet Kenta. Kenta brought us to the bus station and I had a fun 2 hour ride listening to music and shooting the shit with Tristan.

We got to the hotel around 12:00. Dropped out stuff off and went to a Hoto restaurant and sat in the bench in the floor and had hoto and a grape sour. It was very good. Walked back to the hotel and got our room. Room was essentially an open space and then a bathroom/shower.

We headed out and walked around the town, rented a couple swans and paddled on the lake. Went back to the hotel and put on our traditional bathrobes and want to down to the hot springs.

There was a locker room and then inside were like 30 seated showers. You shower off first and then go in the hot springs. There’s an indoor, outdoor, and half indoor/ half outdoor. They are hot 44 degrees C, our favorite was the half indoor/outdoor, spent a lot of time with try Kenta and Matt.

Then we met everyone up on the seventh floor for dinner which was a nice buffet. Ate and drank a ton. Then we went back the room and everyone got a little pissed and we played cards. Mike and I went down and got more beers out of a vending machine and eventually went to bed around 11. Somehow a window got left open and we got hammered by mosquitos

4/24:

Got up at 5:15 am, was sweating so much my bed was soaked. We went up to the observatory but still couldn’t see Fuji because of the clouds. We went back down and did another 90min sauna session before going to breakfast.

The other guys woke up like 15 mins before breakfast ended. We checked out and then went up a cable car to a view point above the lake and did a short hike to a small shrine and got some ice cream. We got down and Kenta and I walked back to the bus station while the others took a car. Kenta and I went to get ramen while we waited for the bus.

The bus back was fine, we worked on Matt’s hinge profile using the photos I took earlier. Then we got back to the station and walked to their hotel. It was a bit of a shit show so I headed to the airport early and said goodbye. Dumped a couple beers by a vending machine and made it to the train. Went through Kamata station one last time as I headed to the airport.

The people who were on the first bike tour from Alaska were on the same flight to Honolulu. The flight was pretty easy, talked to this girl I sat next to for a bit and then mostly slept. Was pretty groggy landing in Hawaii, but grabbed my bag and re-checked it in after customs with no issue. The flight kept getting delayed, I could’ve spent some time at the garden but every 15mins it’s get delayed like 15 mins. Almost back to PHX now, a bit tired but just need a shower and a meal. Travel went fairly smooth.

What I learned this week 04-19-2025

Sat, 19 Apr 2025 00:00:00 +0000

What I learned this week

Software

Unpixelate

PG wire

Business/Finance

Battery powered profits not quite there

Interesting Links

Math/Stats

Travel

Flew to Japan, and now I’m in Ota City, Japan!

Other

Furthest Time zones

Mailing Potatoes

What I learned this week 04-12-2025

Sat, 12 Apr 2025 00:00:00 +0000

What I learned this week

Software

#! - interesting/tidbits

https://www.openwall.com/lists/oss-security/2025/04/01/1

Billion lol attack

Old article on airbnb photo loading

Business/Finance

Should we make advertising illegal?

Interesting Links

Math/Stats

Travel

Other

Posting this from Tokyo after 27 hours of travel #dedication

https://youtu.be/ADXX4fmWHbo?si=ce1t10JYyMbK3TPU

What I learned this week 04-05-2025

Sat, 05 Apr 2025 00:00:00 +0000

What I learned this week

Downloading games from the radio it’s cool to read about these times as I’ve never gotten to experience this.

April Fools

Software

Business/Finance

Google Finance’s broken feature

Interesting Links

Math/Stats

Travel

Other

Hilarious post on a vary volatile compound

Awesome Chess Project

Query Banks

Mon, 31 Mar 2025 00:00:00 +0000

That query bank should be a semantic layer

I’ve done it myself:

“Oh we should just cache all these commonly used queries somewhere central, so we always have a nice definition for someone to use.”

You could even extend the functionality of this “Query Bank” and train an LLM, or perform code reviews, or implement search, etc…

Sweet, now anyone can grab some code and use it! They can paste it here and in this scheduled job there, science can use it and slam it into a notebook, and others can run ad hoc workloads against it.

No, ahh! The BANK IS CLOSED

A query bank misses the entire point of DRY, you end up with a hairball of copy/paste code that has no reference to its origin.

Imagine we took all package management and said you know what instead of depending on our source code you should just copy and paste it into your project and reference it there!

That is what the “Query Bank” is doing because you can't inject the latest definition(query) at run time. If you CAN do this then you don’t have a query bank, you have an actual data model, fleshed out with materialized relations that have some established dependency chain which you can reference in your workflow.

The query bank is just a bandaid for broken process/data models:

No lineage of where code has been copied
No ability to update code across all copies
Limited ability to notify users of changes to upstream code
Most likely repeating ourselves which is extra work for people and DBMS
Will 100% cause discrepancies between “Sources of Truth”
No ability to ensure code is being copy/pasted into the correct context

Creating tables establishes consistency, durability and determinism that’s not possible with a “Query bank”.

The Bank is open

Make a table, document the table, expose the table, manage access, and share the table.

Ask yourself a few questions before you act:

Should I really send this code to someone or should I just DEFINE this once and allow them to read from the definition?
This job, model, notebook whatever you want to call it is already like 1000 lines of SQL should I really pile on more logic to this or maybe its time ot break down into a series of relations?
I’ve seen this Temporary table, CTE, etc… in a bunch of people’s queries should I just paste it into mine or is there something better I can do?

It’s your problem until you fix it or find the owner

Here’s a challenge as an example. If you see a long case statement being shared around. Yes, literally just a case statement without any context… take the initiative and see if you can’t centralize that and store it is a relation for the people needing it.

It’s most likely they don’t know any better and will continue to do this until they have a nasty data quality issue/argument.

Advice to Beginner Cyclists

Sun, 30 Mar 2025 00:00:00 +0000

Found myself giving a bit of advice to new cyclists, so I figured I’d store my thoughts here so I can share it. This is primarily for road cycling, but a lot of it could be applicable more broadly.

These are just semi-organized tips this is not an exhaustive list of everything you should or need to do.

I like riding bikes, I’m not a mechanic, I’m not a pro, and my opinions are subject to change. Anything you read is just my experience and nothing more.

Buying a Bike

I would NOT start off with an expensive bike
1. You may not know how much/how long you will enjoy riding
2. Your bike handling skills will be at their worst and you’ll be learning no skills (Clip-less pedals), expect to fall once or twice
3. You will probably make some maintenance mistakes
Here’s a few good threads on buying a used bike.
If you want to buy a new bike I’d recommend buying a bike with Shimano 105, I’d go around to a few bike shops to test ride some bikes, and I’d run tubed tires

Other Things You’ll Want

Get several good pairs of bibs/jerseys. This will make your rides way more comfortable. You’ll want bibs that have shoulder straps not bibs that are essentially padded shorts. Maybe go try some on before you buy…
Get a small red light to attach to the rear of your bike
Get a good hemet, check out the Virginia Tech Helmet Ratings to find a protective helmet at a good price
You will most likely want clipless pedals and the associated shoes
Get a nice pair of non polarized sun-glasses that offer a good field of vision
You should get a saddle bag to carry things like, an extra tube, a $20 bill, a multi tool, some tire levers, etc…
You will want bottle cages, the Trek Bat Cages are pretty damn good for the $$$. You should also get some bottles for your cages
Get a small hand pump to take with you while you ride, you can put this in one of the pockets of the jersey that you bought (Might as well get a decent floor pump too)

Before you ride

I like Strava or you can download an alternative tracking app. It’s really nice to have the data, it’s great for planning routes, and can be helpful with initial social interactions (Group rides, challenges, etc…). Its also very good for tracking the mileage on your components, how many miles are on my tires, do I need to bleed my breaks, should I check my chain wear… you can come to a much more accurate conclusion with an understanding of the overall mileage on these components and inspecting them.
Check your tire pressure, bring a spare tube, bring tire levers, have a multi-tool, bring food and water
Practice using clipless pedals on a soft surface (Grass) before you hit the pavement
You’ll want a bike fit, at the very least watch a couple videos and play around with your position on the bike a bit. Eventually you will want a bike fit

While Riding

If you are swerving trying to open up some food, find yourself in a lot of traffic, or are checking your phone for x,y,z reason… STOP, there are unavoidable accidents and there are avoidable accidents. Don’t hurt yourself/ your bike, just unclip put your foot down and re-orient yourself or leave the busy road in a safe manner.
Nobody gives a shit about your average speed, don’t take silly risks with some thought of setting your fastest time or keeping your speed high… not worth it
Try and ride as consistently as possible. Don’t spike your power out of every stop/turn, attempt to maintain the same power as much as possible
Let people know you’re going to pass them by telling them you are going to pass them and on which side you are going to pass them
If you find yourself about to hit a patch of sand/gravel on the side of the road do not attempt to turn or suddenly brake, apply a bit of power and ride through it. Do not intentionally take your road bike on a gravel ride it won’t perform well. When turning look out for sand/gravel as it can cause you to slide out
If you’re riding with others, give hand signs regarding road debris/pot holes, let them know when you are slowing, and point to where you want to go or let them know. Don’t half wheel people.
When taking a tight turn raise your inside knee and apply pressure through your outside foot and inside hand. Don’t break in the middle of a tight turn… break before.
Bring food! Eat while you ride especially for long rides > 90 minutes. Bonking is NOT fun and completely avoidable. If its a short ride, and I plan to do some intervals I’m bringing a lot of high carbohydrate snacks. If its a longer ride I’m bringing a mix of high carbohydrate, dried fruit, trail mix, etc…
Take in fluids and do so in a manner that’s conducive to rehydrating yourself. Don’t waste your money, mix sugar, salt and water. Uh oh… turns out sugar isn’t “bad for you” actually no food is necessarily “bad for you” its chronic over-indulgence and under-exercise that’s “bad” for you. I’m getting on my soap box here a little bit because some lady in Sprouts was telling everyone how bad sugar was for them.
Be respectful of vehicles, keep a keen eye on drive ways and entrances/exits. Watch out for newly parked vehicles and cars people are getting in/out of. If you are at an intersection operate under the assumption that you are invisible and you’re essentially playing a game of Frogger. Don’t ride against traffic. If you’re coming up on the right side of someone about to make a right they may not notice you.
Try and ride at times when there are less cars/people. This can depend on your location but early morning rides and weekends are typically better. You can use Strava heat maps to plan your routes and see where others are riding.

I want to get faster

You will want to get a power meter/heart rate monitor. This will make training much easier/reliable
Best weight/speed improvements per dollar: Get in a more aerodynamic position on your bike, better Tires/Rims, better Shoes/Pedals, better Saddle/Seat Post
Check out this channel and look up some of the basic videos on how to get faster. I think this channel probably has just about all the knowledge you need to train optimally baring hiring a good coach but that’s going to cost you $$$
Ride a lot

Basic maintenance

This stuff is pretty good at cleaning your drive train. This stuff is good at lubing your chain (it’s wax based, check out the difference between wax/oil based depending on where you live)
I thoroughly clean my drivetrain every 2 weeks and lube my chain 2x per week but this varies based on how much I’m riding. Thorough cleaning currently involves spraying the drive train with the SILCA Drivetrain cleaner and brushing the metal components to get at any dirt/grime, washing everything away with water. I use a microfiber cloth to dry things off and get in between the cassette and chain rings. Once dry I apply the Rock n Roll Gold and use a microfiber cloth, running my chain through the cloth and cleaning the pulley wheels as well.
I like to clean my bike after I ride with a microfiber cloth and/or with some baby wipes to wipe off dirt and mud.
Track the mileage on your components (I mentioned above that I use strava for this)
Put grease on your things… “Used with any type of ball bearings, threads, or press fit applications found on a bicycle, including headsets, bottom brackets, cranks, pedals, and hubs.”
I love picking up new maintenance things and learning hwo to do them, but when in doubt take it to a shop. If you’re friendly they will most likely let you watch if it’s a small ordeal like swapping out your chain

Other useful things you will eventually want

Heart Rate monitor
Bike computer
A power meter
Tubeless tires
Chain tool
Chain wear checker
A bike stand
Torque wrench
Quicklink Pliers
Valve stem remover
Sealant injector/measure

Other weird road cycling stuff

Put the arms of your sun glasses over your helmet straps
Remove your pie plate/reflectors, better yet if you bought your bike from a bike shop have them do it
Get socks that end just at the base of your calves, try and match the color of your shoes

What I learned this week 03-29-2025

Sat, 29 Mar 2025 00:00:00 +0000

What I learned this week

Software

Exploring UV

You can run a tool in an ephemeral environment uvx pycowsay 'hello world!'
uvx is an alias for uv tool run
install different python versions uv python install 3.10 3.11 3.12 and pin them in a directory uv python pin 3.11; however, by default, Python versions are automatically downloaded as needed without using uv python install
The .python-version file can be used to create a default Python version request. uv searches for a .python-version file in the working directory and each of its parents. If none is found, uv will check the user-level configuration directory.
Executes standalone scripts with run. echo "print(2+2)" > example.py | uv run example.py
uv pip freeze: List installed packages and their versions.

Cool project to visualize directory structure with tree

Fun introduction to Recursion from Al Sweigart

Updating local commits with more changes

Redshift Posix Operators~ And A ~~ is a synonym for LIKE. Pretty cool, had no idea!

Falsehoods Programmers Believe About Names, tried to read this in the Japanese translation and I can’t!

Business/Finance

Interesting Links

Math/Stats

Travel

Other

Companion Planting French Marigold seems to ward off pests!?

LA Spring 2025

Sat, 22 Mar 2025 00:00:00 +0000

Recap of my trip to LA from 3/13/2025 -> 3/19/2025

13th Flew to LA, checked the bike no problem (this was my first time flying with my bike). I borrowed a bike bag from a friend, it was pretty easy to pack. Flight delayed 1 hour, so I walked six miles in the airport. Few weirdos on the flight. Got to bed around midnight.

Bike all packed up:

The Cat:

14th

Woke up at 9:15 did a few things for work, built my bike and went to Sprouts. Very rainy but got stuff for a pot roast. We went back to Ben’s and then drove over to meet Jacob (Another 3L at USC). We did a 9 mile walk through a lake and over to dinner at tu madre. We ended the night going through a few bars before walking back to Jacob’s and driving home. Good guy, it was a fun time.

Saw this cool Isuzu Trooper:

15th

Woke up pretty late. Bike was shifting terribly. Did a 2:30 hour ride through some really nice neighborhoods in Beverly Hills. Not much shoulder, plenty of cars, but it went fine. I put the pot roast on before we left. Very simple roast just seared up some chuck and then letting in simmer in stock/vegetables for a few hours. Took the bike to REI and Brian got it shifting better. Drove home and had dinner with Gabi and Bishop. And then we walked to salt and straw and got ice cream.

16th

Woke up at 10am, watched some premier league soccer . Went for 90 minute ride up the same nice neighborhoods. Running a little late, Mitch came over and we drove to the intuit dome. Clippers smacked the hornets. Dome was extremely annoying to get into but the large screens were cool. Watched sexy beast which was ok.

17th

Woke up going to do a big ride with Jeremy (Another 3L at USC) today around Palos Verdes. Was really nice to start with a bit of traffic, the beach path was quite sandy. The coastal road was basically falling apart and closed to bikes but we went anyway there was an incredibly steep pitch. The rest of the ride was cool there was some switch backs that this McLaren kept doing over and over. Descent was a little sketchy. We stopped at Mickeys Italian deli for lunch and got corned beef sandwiches for St.Paddy’s day. After leaving, we had about an hour to get home and it started raining super hard… we were freezing. We got home and had to dry all our clothes out. Then we had leftovers and watched collateral.

18th

Drove up to Santa Barbara. Airbnb small but nice, dropped our stuff off and went for a ride. Climb was super long and steep. Saw a vulture grab a dead gopher and drop it right in front of us. Some of the coolest riding I’ve ever done. We chose the right direction to do it in, as the pavement on the descent was really nice. We showered and got Mexican food. Gave my pozole to a homeless women, got ice cream at Ralph’s.

19th

Walked to the beach in the morning and got some avocado toast. Drove back to LA. Put my bike back in its case for the flight and got dropped off at LAX. Flight was uneventful, and I made it back safe.

The international terminal at LAX is wild:

What I learned this week 03-22-2025

Sat, 22 Mar 2025 00:00:00 +0000

What I learned this week

Software

git stash list list git stashes and then you can drop via: git stash drop stash@{0} git stash

REPL stands for Read Evaluate Print Loop

Business/Finance

Interesting Links

How air breaks work. Essentially uses pressurized air instead of hydraulic fluid. Comes with the benefit of being “on” if the system breaks causing the vehicle to break… which has the tradeoff of making them difficult to move.

Math/Stats

Travel

Other

Table Creation

Mon, 17 Mar 2025 00:00:00 +0000

Collection of thoughts on table creation.

This is specifically regarding tables for analytics (OLAP queries, Data Warehouse, etc…). This is also specific to a larger organization, I don’t necessary apply these when working locally with duckDB per se.

Get a good understanding of who’s going to use the table, what they need, and when they’ll need that data by. This will help inform your data model and manage dependencies. If you are working directly with another team have them list out all of the requirements and set a date when this list will be final.

When working on a request for our finance partners, I was starting on it and everyday they wanted to add this, change that…. nip that in the bud… have them curate what they want with the expectation that it will lock once you start building it. This will force them to stop, think critically, and figure out exactly what they need. I have only had to make significant updates to this table once in the last 14 months because of this. (I guess I can’t prove causation, but I think it helped)

Don’t try and do too much in a single relation, you may think you are doing people favors by adding columns, nested data structures, and complexity but there’s a delicate balance here. A classic mistake is building extremely wide tables. These tables are for everybody and nobody because each aspect of the table wont be “good enough”. The goal is commonly to try and bring in hundreds of attributes at a specific granularity, but it turns into a dependency and data quality nightmare without being able to satisfy many questions without making comprises that nobody can accept.
Document your stuff. I’m a firm believer that what you build is only as good as your documentation. Don’t continue to contribute to the sea of blank column descriptions and opaque DAGs that your stakeholder/peers will have to waste time figuring out. You will also waste your own time explaining things to your stakeholders over and over. Instead, create a thorough explanatory resource and point people to it.
Publish it to a shared data catalog if possible… I’ve made this mistake countless times of creating local tables, them gaining a solid user base and then having to later share it and manage access across various people/software entities. A few caveats in that I would not do this if you’re still developing the table and making fast iterations… but otherwise it’s going to give you the ability to manage access, control updates, configure dependencies, and cascade relevant notifications so much easier.
Write tests and perform code reviews. We are working on better testing frameworks to role out.

The last thing you want is people/down stream jobs consuming trash that could’ve been blocked from being written to the table in the first place.
Having someone review your code/table is a no-brainer. Writing SQL against the table and using it as it would be used in production will uncover oddities/issues that would otherwise plague consumer.

What I learned this week 03-15-2025

Sat, 15 Mar 2025 00:00:00 +0000

What I learned this week

Software

Duckdb with the… UI

Tarball

Merge Statements

Git Bundle + CDNs and such

Business/Finance

Redfin getting acquired

Interesting Links

The largest ZIP Code, in square miles, in the contiguous U.S. is 30,683 square miles.

Math/Stats

Travel

Traveling to LA and Santa Barbara to visit Ben for his Spring Break.

Other

I don’t take sides I take pictures

What I learned this week 03-08-2025

Sat, 08 Mar 2025 00:00:00 +0000

What I learned this week

Software

DuckDB has a bar function which draws a band whose width is proportional to (x - min) and equal to width characters when x = max.

You could also implement this in other DBMS with ratio_to_report() and repeat()… repeat I hadn’t heard about until this week.

There’s some other fun ones in Redshift like initcap() which capitalizes the first letter of each word.

Postgres has nice docs on Character Types

There is no performance difference among these three types, apart from increased storage space when using the blank-padded type, and a few extra CPU cycles to check the length when storing into a length-constrained column. While character(n) has performance advantages in some other database systems, there is no such advantage in PostgreSQL; in fact character(n) is usually the slowest of the three because of its additional storage costs. In most situations text or character varying should be used instead.

Business/Finance

Interesting Links

Age and Cognitive Skills.

First, average skills increase strongly into the forties before decreasing slightly in literacy and more strongly in numeracy. Second, skills decline at older ages only for those with below-average skill usage.

Math/Stats

Travel

We hosted two gatherings at our house in Tucson this week. On Thursday, I used the charcoal grill I got for Christmas for the first time, cooking two tri-tips and four new york strips. A few tips on charcoal grilling:

A chimney is critical to getting the coals lit, and you need to get a decent fire underneath the coals to really get them going
You can throw more coals on already lit coals if you need to, but this will come at the cost of more smoke/flames
Try to leave the lid on as MUCH as possible as the temperature drops significantly when it comes off.
Start off with more coals then you need and create hot/cold zones by pushing the coals to one side and moving them around as needed. Just use a meat thermometer to take the guess work out of “doneness”

On Saturday, I made a bourbon braised chuck roast using seven lbs of chuck. We were going to watch a movie outside, but it was really cold so we played games inside.

Other

Reminds me of that famous tombstone inscription: “I told you I was sick!”

“It is easier to fool a man than to explain to him that he was fooled.” (Mark Twain)

The Conversation was good R.I.P Gene Hackman

What I learned this week 03-01-2025

Sat, 01 Mar 2025 00:00:00 +0000

What I learned this week

Software

Mac: To quickly find text, select some text and press ⌘E followed by ⌘G.
After copying a file, press ⌥⌘V to move the file instead of pasting a copy of it.

Business/Finance

Interesting Links

Markov Chains

One use of Markov chains is to include real-world phenomena in computer simulations. For example, we might want to check how frequently a new dam will overflow, which depends on the number of rainy days in a row.

Math/Stats

Travel

Went camping this weekend at Dead Horse Ranch State Park in Cottonwood, AZ. We drove through Prescott and up Mingus Mountain on Saturday and did a short hike to the fire tower. I was originally thinking of riding up Mingus Mountain from Cottonwood, but I’m glad I didn’t as there is no shoulder and very tight turns. We drove down to Cottonwood through Jerome, AZ which I’d like to go back and visit.

After stopping for some sushi at Safeway, we made it to the park and met up with our friends. It was very windy during the day and the trees were all hibernating. We walked around the three lagoons and headed back to the site to to hang out and start dinner. We cooked up hotdogs and smash burgers, chatted for a while, and made s’mores. There were showers directly across from out site and we got everything ready to go to bed.

Ally and I set up our tent on top of a raised platform. Our camping site happened to be the handicapped site so while most sites had a small plot of dirt on the ground for a tent, ours was raised off the ground about 12 inches by this small platform. As everyone was going to bed, I had secured all the food in our car or left it in a secured bag that animals couldn’t get into. We had some dirty dishes which I put in a trash bag and put in the car as well. For some unknown reason I decided that leaving a cooler containing steaks with no lid on it outside would be fine even after putting everything else away… I was very wrong.

Around 1am, I believe some skunks got into the cooler and were able to tear open a package containing steaks. A little later on, we believe there was a skunk fight over said steaks… this fight occurred directly next to our tent on the platform. It was a quick but loud skirmish that woke up Ally and I, everyone else was sleeping in their cars completely unbothered. The skirmish smelled awful but with no further noise and the smell being subdued by the wind or our olfactory fatigue, we tried to go back to sleep.

This was short lived, as we were quickly awakened to more snarling noises right by our tent. I was fed up and it was 2am, so I put my shoes on to go face down whatever was outside. Opening the tent and pointing it at the noises a saw nothing… because the skunk was directly beneath the tent under the platform! I investigated the cooler and saw what happened to our meat. I cleaned up the mess packed this into the car. I pounded on the platform tried to through rocks underneath to get the skunk to leave; however, this seemed to result in more noises and smells. I couldn’t even get a good look at the thing as it was mostly dirt and rocks underneath the platform it was impossible to see clear under it.

We grabbed our sleeping bags and shifted to the car, unfortunately this also smelled like skunk (possibly it was just us or how the wind was blowing). I ended up driving us away from the site to a parking lot and getting a few hours of sleep there.

That was a wild night, but we still had fun although we are certainly a bit more tired than we need to be. I’m going to have to soak the tent to get the skunk smell off of it as it to the brunt of the skunk spray I believe.

Fun times!

Booked trip to Japan, I’ve got a 9 day streak on Duo Lingo now… Japanese is hard!

Good ACQ2 episode with Luis von Ahn.

Other

Beneficentrism

It’s just the claim that it’s really important to help others—however distant or different from us they may be.

What I learned this week 02-22-2025

Sat, 22 Feb 2025 00:00:00 +0000

What I learned this week

Software

To initialize a submodule in a local repo: git submodule init which reads from .gitmodules and then git submodule update

Nice intro to cuda with embedded quiz

Docker & UV in Dagster

Click house query performance

Guide yourself to a relation db?… JK it has its place

Business/Finance

Interesting Links

Electricity

Math/Stats

Travel

San Diego for a friends engagement last weekend, was a good time. I hate driving in San Diego.

Other

Broken Legs heal faster if you walk on them

What I learned this week 02-15-2025

Sat, 15 Feb 2025 00:00:00 +0000

What I learned this week

Software

Business/Finance

Interesting Links

Math/Stats

Travel

Got back from Hawaii on a red-eye and then the next day drove to San Diego for a friends proposal. Brought my bike and road the coast with Ally which was pretty sweet.

Other

Reading a bit of Invisible Influence:

The more something matters to us the more distinctive we think it is.
Mimicking people (subtly) makes you seem more agreeable/appealing.

What I learned this week 02-08-2025

Sat, 08 Feb 2025 00:00:00 +0000

What I learned this week

Software

Some light Wikpedia reading on db normalization: Boyce–Codd normal form (BCNF or 3.5NF)

If a relational schema is in BCNF, then all redundancy based on functional dependency has been removed
A 3NF table that does not have multiple overlapping candidate keys is guaranteed to be in BCNF Elementary key normal form Functional Dependencies
A dependency FD: X → Y means that the values of Y are determined by the values of X. Two tuples sharing the same values of X will necessarily have the same values of Y. Candidate Keys
A candidate key, or simply a key, of a relational database is any set of columns that have a unique combination of values in each row, with the additional constraint that removing any column could produce duplicate combinations of values.
A candidate key is a minimal superkey,[1] i.e., a superkey that does not contain a smaller one. Super Key
A superkey is any set of attributes that uniquely identifies each tuple of a relation. The set of all attributes is always a superkey (the trivial superkey).

Committing to submodule:

cd into submodule, add changes, commit and then git push
cd .. out and back to parent repository, add sub modules directory, commit and push

Business/Finance

Interesting Links

Math/Stats

Travel

Spent the week in Hawaii which I’ll write a post on in the outdoors section.

Other

Kauai Trip

Wed, 05 Feb 2025 00:00:00 +0000

Recap of our trip to Kauai, HI from 2025-02-05 to 2025-02-12.

2025-02-05:

Landed after a 7 hour flight due to a strong head wind, but we had the middle seat open! It was raining as we walked over to Avis to pick up the rental car, but it was that nice Hawaii mist rain so it felt good.

Guy at Avis tells us there are no small sedans left (what we rented) so instead we get a 2023 Ford F-150 XLT…

We immediately drove to Dukes #America and had mediocre burgers on the beach. After Dukes, we drove to Costco in some semblance of an attempt to spend less then $1000 on food in Hawaii (things are expensive). I got some tuna jerky (which was very good).

After Costco, we did our 65 minute drive up to Princeville and unpacked at our place.

2025-02-06:

First full day, we had big plans of doing the Jurassic park trail. We drove down a small road, clipping tree-branches with the truck, and parked at the arboretum. We saw some cool Rainbow eucalyptus trees by the stream.

As we started to hike towards the trail a construction worker told us it was closed that day, so we pivoted and did the Powerline trail instead. The Powerline trail had incredible views, we did a good amount of it and turned around as it started to get a bit muggy.

We drove back towards Princeville and stopped in Kapaʻa to walk around. We bought some over priced Mango and Eggfruit and ate it on the beach. Made it back home, cooked dinner and watched the sunset.

2025-02-07:

Woke up early, made smoothie bowls, and watched the sunrise. We drove to the Kilauea lighthouse and saw whales, birds and some big waves. I was chased by a Brown Booby who was protecting its little chicks! After the light house we stopped by the agriculture center and got a big avocado and a roast beef sandwich, both were excellent.

We drove to Anini beach and found a nice spot a short walk down the beach underneath a large tree.

We snorkeled a bit and relaxed for the afternoon. We left Anini and drove to Hanalei to get some shaved ice. The sharp left turn as you follow the road down into Hanalei gives an incredible view of the mountains and beach. We cooked dinner at home and watched a bit of The Prestige

2025-02-08:

We couldn’t get enough of Hanalei, so I found the Okolehao trail which seemed to give great views of Hanalei… it did not disappoint. We only went up 3.6 miles, but it took 2 hours due to how steep and slippery the trail is. It’s very muddy after the fist viewpoint and the trail is overgrown with steep drops on either side. It continues to go on further, but it only gets steeper and muddier so we decided to turn around. I would probably not do this one (passed the first viewpoint) if it was raining or if it had just rained a lot.

After making it back to the car and changing into swim suits, we had some PB&J sandwiches, that we made earlier int he day, in town as well as some more shaved ice. We spent the afternoon at Hanalei beach. We were looking for rocks and Ally said “Oh there’s a pink one” and grabbed my big toe thinking it was a rock.

We drove back to our place and cooked up some steak sandwiches. One of the neighboring resorts over looked the ocean from some cliffs, so we brought our sandwiches over to a picnic bench and watched the ocean while having dinner and saw a whale.

That night we slept with the windows open to keep things cool, and some guy in the up stairs unit started snoring soooo loud we couldn’t believe it. We ended up switching bedrooms which gave us some reprieve. The moon was really bright this night, but I didn’t take a picture.

2025-02-09:

We decided to have an easy day because we are getting up early tomorrow. We went back to our spot at Anini and snorkeled. It was kind of cold and I wasn’t sure I wanted to snorkel, as Ally started to go out by herself, I said screw it and put on my stuff and got in. I was lucky I did, as not 5 minutes later we came across two sea turtles and got to watch/swim with them for a while! It was awesome. I got really cold and had to go back in though after ~30 minutes. Ally realized she forgot her medication, so I drove back and got that for her.

We cooked up salmon for dinner and went to bed early.

2025-02-10:

We woke up really early and went kayaking up the Wailea river and then hiked to a waterfall. It was pretty crowded as different guides shuttled hundreds of tourists to and from the waterfall each day. Leaving early was definitely a good idea, as it was quite hot on the way back. It was nice to meet some of the other people on the tour, and it was something I’m glad to have done once.

After kayaking, we drove south to the Koloa Fish Market and got poke (The tuna was really goog) and a smoothie. We continued our journey to Waimea canyon and did the hike to Waipo’o falls that I had done three years ago. There was not a ton of water flowing, but you can’t beat the views and there was still a small water fall at the end. Going later in the day, we saw very few people on the trail except for this couple that were absolutely struggling to get back up the climb to the parking lot. The drive to and from the canyon, it made the F-150 feel like a Porsche.

We then drove the two hours back to Princeville and cooked up the last of our food and watched some of the hunting party RIP Gene Hackman.

This was a huge day, but definitely worth it. If we had more time I would probably have split it up.

2025-02-11:

We got packed up and threw everything in the truck. With a red-eye not until 10pm we planned on driving around doing various activities on our last day. We went to Holey Grail Donuts in Hanalei which were really good, I also purchased a shirt from them.

After donuts, we navigated single lane bridges on our way to the Limahuli gardens. The gardens are beautiful, surrounded by steep mountains and jungle. After walking around for a bit one of the ladies asked if Ally and I were married and said we were glowing (May have just been from how pale we are hahaha).

We drove all the way back to Lihue and ate pad Thai at Big Monster 7.2/10 and talked to our server from Thailand named Tom (Very nice guy!).

Then we went back to Old Koloa town and got a smoothie from the same smoothie place. She was out of mangos, so we walked over to a guy selling produce and bought three mangos and give them to her. Ally has been using her recipes at home to make some excellent smoothies.

We then drove to shipwreck beach and hiked along the coast. We saw a lady get proposed to. Then we sat overlooking the ocean, I pointed out to the ocean to make ally think there was a whale as a joke… but then there actually was a whale that breached right as I pointed! Ally was not as impressed by this, but I still can’t believe this happened… I mean what are the chances.

Then we drove to Poipu beach and saw the turtles, the place I stayed at 3 years ago on my last visit, and a seal. We watched the sunset and then had a good meal at Kenji burger before filling up gas at Costco. Kenji burger was excellent, we will be stopping there instead of Duke’s for quick bite next time.

Last but not least, I managed to average 19.7 mpg in the f-150! I think this is a huge accomplishment.

What I learned this week 02-01-2025

Sat, 01 Feb 2025 00:00:00 +0000

What I learned this week

Software

To obtain only the remote URL of a local repo: git config --get remote.origin.url

Can also remove and add a new origin: git remote remove origin git remote add origin

Nice explain plan visualizer

Go Daddy infosec issues

Business/Finance

Interesting Links

IRS auctions

Math/Stats

Travel

Other

What I learned this week 01-25-2025

Sat, 25 Jan 2025 00:00:00 +0000

What I learned this week

Software

Splitting network into two subnets to make life miserable for wifi stealers

Redshift max recursion rows is defaults to 50,000,000 SHOW max_recursion_rows;. Seems like this is actual recursion count not rows… Testing out with the following code:

WITH RECURSIVE digits(d) AS (
    SELECT 1 AS d
    UNION ALL SELECT 2
    UNION ALL SELECT 3
    UNION ALL SELECT 4
    UNION ALL SELECT 5
    UNION ALL SELECT 6
    UNION ALL SELECT 7
    UNION ALL SELECT 8
    UNION ALL SELECT 9
    UNION ALL SELECT 0
),
    integers(i) AS (
    SELECT d FROM digits
    /* */
    UNION ALL
    /* */
    SELECT i + 1 AS next_i
    FROM integers
    WHERE i < 50000800
)
SELECT min(i), max(i), COUNT(*) FROM integers;

We get: min max count 0 50,000,800 500,007,965

And a small snippet of the results reveals an interesting pattern:

i	rowcount
0	1
1	2
2	3
3	4
4	5
5	6
6	7
7	8
8	9
9	10
10	10
11	10
12	10
13	10
14	10
15	10
16	10
17	10
18	10
19	10
20	10
21	10

I wanted to test, how the initial set of numbers propagates and how new numbers are added in each iteration. I reduced the size of digits and added a static source column in integers to track over time.

-- Smaller exmaple multi-row set
WITH RECURSIVE digits(d) AS (
    SELECT 1 AS d
    UNION ALL SELECT 2
    UNION ALL SELECT 3
),
integers(i, source_d) AS (
    -- Base case: Add source_d to track where each number came from
    SELECT d, d FROM digits
    UNION ALL
    -- Recursive case
    SELECT i + 1, source_d
    FROM integers
    WHERE i < 10
)
SELECT
    i,
    COUNT(*) OVER (PARTITION BY i) as row_count, -- How many rows we have for each value of i
    source_d, -- Where each number originated from
    ROW_NUMBER() OVER (PARTITION BY i ORDER BY source_d) as row_num -- The ordering within each i value
FROM integers
ORDER BY i, source_d;

i	row_count	source_d	row_num
1	1	1	1
2	2	1	1
2	2	2	2
3	3	1	1
3	3	2	2
3	3	3	3
4	3	1	1
4	3	2	2
4	3	3	3
5	3	1	1
5	3	2	2
5	3	3	3
6	3	1	1
6	3	2	2
6	3	3	3
7	3	1	1
7	3	2	2
7	3	3	3
8	3	1	1
8	3	2	2
8	3	3	3
9	3	1	1
9	3	2	2
9	3	3	3
10	3	1	1
10	3	2	2
10	3	3	3

I noticed:

The row count grows until it equals the number of initial digits (3 in this case)

So I guess there is an initial growth phase: i=1: 1 row (from digit 1) i=2: 2 rows (from digits 1,2) i=3: 3 rows (from digits 1,2,3)

And then this stabilizes: i=4: 3 rows (maintains all 3 chains) So my mental model is that each individual initial digit is maintaining a chain of values: Starting Digits: 1, 2, 3

Chain 1 (starting from 1): 1 -> 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> … ^ First chain always present

Chain 2 (starting from 2): 2 -> 3 -> 4 -> 5 -> 6 -> 7 -> … ^ Second chain joins at i=2

Chain 3 (starting from 3): 3 -> 4 -> 5 -> 6 -> 7 -> … ^ Third chain joins at i=3 So for instance at i = 6: i=6: source_d=1, row_num=1 (from Chain 1) source_d=2, row_num=2 (from Chain 2) source_d=3, row_num=3 (from Chain 3)

Each row at i=6 came from a different original chain, but they've all reached the value 6 through different paths of recursion.

Business/Finance

Interesting Links

Viagra laced contraband honey is pretty wild

Math/Stats

Travel

Other

Work on what’s most important not what’s most urgent.

What I learned this week 01-18-2025

Sat, 18 Jan 2025 00:00:00 +0000

What I learned this week

Software

uV is the way

Cute namesapce Zine

Great explanation of a Hash Join from the cofounder of DBT back in 2018

Business/Finance

Bad Econ of WTF happened in 1971

Interesting Links

Math/Stats

Travel

Other

Since 1989, Kenny G’s recording “Going Home” from the Kenny G Live album has become an unconventional mega-hit throughout China.

Stop building on top of bad decisions

Thu, 16 Jan 2025 00:00:00 +0000

One step back, two steps forward:

Stop building on top off bad decisions and fix the bad decisions.

Lets say you work with a bunch of people that are setting weekly goals. Each year goals are appended in a particular format to a table referenced by everyone to present goals throughout the organization.

Long long ago in a galaxy far far away someone decided set the goal date column as type VARCHAR() so now you have goals associated with strings that range from ‘2025-05’ to ‘2025-02-05’ to ‘02/05/2025’. In fact for a single goal, the weeks may be recorded in wildly different formats each year based on whoever uploads them.

THIS IS A BAD PROCESS.

This should be screaming bad process, as you disentangle different week to week formatting changes each year or worse what if someone updates it in the middle of the year and your logic just suddenly breaks… Your best way to test that logic in the first place would have been to establish a proper schema and not use a VARCHAR to store a date, but we’ve absolutely blown the rails off that one.

CREATE TEMP TABLE TEST123 AS (
SELECT '2024-12-05'::DATE AS SHIP_DATE
);

-- ERROR: invalid input syntax for type date: "2024-49"
-- Yes! This is what we want
INSERT INTO TEST123 VALUES
('2024-49');

Stop putting band-aids on and fix it… that’s the only way this is going to get better. End of story. But there’s a lot of important dependencies on this table we’d have to change a lot of logic… its a big risk There’s a much bigger risk in parsing semi-random strings to match dates then making it right.

Change your automatic response from ~~“What do I need to do to make this work”~~ to ~~“How do I fix this and make this better for everyone moving forward”~~. Everyone will benefit and your organization will be more flexible and testable.

Flexible and testable, good qualities to live by.

What I learned this week 01-11-2025

Sat, 11 Jan 2025 00:00:00 +0000

What I learned this week

Software

Symbolic Linking is pretty cool: ln -s /Users/iamsam/Desktop/symlinkorigin.txt /Users/iamsam/Desktop/symlinkdest.txt Can set up a file to point to another file…

Xcancel is an instance of Nitter.

Nitter is a free and open source alternative Twitter front-end focused on privacy and performance. The source is available on GitHub at https://github.com/zedeus/nitter

No JavaScript or ads
All requests go through the backend, client never talks to Twitter
Prevents Twitter from tracking your IP or JavaScript fingerprint
Uses Twitter’s unofficial API (no rate limits or developer account required)
Lightweight (for @nim_lang, 60KB vs 784KB from twitter.com)
RSS feeds
Themes
Mobile support (responsive design)
AGPLv3 licensed, no proprietary instances permitted

Business/Finance

Texas Roadhouse Stock Market returns outpacing tech

Interesting Links

U Haul One way movers list

Math/Stats

Travel

Other

figure 8 racing is wild

What I learned this week 01-04-2025

Sat, 04 Jan 2025 00:00:00 +0000

What I learned this week

Software

Couldn’t find where Datagrip stores its live templates.

find / -name templates and it is here:

/System/Volumes/Data/Users/my_user/Library/Application Support/JetBrains/DataGrip2024.1/templates stored as xml.

If you’re striuggling to find your local templates you can use find from a parent directory find / -name templates

Great overview of all that goes into a fab

Andy’s overview of DBs in 2024

Business/Finance

I wasn’t familiar with this ev start up until now

My 2025 prediction is that a Lucid or Rivian will foldthis year

Interesting Links

Interesting paper on immigration

Self driving 1993 Volvo

Math/Stats

On sharing Rejection

Travel

Other

Sometimes magic is just someone spending more time on something than anyone else might reasonably expect.

Damn, actually made it all the way through the end of the year doing this.

What I learned this week 12-28-2024

Sat, 28 Dec 2024 00:00:00 +0000

What I learned this week

Software

Turing Machines

A system is Turing complete if it can be used to simulate a Turing machine.

Don’t throw any “real time” analysis on a shared resource. That resource will get backed up, your “real time” analysis running every hour will stack up in the que as the cluster slows, making the backlog worse, with a ton of work that is no longer real time.

Business/Finance

Interesting Links

North Korea with the largest stadium capacity

Giant Sloths

Math/Stats

Travel

Other

So dumb, let people wear whatever they want

Black pink is a K pop band a friend told me to check out. Not my thing but they have almost 100MM subs on Youtube.

My Dad's Good Qualities

Thu, 26 Dec 2024 00:00:00 +0000

It was my dad’s birthday recently, which sparked a bit of reflection.

What are the past situations where I’m grateful for having learned appropriate behavior?
What positive qualities have I successfully incorporated into my own life?

Vanity purchases

Ever since I was born, I’ve known my dad to only be extremely reasonable when purchasing things. Never flaunty or flashy always purposeful and value oriented. He buys a Honda accord and drives it for 150K miles. You’re not turning heads but the insurance is cheaper, same with the registration, and the maintenance is low. He has no desire to impress others with money or impress upon others that he has money.

Maintenance

When my dad buys something of value, he maintains it. It doesn’t get destroyed, he doesn’t abuse it… he takes his time and understands what he needs to do to keep it in good working order and he does that. I think he takes a lot of pride in this.

He Values Experiences

Rather than flashy purchases of tangible goods my dad will buy experiences. Taking everyone out and going axe throwing, vacations, or even just going for a walk. Time with loved ones is important. I still spend ~half the year at home. I’ve been trying to hike/bike as much as I can with him because I know there’ll be a time in the future where we don’t live close enough to do that very often.

Hard work

My dad still works hard. He’s run business for half a century now… which is absolutely insane. I mean year after year, navigating a small business through five decades of changes, modernizing over time, and making it successful. After all that time, he’s still working hard. What an achievement, how many people can say they’ve successfully owned and operated a business for that long.

Integrity

One of his biggest ideals has to be integrity. He hammered that home to us as kids. Do things with integrity. We had a “family constitution” plastered on the fridge in high school to remind us, and I think integrity was point number one hahaha. I guess it worked… I won’t forget it.

My dad will do a fair bit of listening in social situations, especially in larger groups. Not dominating or being overly loud. He knows when to interject, sometimes he’ll tell a story. One thing I’ve picked up recently is he never talks bad about anybody. He doesn’t like the Cardinals current quarterback, but other than that you’d never hear him just say something mean about someone. Not much good ever happens from saying mean things about people.

Health

My dad’s had his fair share of injuries and health concerns since I’ve been alive. However, he’s always stayed active. Some of my fondest memories are out hiking, biking, rafting, camping, etc… with him.

Justice

Whenever my dad hears about an injustice being done, he can’t stand it. He will go way out of his way to help someone out of nothing more than fixing something that is total bullshit. This is probably why most of the people who work for my dad have been there for decades because he won’t put up with any mistreatment whatsoever. I think this goes hand in hand with integrity.

Other

He once showed me a list of things he wanted to accomplish in his life that he’d written down in his late twenties that I really enjoyed.

Buying a new bike as a kid, I really wanted a cruiser with flames on it. My dad said I’d be wanting a mountain bike after a few weeks on the cruiser. I ended up buying the cruiser, and I also ended up wanting a mountain bike after a few weeks. Ever since then, we agreed that if my dad says “bike” I know I need to deeply consider the decision I’m about to make.

What I learned this week 12-21-2024

Sat, 21 Dec 2024 00:00:00 +0000

What I learned this week

Software

Better python notebooks

Map of github

Bloom Filters to speed up Joins in SQLite

A good deep dive on Bloom Filters from Sam

Bloom filters can give definite “no"s, but they can’t be certain about “yes.”
A bloom filter will never give a false-negative, and this is what makes them useful.
Google’s BigTable is a distributed key-value store, and uses bloom filters internally to know what keys are stored within. When a read request for a key comes in, a bloom filter in memory is first checked to see if the key is in the database. If not, BigTable can respond with “not found” without ever needing to read from disk. Sometimes the bloom filter will say a key might be in the database when it isn’t, but this is fine because when that happens a disk access will confirm the key in fact isn’t in the database.

Business/Finance

Interesting Links

Math/Stats

Travel

Palau looks really cool

Other

Got a bike a professional bike fit this week. Highlights how tiny changes in position can make a huge difference especially over thousands fo miles.

Randomly finding the best tetris player in the world

Radiation

Butcher’s Crossing, adding to reading list

What I learned this week 12-14-2024

Sat, 14 Dec 2024 00:00:00 +0000

What I learned this week

Software

DuckDB interview & cocaine fund.

You find yourself in a quest for productivity, feeling productive because of the quest, but not really doing any productive work.

DNS rabbit hole Stripe Blog

Business/Finance

Interesting Links

Math/Stats

Travel

Other

Interesting things learned

One thing I found particularly absurd

Inhaling carbon monoxide for performance

Human brain is unreal

What I learned this week 12-07-2024

Sat, 07 Dec 2024 00:00:00 +0000

What I learned this week

Software

Great video on threading

Cluster by, partition and sort does not guarantee a total order of output.

Business/Finance

Interesting Links

Math/Stats

Travel

Other

Movies to Watch: Barry Lyndon, Casablanca, and Network.

Went to chess club on Wednesday, played over the board for the first time in several years. I played a decent French Defense in my first game and ended up losing after miscalculating. I completely imploded when playing another guy losing three games quickly and even accidentally swapping the position of my king and queen when setting up the board. Over the board chess is a skill!

Trying to a little better at gift giving this year, I’m taking notes of things people want/need as they mention them and its working fairly well.

What I learned this week 11-30-2024

Sat, 30 Nov 2024 00:00:00 +0000

What I learned this week

Software

Iceburg

S3 Conditional Writes

S3 is a database

Business/Finance

Interesting Links

Math/Stats

Travel

Other

Enjoyed spending time with family on Thanksgiving. Tried my best to do focus as much as possible with them and be off the phone. I worked a few days during the week but was very efficient with my time and made sure I stopped working and stayed not working through evenings and early mornings.

I bought my girlfirend a bicycle, we’ve got it all set up now and I’m really excited for her to come out and ride with me.

DRY Analytics

Sun, 24 Nov 2024 00:00:00 +0000

I was talking to one of our partner teams in a developing market. They are a newer analytics department of savvy Business Analysts & SCMs (Supply Chain Managers).

Long story short, they are working on this new metric and I was like, How are you filtering for our specific products in your marketplace? And they were like: Oh we have this case statement that we include in all of our jobs…

SELECT COl1
     , COL2
     , COL3
     , ...
     , CASE WHEN PRODUCT_WEIGHT > 45 THEN 'OUR_PRODUCT'
            WHEN PRODUCT_LENGTH >= 37 AND PRODUCT_WIDTH >= 33 THEN 'OUR_PRODUCT'
            WHEN CATEGORY = 'SOME_CATEGORY' OR EARTH_IS_ROUND_FLAG = 'Y' THEN 'OUR_PRODUCT'
          WHEN COLOR = 'RED' THEN 'OUR_PRODUCT'
          ELSE 'NOT_OUR_PRODUCT' END AS PRODUCT_DEF
FROM BIG_TABLE
WHERE PRDDUCT_DEF = 'OUR_PRODUCT'

Quick aside: When I write Job, Profile, Model, or Asset I’m talking about SQL that creates some result set and is most likely scheduled to run at a specified interval or once all dependencies have satisfied.

I was like ohhhh man can I save you some pain. Maintaining this kind of logic in tens or worse hundreds of jobs is going to inevitably suck! At some point someone’s going to come along and say:

 Hey… color = 'BLUE' products should also be in our list of products.

Now when this day comes and you’ve got a bunch of different teams that all maintain product definitions… someone is going to have to coordinate this migration and get the timing right. Heat flashes across your forehead and your palms get a little sweaty. For so long we’ve been copying and pasting case statements but we’ve officially reached a crescendo of misaligned definitions and numbers not matching up. The CAT is way out of the bag on this one, it does NOT want to go back in the bag and you need it back in the bag if you ever want to make it to the FIFe World Show.

This change will need to be cascaded across all the teams that maintain such a definition

a. Some teams might disagree and say Blue products shouldn’t be included

b. It may be difficult to locate everyone who is defining products so there will be some level of uncertainty that everyone has been notified

c. And still others will ask, since when did we define OUR_PRODUCT with a color dimension????
Somebody won’t have bandwidth to implement the change and will need to push the date back, others may need time to figure out where exactly they are defining product definitions in their pipeline, someone’s OOTO. This will bump around a bunch of teams and there’ll be several calls to organize.
Some teams may have different data models, we only keep the latest definition, we keep a versioned definition over time, we have a denormalized table and qualify row_number() over(partition by product_id order by last_updated_date desc) = 1 every single time to derive the most up to date definition.
Now that we’ve updated the definition do we need to rewrite all historical tables that were derived using the previous definition? Now those blue products are OUR_PRODUCT but at any point time pre-change they would never be OUR_PRODUCT and would not reside in any stored result in any relation. And don’t forget this is happening across multiple teams in parallel and that change will need to be cascaded through several sequential jobs!

Ok so how do we avoid pure pain?

Maintain a centralized definition.

DENORM_PRODUCT_DEFS [Captures all product changes over time] denormalized table appending all changes to products over time. Our products may change color or a vendor my adjust the packaging changing its dimensions this can all impact the classification of a specific product. This table could hold when the records were first inserted, when they were most recently updated, and denotes the active definition. This will save you a bunch of LEAD/LAG stuff if you’re reading from this table a lot.
VERSIONED_PRODUCT_DEFS [Tracks historical versions for auditing and reproducability] versioned table that contains a snapshot of the product definition at a specific point in time. Back in JUNE we used this definition which would have resulted in this classification of these products and now in OCTOBER we use this definition which would have resulted in this classification of these products. This table would nicely be partitioned on version and would also be great to have columns denoting start and end dates for the version.
FACT_PRODUCT_DEFS [Current, streamline view for active use cases] only the most up to date definition for products that is pre-filtered to only include your products.

So now we have profiles that look like:

SELECT B.COl1
     , B.COL2
     , B.COL3
     , B...
FROM BIG_TABLE B
    INNER JOIN FACT_PRODUCT_DEFS F ON B.PRODUCT_ID = F.PRODUCT_ID

And even better, now we can update how this reference is configured and if we have this query written fifty times everything points back to that reference and is updated [ref dynamically resolves table and schema names, so changes to a single reference propagate across all queries]:

SELECT B.COl1
     , B.COL2
     , B.COL3
     , B...
FROM {{ ref("BIG_TABLE") }} B
    INNER JOIN {{ ref("FACT_PRODUCT_DEFS") }} F ON B.PRODUCT_ID = F.PRODUCT_ID

Distinct logic across teams is pain. Host a central definition. Argue about said definition, update said definition, cry about said definition, whatever… but allow everyone to read from a single definition.

Any time you see the same code written over and over alarm bells should be going off, there is a fire in the kitchen! Not only is this just MORE lines of code, but every instance its re-written is another opportunity for logic to become stale or be slightly miswritten. DRY DRY DRY. Do not pour water on the grease fire in the kitchen, suffocate it with a deluge consistent data/reporting instead.

This is such a common source of the “This number doesn’t match that number” problem. This is the “but it works on my machine” equivalent for day to day analytics.

The numbers don’t match up because distributed hard coded definitions are hosted across teams using different dependencies running at different times… I would be FAR more perplexed if the numbers DID match up. In fact I’d be completely dumb founded. The numbers have absolutely no business to be matching up because they don’t have a data model/governance to support them actually doing so. Its not even automatically testable because there’s no way to determine where the code diverges without, I guess, literally parsing jobs as text to rip out the dependencies and business logic.

There are so many different “types” of what I described above. For instance, possibly you have a data migration campaign and need to prefix all of your jobs with a new Database. In current state, hundreds of profiles have FROM clauses like FROM SCHEMA.TABLE but now needs to be FROM DATABASE.SCHEMA.TABLE to every single one… hopefully this illustrates why the above idea of ref is so helpful.

I’m not here to praise DBT but the Jinja templating solution to make SQL more dynamic handles some of these problems nicely. I think a better data model can resolve some nasty issues, but if you’re repeating logic or referencing specific relations over and over across profiles, assets, whatever… be prepared for pain.

If you’re working with an analytics team, one of the worst things that can happen is an erosion of trust in data quality from that team’s stakeholders. If you ordered crayons from Amazon and you receive pencils, or you set up an AWS billing alarm to send you an email when your spend has exceeded $100 and it doesn’t… you lose trust. One of the key pillars of a successful analytics team is building trust in their outputs (Also, move quickly I need this data yesterday lets go lets go… but that’s a given).

This is the unglamorous plumbing that underpins a successful analytics organization. While you can never fully control the madness that occurs in a 30 tab excel file authored by ten people off of windows shared drive you can rest easy knowing that at least whatever was copy pasted in there was reputable and “correct”.

What I learned this week 11-23-2024

Sat, 23 Nov 2024 00:00:00 +0000

What I learned this week

Software

Add line numbers to head: head -n 5 example.csv | nl

R cheatsheets

Hamming codes

When using psql from the command line don’t forget to end statements with a ‘;’.

Business/Finance

Acquired Ikea episode

“It is not all that difficult to reach set targets if you do not have to count the cost. Any designer can design a desk that will cost 5000 krona. But only the most skilled can design a good, functional desk that will cost 100 krona. Expensive solutions to any problem are usually the work of mediocrity. We have no respect for the solution until we know what it costs." – Ingvar Kamprad

Interesting Links

Math/Stats

Travel

Other

“I would say that you have fallen into the commonest fallacy of all in dealing with social and economic subjects—the ‘devil theory.’ […] You have attributed conditions to villainy that simply result from stupidity. […] You think bankers are scoundrels. They are not. Nor are company officials, nor patrons, nor the governing classes back on earth. Men are constrained by necessity and build up rationalizations to account for their acts.” ― Robert A. Heinlein, Logic of Empire

There may be another side to that

Sat, 16 Nov 2024 00:00:00 +0000

Intelligent and deep-thinking people may hold opposite opinions of one another. If you hold an unshakeable opinion about anything that exists in some state of probability or confidence its almost certainly a mistake to be doing so.

One of Amazon’s leadership principles is Have Backbone; Disagree and Commit. I wish it would elaborate a bit further, Have Backbone; Disagree, Reevaluate and Commit. It shouldn’t be disagree, hunker down, and ram my opinion through… but rather an exploratory process where people that respect each other’s opinions get closer to the truth.

There are certain physical facts that aren’t worth disputing. Anybody arguing that I am NOT sitting here physically typing this at my computer would be wrong. Its undeniable and for better or worse I’m typing these words into some word document (Document2).

Now let’s pick at something with an air of doubt about it: Is the United States economy doing good or bad?

Some people may say “bad”, some may say “good”, and some may say:

“What a terrible question, how can a complex economy be broadly categorized as good or bad”.

While this example is heavy handed, it certainly demonstrates the need for clarifying questions.

“Oh… I said “good” because I assumed we were talking about the performance of a market-cap weighted index of U. S. equities.”

“Oh… I said “bad” because I was talking about real GDP growth slowing year over year”

If you look around, deep questions are reduced to simple statements and assumptions all around us.

Political candidate ‘X’ is the sole perpetuator of problem ‘Y’
The project failed/succeeded because of ‘Z’
The P value was < .05, so we should reject the Null Hypothesis…

How can one possibly attribute the entirety of a complex issue to a political candidate, the failure of a project to a single problem, or why even use .05 as a cut-off for rejecting the Null why not .04, why not .049?

“Man’s mind cannot grasp the causes of events in their completeness, but the desire to find those causes is implanted in man’s soul. And without considering the multiplicity and complexity of the conditions any one of which taken separately may seem to be the cause, he snatches at the first approximation to a cause that seems to him intelligible and says: This is the cause!"

– Leo Tolstoy, War and Peace

Probabilistic thinking is such an important skill as we fight the urge to quickly categorize into binary values: buy or don’t buy, right and wrong, good and bad. This isn’t an argument for never making a decision or paralysis by analysis, but rather for careful consideration of beliefs, a willingness to update opinions, thinking in probabilities and asked clarifying questions in light of a vague inquiry.

I think we are all on some gradient descent toward truth. We wake up every day with a goal of minimizing loss (regret) through being correct, producing positive change in our environments and updating our beliefs. We inch toward this over time, but… our calculus is a bit rusty, sometimes we take a step in the wrong direction, sometimes we over shoot our destination, and sometimes we find ourselves stuck in a local minimum and only with great difficulty can we eject ourselves and find the global minimum.

This is an arduous but worthwhile process.

“The inactive investor who takes up an obstinate attitude about his holdings and refuses to change his opinion merely because facts and circumstances have changed is the one who in the long run comes to grievous loss.”

―- J. M. Keynes

What I learned this week 11-16-2024

Sat, 16 Nov 2024 00:00:00 +0000

What I learned this week

Postgres knowledge dump

Software

Business/Finance

$26BB of BTC $71BB market cap…

Interesting Links

Math/Stats

Travel

Other

Cool fossils building a picnic table I really enjoy Beau’s videos.

What I learned this week 11-09-2024

Sat, 09 Nov 2024 00:00:00 +0000

What I learned this week

Software

Nice overview of jq, was using the aws emr cli to describe clusters so I could SSH into them aws emr describe-cluster --cluster-id and decided to use jq instead of grep -i dns

I guess you can use spark in Athena

echo {1..10} returns 1-10

< lines awk 'NR%2 == 0' filtering raw lines out of text file with awk is pretty handy

< lines sample -r 10% quick “random” sample

In an awk rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for every input line. If the action is omitted, the default action is to print all lines that match the pattern.

So, awk '/li/ {print}' mailing-list == awk '/li/' mailing-list

In make, it’s all about files. make only cares about the name of the target. It does not check whether a file of the same name actually gets created by the rule.

git shortlog command is used to summarize the output of git log.

git stash branch <new branchname>, creates a new branch, checks out the commit when you stashed your work, reapplies your work there, and then drops the stash if it applies successfully.

Business/Finance

Interesting Consulting Read RIP Chegg

Interesting Links

Math/Stats

Travel

Other

“A man on a thousand mile walk has to forget his goal and say to himself every morning, ‘Today I’m going to cover twenty-five miles and then rest up and sleep.” ― Leo Tolstoy, War and Peace

Pizza & Orange Juice

Sun, 03 Nov 2024 00:00:00 +0000

Last Tuesday I went to The Parlor. I was back in Phoenix and wanted some pizza. I couldn’t find anyone willing to drop everything and go with me, so I went by myself. I rarely go to restaurants by myself, its somewhat uncomfortable because I feel limited in what I can stare at. I’ve been trying to fight the feeling that I always need to stare at something. Sometimes, I get pretty in my own head about what/who I’m staring at, how much eye contact to make, etc… Anyway, reading that back that sounds pretty weird, but I’m just going to leave it… this is off the cuff.

Ok so I go to the Parlor it’s ~8pm and not busy at all. I’m sitting at the bar, and I’m just staring at the guys making the pizzas. Now they seem to notice that I’m just staring and one of them looks at me and goes “Are you waiting to pick up?”… and I was like “Oh… no, I like watching.” He goes back to his thing and then after a few minutes tells me “This one is yours” and hands me the pizza directly. At this point I was a little worried that I pissed him off because I was staring or something idk.

I’ve chowed down about half the pizza and I notice the chef walking behind me. I tell him the pizza was great as always, and we ended up having a thirty-minute conversation. Turns out that was the executive chef at the parlor, Zeus Miranda-felix. He’s a super nice guy and makes some of the best pizza in the state. He’s been working since 8am chopping 50lbs of onions, preparing dough, and now its 8pm and he’s still happy to spend half an hour chatting with some random guy watching him make pizza. He’s in his late twenties and told me he’s dedicated his entire adult life to cooking… his art. He’s dedicated his entire adult life to meticulously getting the small things right every day.

When I was a kid my brother and I would go over to my Grandparents house, climb up into the orange trees, and pick bags of oranges. My mom had this little juicer and we’d run all the oranges through it until we had a good jugs worth. Just one ingredient… unfiltered orangey goodness. We’d finish playing outside and want a glass of OJ before water.

Could we have just gone to the store and bought a bunch of juice? Sure. Unfortunately, most orange juices from the store are kind of garbage. They taste like the container… there’s just something off about them. There are probably many difficulties with mass producing orange juice at scale, having it sit on a truck, on a shelf and comply to regulations that slowly erodes the flavor.

I think orange juice is a nice encapsulation of getting the small things right. If your orange juice is good, it means you took the time to get nice fresh oranges and dealt with the slight inconvenience of juicing them. It’s one ingredient, there’s no hiding behind gobs of fat and salt and your customer is probably not staring intently as you make it. It’s pretty common to find on menus. It’s also pretty easy to cut corners on.

I’m not going to right hook anybody with a shitty analogy now but the details matter and orange juice speaks volumes. The Parlor actually doesn’t have orange juice on their menu, but it’d probably be good if they did. I’d like to have an orange tree in my backyard. I wish I lived closer to The Parlor.

What I learned this week 11-02-2024

Sat, 02 Nov 2024 00:00:00 +0000

What I learned this week

Software

arp -a The arp utility displays and modifies the Internet-to-Ethernet address translation tables used by the address resolution protocol. -a displays or deletes all of the current ARP entries.

I passed the AWS Certified Cloud Practiioner Exam this week. I studied for a few hours to make sure I was familiar with some of the more niche offerings and all the marketing jargon that will appear on the test. Overall it’s really just a lot of content, if you’ve used AWS for some time most of it is failry rudimentary.

Most useful things to study would be niche offerings (Wavelength, Lex, Rekognition, …), Networking (VPC, VPN, WAF, Shield, Route 53, Subnets, Regions, AZs, Local Zones, Outposts, Securiy Groups, NACLs, ELB …), and the shared responsibility model.

Business/Finance

RH Election markets…

Saline Shortage This industry kind of baffles me so I spent some time reading about it. Basically Baxster ($BAX) produces the majority of Saline for the US. Saline in and of itself is not that complicated but producing it at scale while complying with many regulations seems to be very complicated. It seems as though its VERY low margin for these manufacturers. This stuff also takes up a lot of space… Low Margin + Product that takes up a lot of space and is heavy is usually a nasty combination especially when it comes to logistics. It also seems that becasue of the margins at play, it only becomes econmically viable to produce at a few specific locations. When those locations get hit with a massive hurrican it just knee-caps production leading to shortages as there is no room for redundancy in the supply chain. Seems like production is fairly inelastic and can’t be ramped up or down fast.

ALL that being said, it seems like hospitals are still charging outlandish amounts for Saline. This is getting into the messy world of health care costs and billing that I have no authority or knowledge to write about, but I thought it was interesting.

I can only imagine the complex pipeline all the way from manufacturing to administering this to a human… and the variety of incentives, regulations, and people inserted into this process along the way.

Interesting Links

Math/Stats

Travel

Other

Aesthetically pleasing facts live longer

What I learned this week 10-26-2024

Sat, 26 Oct 2024 00:00:00 +0000

What I learned this week

Software

**.suffix in git ignore, especially helpful if you have a 100mb database locally and a write ahead log.

Postgres MVCC

Business/Finance

Interesting Links

Space Suits

Math/Stats

Travel

Other

Building a Neural network in pure SQL is hard!

What I learned this week 10-19-2024

Sat, 19 Oct 2024 00:00:00 +0000

What I learned this week

Software

Shell

You can us curl to make requests using the DICT protocol curl -s "dict://dict.org/d:windmill" | trim

Other curl things: -I for HTTP header, -L for redirection

unpack handy tool for decompression

You can use rm and a pattern to remove all files in a directory rm E1F* removes all files that start with E1F

Dagster + DBT

Dagster Assets = DBT Models: dbt models are named using file names, which Dagster will also use to create the corresponding asset’s asset key.

Python venv [Stashing because for whatever reason I forget the syntax]

python -m venv .env
source .env/bin/activate

DuckDB, you can copy a result set directly into a new csv file in your working directory

COPY CHECK123 TO 'output.csv' (HEADER, DELIMITER ',');

Athena performance tips

Business/Finance

7-11 closures Absoultely insane fraud from a Jags employee

Interesting Links

Math/Stats

Back Prop granular example

Travel

Other

Accountability sinks

“You can catch more flys with Honey than vinegar”

The Art of attention

On Context

What I learned this week 10-12-2024

Sat, 12 Oct 2024 00:00:00 +0000

What I learned this week

Software

ACID

GIT_CURL_VERBOSE=1 git pull --rebase interesting verbose info on HTTPS

A few more linux commands: seq generates series of numbers rev reverses input, prints to stdout wc outputs count of lines, words, and characters

I was attemping to pull changes from a remote repository, running git pull rebase and just randomly got the response:

BUG: remote-curl.c:1528: The entire rpc->buf should be larger than LARGE_PACKET_MAX
error: git-remote-https died of signal 6
fatal: expected flush after ref listing

I’d never gotten this before.

I tried increasing http.postBuffer git config --global http.postBuffer 524288000… nah
git fsck --full && git gc… nah
brew upgrade git from 2.46.0 -> 2.47.0… nah
Ok… no more HTTPS lets use SSH instead… works fine

Not great info online and I’m not sure exactly why this randomly occured. Also the data I was fetching and merging was… tiny:

git pull --rebase 
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (4/4), done.
remote: Total 9 (delta 7), reused 7 (delta 5), pack-reused 0 (from 0)
Unpacking objects: 100% (9/9), 850 bytes | 44.00 KiB/s, done.

Reviewed DBT:

Model: Just SQL file, added in models folder
Version Control: Connect to GIT or GIT Lab, allows for analytics folks who don’t get to easily create branch, merge, push and pull
Preview: Run model and return limited result set back to DBT cloud IDE
Connection: Underlying DW that DBT uses for compute (Snowflake, Databricks, Redshift)
Jinja: Templating engine that allows you to reference other models, define how you want your model to be materialized in your DW [Running Compile in the ide shows the actual SQL]
DBT Run: Takes all you models, looks through all the code, wrap it in correct DDL and construct objects in DW in order of the DAG
DBT build: just Run + Test for each step in DAG
Lineage shows dependencies [Jinja this is the REF function used in the downstream model]
Sources [From Fivetran, Stitch, etc…] -> Staging -> First Layer models -> Fact Models / Dimension models
dbt_project.yml: Can tell DBT to build all models under a certain directory as tables instead of doing this with {{config}}
Sources: Document raw tables in yaml, then you reference these tables using jinja, so if the names change you only need to update this file not every single model. Can also monitor source freshness dbt source freshness
Tests: Defined inside yaml that lives with a models directory, supports simple tests against columns out of the box and customer SQL tests by adding models under the tests directory
Add descriptions to your yaml as well, this can be and probably is better done via markdown. dbt docs generate spins up a small web page detailing the docs
Deploying: Deploy models to production, dedicated branch, schema, and schedule/run dbt commands. To do this create a prod env in the deploy tab

Business/Finance

Interesting Links

Great tutorial for anyone brand new to building websites

Math/Stats

Travel

Other

Why is the speed of light so fast

Wide Tables

Sun, 06 Oct 2024 00:00:00 +0000

A few considerations before building that extremely wide table.

Every once in a while, an idea comes along to build a table with a wide array of columns at a particular granularity. This is usually associated with “self-serve analytics” and/or an attempt to define a “source of truth” table. Wide tables, self-serve analytics, and source of truth tables aren’t inherently good or bad… but there are some considerations to this type of table that I’m going to rant about.

When I talk about a single “wide table” I’m kind of talking about a denormalized table just an especially wide one with many loosely related columns. The opposite of this would be a normalized design where one would “store different but related pieces of information in separate logical tables (called relations)”. I don’t want to linger on this topic too much but typically a denormalized scheme speeds up reads at the expense of writes, and vice versa for a normalized schema.

I’ll also mostly be talking in the context of OLAP, not OLTP.

Lets define our wide table

For the sake of the rest of this thing lets define a wide table.

You work for a company that sells things. You record data on what was sold, how much was sold, what was projected to be sold, how much of that thing you had, how many distinct customers bought that thing, how quickly did you get that thing to your customer… I’m not going to hash out the full DDL for this table but you get the picture.

It should be apparent the sheer number of unique metrics that can be generated just from selling things!

You know what would be really nice? What if we decided on some granularity say… for a particular thing in a particular week we just record every metric about that thing for that week! So, a unique row in our table will be a combination of Thing + Week… and then we’ll store the name of the thing, the inventory, the shipments, the forecast, etc…

Note that this is a BATCH solution where data is written periodically at specific intervals in time… NOT a streaming solution where shipments are recorded in real time. Maybe the upstream sources this table is generated by are populated via some Kafka/Redpanda/Kinesis situation.

Why would you want to do this?

Maybe this table is for an LLM or ML based query generation solution, pulling data for non-technical users with ad-hoc business questions, maybe it’s to simplify the SQL associated with these ad-hoc queries to move joins/other logic upstream, speeding up reads (SELECT), or maybe the table needs to have a ton of columns for some other reason. That’s fine! Standardizing definitions is generally a good thing and improving the speed at which ad-hoc questions are answered is also good!

Dueling parties wielding Excel files generated by random queries stored and iterated on in Notepad, resulting in different outputs is always annoying. It’d be easier to trouble shoot if everyone was relying on the same source and using some form of version control (I know I’m only dreaming at this point). It’d be even better if you didn’t need to join any other relations on that source, simply SELECT & FILTER and you’re good to go, thus, our wide table.

What are some issues? [Begin Rant]

This table is now a dependency bomb.

All of these columns are derived from upstream tables, your inventory data is coming from scan events at your warehouse, outbound shipments from another source and data science is producing the forecast in a local Python Notebook triggered via CRON that writes to the windows shared drive, you’ve asked if this could be moved to sage maker and the resulting dataset cataloged and directly written to the data-warehouse but that’s “below the line” for this quarter.

You could very well now have a ball of 70 dependencies.

If any of those dependencies are delayed every metric will be waiting… so instead of just having the guy who needs to report inventory levels to the director on Monday pissed, everyone is pissed.

Now you’re like ok… if any dependency hasn’t satisfied by Monday at 5:00 am, it doesn’t matter I’m just going to force the job so that at least some people get their data. However, now you’ve just written dirty rows to this table… the inventory guy is still pissed because inventory is half of what it should be because you forced the data set before the latest scan events were written to the upstream table. You also KNOW that by forcing dependencies you’ve got incorrect data in your table and you’re just going to have to re-run your transform job again and actually let the dependencies satisfy this time. If you have a single transform job that means running everything again on your compute even if it’s already been written to the table successfully. You’ll also need some mechanism to notify the inventory guy that his numbers are going to be garbage Monday am, so he doesn’t accidentally report something ridiculous to the director.

Inventory guy does not like you he thinks you’re an ass, you think he’s an ass, inventory guy keys the right side of your 2022 Toyota Avalon, the Toyota Avalon is now discontinued and you’re sad because you should’ve waited and purchased the Toyota Crown, but you didn’t realize the Crown was coming out so you bought an Avalon, anyway… right hook to inventory guy! Now you’re being charged with assault! Now you’re in jail, your arraignment isn’t until Tuesday, but you need to get inventory numbers out by Monday! Who’s going to force those dependencies???

What about storage?

With this wide table, we’ll have the name of the product, maybe the product belongs to a broader group of products and we’ll throw the description of the product in there as well.

These facts about the product may change… but probably fairly infrequently.

In our table every week the same values will be included for those products over and over. Instead, they could be stored in a separate table that contains each product, the description, color, etc… and then that could be joined to our fact table. Now we are again beginning to talk about Normalization which is critical to think about when setting up your relations.

A few solid rebuttals to the above statement given that we are mostly talking about an OLAP situation:

Modern columnar storage formats like Parquet are efficient at compressing data, especially repeated values in a column… something like MVC
Data storage is relatively cheap… if we are only talking in the millions of rows, it’s just not going to be a major driver of cost
Denormalized data often performs better for analytical queries, with columnar storage, queries will only need to scan the columns as denoted by the query

A few points on Inserts/Updates/Deletes

This is where denormalization can be slower and get more complicated. Let’s say we want to change the product group that one of our products belongs to in the denormalized table. We’ll want all historical data to be attributed to the new group so we’ll need to update the previous product group in every row.

We can also launch a new product, but it’s possible data science hasn’t finished the production forecast for that product yet… now the forecast column is NULL because we aren’t going to stall inserting the rest of the columns for that product just because the forecast is NULL. However, people that read from this table will need to understand that the forecast is NULL and choose how to deal with it accordingly. If an analyst is tasked with deriving forecast accuracy, and they notice some NULLs and just NVL(forecast_column, 0)… that’s just not true, science has a beta forecast for the product and its certainly > 0… there’s a reason we decided to launch the new product after all, we predicted customers will want to buy it!

To be fair, if you launch something like the Pontiac Aztek maybe anticipating 0 demand is the correct choice when attempting to sell an angry kitchen appliance of a car.

Where will your data model break down?

Q: Hey I know we have that inventory column, could we break it out into the different inventory dispositions (Sellable, In transit, and Damaged)?

A: Sure, now our table increases in... width...

Q: Hey I know we have shipments for all of last week, and its only Thursday but I’d really to see what we shipped out Mon/Tues/Wed of this week to see where we are at.

A: We cant really answer that with the demoralized table at the current granularity… even if it was daily we’d probably be waiting on dependencies and wouldn't be able to get back to this person until Friday. Now this ask simply becomes a query against the upstream shipments table for the latest data.

Q: Hey I’m using that wide table you made but I want to see how the descriptions for this product have evolved over time; however, all the descriptions are the same in this table?

A: This question will need to be directed towards another table that contains this info as the denormalized table has been updated to only reflect the latest definition.

Just as with anything there are tradeoffs. Above, I’m posing specifically tricky questions of the table, but it would do extremly well with basic aggregations and reducing the knowledge burden for end users of all the different upstream tables.

I only bring up these points because I think they are relevant to think about when designing how you want to capture your data. Again, there is no right or wrong, it’s simply context/data dependent and the right choice will be different across different teams. However, there is a clear right and wrong when it comes purchasing a Pontiac Aztek; I’ll leave that exercise to the reader.

What I learned this week 10-05-2024

Sat, 05 Oct 2024 00:00:00 +0000

What I learned this week

Software

grep -i for case insensitive search (I always forget this flag for some reason so writing it down)

cd -

history 5

Business/Finance

Taxes

Interesting Links

https://moxie.org/2013/01/07/career-advice.html

Math/Stats

Travel

Looked at Saturn through 16” telescope at Kitt Peak observatory an hour outside of Tucson. Will need to come back and do the climb up to the obervatory on a bike. It sounds like they have some additional cool exhibits opening up that would be fun to check out.

Other

A light year is the equivalent of 5.9 trillion miles.

“Their stuff is shit, and your shit is stuff!”

Fat bear week

https://polarisprogram.com/dawn/

Insane amount of watts.

What I learned this week 09-28-2024

Sat, 28 Sep 2024 00:00:00 +0000

What I learned this week

Software

Cool visual code editor

TMUX after creating a new session tmux new -s gvbackfill: Split window vertically CTRL + B, # Split window horizontally CTRL + B, : and then type split-window Switch window CTRL + B and arrow key

AWS Athena is based on Trino, I don’t know why I never know this, but yeah I guess it is.

Plain text… SMH

Business/Finance

Imports dont subtract from GDP

Interesting Links

Thinkig about what I’d like to do for MTA open data challenge I’m thinking something related to buses… maybe something with the fastest bus?

Math/Stats

Travel

Other

Megalopolis was one of the worst movies I’ve ever watched. I understand its trying to do something, but… wow!

Movies I’ve enjoyed recently: The Killing Fields & At Close Range

Movies I want: to watch: Dante’s Peak and At Rivers Edge

I tried making Tomato Jam… but I kind of just made ketchup. I made some Pumpkin Spice Coffee Cakes with my friend Lauren and they turned out pretty well even though we kind of mixed the flour too soon.

I want to try and make Norwegian crisp bread.

Thoughts on personal finance

Sun, 22 Sep 2024 00:00:00 +0000

$$$

I wanted to quickly jot down my current positions on personal finance topics. This will be quite interesting to see if I maintain these views into the future or if I change my mind.

On Saving

Saving money now (Currently 25 years old as of writing) is incredibly important due to the nature of compounding a dollar saved now is extremely important given the long time horizon it has to compound.

I currently live below my means, and hold a high savings rate most months.

I avoid status seeking purchases. Sometimes I have urges to make these purchases, but I frame them as large drains of my limited time… this makes them extremely off-putting to me. For instance, I could buy a $10,000 outfit to show off my status and attempt to attract attention. The tradeoff of my time to do so simply isn’t worth it.

$10,000 compounding annually for 40 years at 5% interest turns into $70,399.89.

Purchasing power will be reduced due to inflation, the 5% return is an assumption (Could be +/-), and there are no additional contributions to this initial investment. Irrespective of any of that, the point stands that that initial investment would be worth much more than in an outfit that will most likely be worth far lass than the initial investment in 40 years.

The other side of Saving

While I closely monitor unnecessary purchases, I view travel as necessary and not something to wait until later to enjoy. I don’t go out of my way to spend as much money as possible when traveling, but I take most any opportunity to visit a new place. I enjoy maximizing my time and being comfortable while visiting and am willing to spend the $$$ to do so.

I’m also willing to spend money I wouldn’t normally while hanging out with friends or going out to dinner with family.

Investing

I may currently be too risk averse, but my current investing philosophy is as follows:

Buying individual equities/crypto is essentially gambling, doing this with a small amount of money is ~~fine~~ (Less then 2% of total net worth)
I split investment 40/60 between sure return products(CDs and a High Yield Savings account) and indexes(Small Cap Value and VT)… in current state I think this is probably over-weighting CDs and HYS, although it does allow me a lot of liquidity if the stock market goes down
In light of the last statement, I do NOT believe I can time the market, and I am currently acting not in the interest of maximizing returns by withholding so much cash from the stock market
I try to weight my Index holdings away from Large Cap US tech companies as that is currently my main source of income and I’d like to diversify away not increase my exposure

I think Rational Reminder is the basis for most of my opinions and current investment philosophy.

What do I think will change

I’ve never made a large purchase like a home, but it currently makes me nervous concentrating so much of one’s net worth in a single asset.

I think having a large expensive wedding is silly, but maybe I’ll see that as a once (Can only get married for the first time once) in a lifetime opportunity and change my opinions.

If I have kids, I think my financial outlook changes quite dramatically. This is not to view children as an expense, but simply to note that my current thoughts around personal finance don’t take dependents into account whatsoever. Maybe I will become even more risk averse at this point?

A reason for being

My end goal is to have the freedom to spend my limited time on the things I want to do with the people I want to do them with. My thoughts, everything I wrote and didn’t write above is with this single goal in mind. I think this is a broad statement but I don’t think I can make it any more granular.

Its all a weird balancing act of present satisfaction vs future enjoyment… a rubber band of fond memories and regret. I guess I’d like to have fond memories, and leave others with fond memories as well.

Ok so maybe its:

I want to have enough $$$ to do the things that bring me happiness currently
I want to make sure my future self has enough $$$ to do things that bring him happiness
I want the freedom to be able to choose how I spend my time
I want to help others and leave a net-positive reputation with both strangers and loved ones

If I can leverage my finances to do that, I think I’d be happy.

What I learned this week 09-21-2024

Sat, 21 Sep 2024 00:00:00 +0000

What I learned this week

Software

Excel is not a database! (This is not a new learning, but something I’d like to shout loudly)

Business/Finance

Interesting Links

The Man in the Arena:

“It is not the critic who counts; not the man who points out how the strong man stumbles, or where the doer of deeds could have done them better. The credit belongs to the man who is actually in the arena, whose face is marred by dust and sweat and blood; who strives valiantly; who errs, who comes short again and again, because there is no effort without error and shortcoming; but who does actually strive to do the deeds; who knows great enthusiasms, the great devotions; who spends himself in a worthy cause; who at the best knows in the end the triumph of high achievement, and who at the worst, if he fails, at least fails while daring greatly, so that his place shall never be with those cold and timid souls who neither know victory nor defeat.”

Math/Stats

Travel

Went on a trip to Denver. The fall colors in Breckenridge, CO are incredibly beatiful, and I’d love to spend a full week there next fall/summer. The skiing looks excellent too, but quite expensive.

Other

Quite enjoyed Tuesdays wIth Morrie

What I learned this week 09-14-2024

Sat, 14 Sep 2024 00:00:00 +0000

What I learned this week

Software

IEEE 754 on floating point numbers

“If you want to know what’s actually reflected in your data, you may need to profile it. Needles in haystacks aren’t revealed by “select * from haystack limit 100”. You’ll probably just get 100 rows of hay.”

Business/Finance

Walgreens Debt Burden apparently a lot of their fundamental real estate is leased.

Interesting Links

How does Linux move from an awake machine to a hibernating one?

Math/Stats

Travel

Kitt Peak Observatory

Other

“I know that a Tomato is a fruit, and I also know not to put it in a fruit salad”

Interesting take on Student Debt

What I learned this week 09-07-2024

Sat, 07 Sep 2024 00:00:00 +0000

What I learned this week

Software

RAND(1) Does not throw an error in Redshift SQL but RANDOM(1) does. I can’t find any info on RAND

echo "temp/" >> .gitignore untrack a folder

git reset HEAD^ reset to previous commit

Business/Finance

Interesting Links

Tetris SQL

Math/Stats

Cosine Similarity

Travel

Other

“I wish I learned sooner that being the new guy has a lot of perks” – Some guy

What I learned this week 08-31-2024

Sat, 31 Aug 2024 00:00:00 +0000

What I learned this week

Software

echo $? will return the exit status of last command.

I was manually creating a test-RDS instance running Postgres in the AWS console… if you don’t find a small area in the additional configuration section where it says “Specify a Database Name” [This is optional]… it won’t create a database.

Date +%s on my mac to show time since epoch

Been using tmux for long running tasks on my dev desk. I hadn’t realized it spawns a new procss ps aux | grep tmux, makes it really useful.

[Go learning] (https://go.dev/tour/welcome/1)

awscurl is pretty nifty, used it for the first time.

git restore --staged directory/ to unstage all files from a commit in a specific directory

Business/Finance

REI lost ~300MM dollars last year.

Interesting Links

Math/Stats

The Tea Experiment

Travel

Anello - Tucson (Excellent pizza and chocolate tart)

Other places I want to try: Cherry Blossom and El Torero

Other

Had a friend who now lives in Spain stay with us in Tucson, which was quite nice. I was reminded of how great Mt.Lemon is even when most think its too hot to go outside in Tucson.

Sat in on a med school class and listened to a friend’s mom, Dr. for 30 years, give a pep talk to a bunch of first year med students.

Been calling old friends and talking on the phone. They are typically good conversations, most of them over-due, and always welcome. Not sure why I haven’t done this… maybe its just out of mind as you never see this people in person.

Gave a kirkland sports drink to a security guard, they actually make a decent sports drink… I would drink it over gatorade.

What I learned this week 08-24-2024

Sat, 24 Aug 2024 00:00:00 +0000

What I learned this week

I’ve decided to build a habit around recording interesting things I learn/refresh my memory on each week.

I believe this will be a collection of random facts, statements, and quotes. I’ll attempt to organize them into broad categories, and add categories as the week goes on. I have no idea what does/does not make the cut of being included it will mostly be random, at least for now.

Initial strategy is to make a note of concepts as they come up and spend fewer then 60 minutes at the end of the week.

I short reminder to myself: You do not care how this looks, this does not have to be comprehensive, and there should zero mental burden on what is added to this list.

“And now that you don’t have to be perfect, you can be good.” ― John Steinbeck, East of Eden

Software

Smithy is a language for defining services and SDKs. Basically set of tools for defining clients and/or servers for a particular service.

curl --verbose --silent google.com Use curl to inspect HTTP exchange

Mockito is a popular Java mocking framework that is widely used in unit testing. The primary purpose of Mockito is to simplify the process of creating and managing mock objects, which are simulated objects that mimic the behavior of real objects in a controlled way.

If you’re cloning a git repo that uses submodules you’ll need to: git submodule init & git submodule update … spent a bit longer than I’d like to admit figuring that out.

Border Gateway Protocol

Pipe Syntax in SQL Google introduces an extension of SQL using pipes

Business/Finance

Games people play with Cashflows - Interesting article on forming opinions, cash flows, and borrowing.

Interesting Links

Anatomy of a 2AM mental breakdown - Well written piece and good discussion on HN

What’s Really Going On in Machine Learning? Some Minimal Models - Amazing visuals still reading through

Math/Stats

Multi Arm Bandit - Explore vs Exploit and minimize regret.

Sampling with SQL

Variance for a binomial distribution can be calculated as n * p(1-p), thus the variance of a coin flip over five trials is 5 * 0.5 (1 - 0.5) = 1.25 the SQRT of 1.25 = 1.118. We know the mean of 5 trials is 0.5 * 5 = 2.5. With this information we can use a normal distribution to approximate a binomial distribution to get the probability of something occurring… say <=4 out of 5 flips being heads. We get a z-score of (4 - 2.5) / 1.118 = 1.34, and looking up 1.34 in a z table we see the area to the left of 1.34 constitutes ~91% a.k.a there’s approximately a 91% chance that <=4 out of 5 flips are heads.

As noted above we can use a normal distribution to APPROXIMATE a binomial distribution, at lower numbers of trials (like we have here), this approximation rears its head. A simple way to solve this problem would be to find the inverse of the probability that all 5 trials would be heads as that is the only scenario not accounted by <= 4 heads. Thus, 1 - (0.5*0.5*0.5*0.5*0.5) = .9686 which is the true probability of <=4 out of 5 flips are heads.

The difference, ~91% vs ~97%, occurs as we are using a continuous probability distribution to approximate a discrete probability distribution. Continuity Corrections can be employed to reduce this.

We can also calculate the Binomial probability: P(X=x), say of exactly 4 out of 5 flips being heads. P{x} = {n \choose x} * p^{x} * q^{n-x}. Under the hood {n \choose x} is really just the number of combinations n! / (n-x)!*x!. In our case this becomes 5! / (5-4)!*4! = 120/24 = 5 and then our second piece is just .5^4 *.5^1 = .03125 thus we get 5 * .03125 = .15625 the chance of getting exactly 4 heads out of 5 flips.

Travel

Encinitas, CA is very nice. The Comedy Store in LA is great, saw Andrew Santino, Bill Burr, Craig Robinson, David Spade, and Ron White. Be careful riding bikes on the PCH on Sat/Sun during the summer… maybe choose a different route/time.

Other

Generally, it is required that police officers have a GED.

Glacier National Park

Tue, 16 Jul 2024 00:00:00 +0000

Travel/Stay

Spent three days in Glacier National Park this summer, and it was one of my favorite visits.

We flew into Kalispell and stayed in Columbia Falls. It was a great little town to stay in with a short 20 mintue drive to the park entrance.

During peak season with the Going to The Sun Road open, its critical to get into the park early. We’d pack a cooler with snacks, cold water (it gets pretty hot during the day) and head out as early as possible (Trying to park before 6:30am).

There are a few different fees you have to pay. I had paid these so long before our visit that the first day we drove in, I thought I hadn’t paid the West entrance fee only to drive back away from the entrance to get cell service and figure out I had bought it.

Hiking

In the middle of the Summer an early start is definitely the way to go. If you’re doing a longer hike that might take a good chunk of the day, starting early will give you lower temperatures and optimal parking.

We planned on doing smaller hikes but more of them, which gets a bit difficult as parking becomes scarce during the day. Our best day consisted of doing Avalanche Lake early in the morning. If you’re doing the popular trails I would highly recommend starting early.

After this hike, we said screw it and decided to drive up to the Logan Pass visitor’s center (~10am) and fight it out for a parking spot. If we couldn’t get anything, we had some back-up less packed areas we wanted to check out and have lunch at.

After about 15 minutes of jostling in the parking lot, we got a spot. I think the best strategy is to wait at the end of a row, there were a decent number of people leaving and if you kept circling the parking it became difficult to time it correctly and snag a spot.

Establishing dominance in parking lot musical chairs allotted us the reward of doing the Hidden Lake Overlook Trail. The trail still had a bit of snow cover and there were tons of mountain goats. Hidden lake was “closed” due to too much bear activity, but the overlook was excellent (I would bring binoculars next time). It was hot, but not bad as we were at higher elevation.

You could spend a full summer here hiking, many of these trails are worth doing multiple times at different times of day. The glaciers in the park are slowly disappearing, so it was special to get to see them in their current state.

Flathead River

We did a rafting trip on the middle fork of the Flathead river which was fun although the water was quite cold. There as a lot of history and wild life along the river, although we didn’t see any bears.

The river runs next to Columbia Falls. After a long day, Ally and I ambled down about a mile from where we were staying to the river. There was a cute little park next to the river with a decent sized community garden and open space with large trees. We slid down a small sandy embankment and picked up smooth river rocks out of the cool water. They had tons of color and unqiue patterns when wet. It was one of those days you already know is going to be a fond memory for the rest of your life.

Other stuff

You can buy bear spray pretty easily in the park. You can’t fly with it, so you kind of have to find some place around the park to buy it. If you’re driving, you can actually get two cans of the stuff from Costco for like $45, I paid ~$55 for a single one in the park. Since I couldn’t fly with it home, I spent another $10 to ship it. The kid working at the UPS store actually didn’t know if he could ship bear spray, but after calling his mom he said it was fine and took it (lol).

Funny enough this trip actually occured during the CrowdStrike blue screen of death incident. As you might guess, bricking tons of computers running windows makes it hard for airlines to function. It actually worked out in our favor as we originally had some nasty connection from Kalispell -> Chicago -> Phoenix, AZ. We ended finding a direct flight from Kalispell -> Mesa, AZ and got home in a few hours. We checked our previous flights just for fun and with the delays it would’ve been a completely unneccesary 10 hour travel day.

Half Dome

Wed, 10 Jul 2024 00:00:00 +0000

Background

Had the opportunity to hike half dome via the (John Muir Trail) JMT on July 10th, 2024. It was a fantastic hike and I wanted to record the experience and a few ways to approach the hike.

Red = Mist trail
Green = John Muir Trail
Yellow Line = Ascent from Nevada Falls to Half Dome
Blue line = Panorama Trail (Not hiked/referenced below)

We put in for the permit lottery when it opened up. Three of us submitted with dates spanning May – August, and only one of us landed a permit for four people to hike half dome on July 10th, 2024.

I booked a place in Groveland which is about an hour and twenty minute drive from the base of the trail (given no traffic). I’ve stayed in this town before, its pretty nice and has everything you’d need. It’ll depend on the person, but you could certainly stay closer to Yosemite… this will likely require planning far ahead.

We flew into San Jose on the 8th and drove 3 hours to Groveland. We made a pit-stop at REI on the way so that one of the guys hiking could buy hiking shoes lol. On the 9th, we hiked Sentinel dome. It’s a short trail and provides great views of the whole valley, you can also see half dome and pretty much chart out the entire trail leading to the top. We did this on a Tuesday and things were busy but not realllllly busy. It was definitely a good amount of driving from Groveland which, compounded with the previous travel day, meant a lot of sitting. Going to the dome in the afternoon was a good reminder of how hot it can get, especially in the sun, even at higher elevations.

After driving to sentinel dome, we headed back to Groveland to cook dinner and get prepared for the hike the next day.

The plan was to wake up at 3:30 AM the next morning, hit the road by 4:00, park by 5:20 and start hiking by 5:30. There were three of us hiking, I took 3.5 liters of water while the other two took 3 liters and 4 liters of water. I choose to wear trail running shoes (Saucony Peregrine 13), I’ve been using this shoe for a few years, and I like it. I usually pick up last years model on Amazon in some weird color-way for half the price of the new shoe. I had bought new hiking boots a month before this hike, but on a practice hike I got some blisters and ended up going with the trail runners instead as I’d never had issues with them.

I packed roughly 2500 calories of food
I did not bring hiking poles
I wore some breathable hiking pants with long socks
I wore a light/breathable t-shirt
I brought a bag of Kleenex, a portable charger, a head lamp, a water filtration kit
I brought some cheap glovesfor the cables!

I used a 2.5 liter bladder which I filled to the brim. I then used two 500 ml bottles which I stuffed into the sides of my pack. The 2.5 liter bladder was nothing but water; however, in the bottles I added ~1200mg of sodium via sodium citrate, 120 calories worth of white table sugar, and a Nuun tablet for a bit of flavor. I am not a hydration expert and I probably was not perfect with my hydration on this hike, but I subscribe to what Dr. Alex Harrison has said on the topic (a quick google search and you’ll get all the info you need).

Total pack weight was ~11-12lbs, I felt confidant with this set up. I’ll speak about it later. Overall, I was happy with it.

For training, I spent a week hiking and biking at > 8000ft elevation in Greer, AZ directly before the hike. In the months leading up to the hike I averaged 15-25 miles hiking per week, 3-4 hours biking per week, 5-10 miles running, basic full body gym routine 2x per week, and ~12,000 steps per day on average. The longest hike I did before half dome was 10 miles with 1500 feet of elevation gain at high elevation (~9000 feet). In general, I was just keeping a decent level of fitness.

I had been continuously battling a strained hamstring, which I initially strained in March 2024 while doing a dumbbell stiff leg deadlift. I was unable to walk without pain for a several weeks and couldn’t work out legs in the gym. The strain got better, and I was nearly back to normal when I reinjured it again in early May. It felt great the weeks leading in to the hike. I was fairly confidant but was still a little nervous given the several weeks of inactivity and reinjury leading up to the hike.

We watched the first half of the Eiger Sanction (great Clint Eastwood movie), before getting to bed around 9:30 PM. I was a little nervous for the cables and for my hamstring, but overall I felt prepared and figured worst case I can always just turn around.

Hiking

We woke up at 3:30 AM the next morning. My lower back was entirely frozen, it hurt to sit down and I couldn’t really bend over. I remember noticing there were some ants that had wondered into our AirBnB’s kitchen, when I bent down to look at them I immediately seized up. I think it was due to all the sitting and travelling I had done over the previous few days as well as sleeping in a new bed. There was certainly some bad luck thrown into the mix as well. The good thing was that I had no issue when I was standing up, it was only sitting down. I took an Advil and figured if I felt bad on the trail I’d just turn around.

We hit the road by 4:15 AM. There was no traffic that early, and we parked about a half mile from the trail head at 5:30. After getting our packs on, walking to the trail head, and taking a photo we began hiking at 6:00 AM.

The first ~ 5 miles of the JMT to Nevada falls took us about 2.5 hours, it’s a steady climb with many switch backs all the way up. I was able to eat a few hundred calories and steadily kept sipping water. We only stopped for a few minutes at the falls to put on sunscreen before continuing. I took another Advil at this time and a caffeine pill. It was about here where I knew I would be fine, I just couldn’t sit down hahaha, so I’d need to pretty much be standing or walking for the next nine hours.

After the falls, the trail flattens out for a few miles, it gets a little sandy but we chugged through it, everyone still chatting and having a good time. As we began to wrap around the back side of Half dome the final two miles begins to get steep again, luckily it was still cool and shaded. The time at elevation really began to pay dividends here and for the rest of the hike, I had no issues. We stopped a few times to rest at this section before finally getting up to the base of the sub dome.

From the base of the sub-dome to the top of half dome couldn’t be more than a ¾ of a mile, but you gain ~1000 feet of elevation all the way up to 8800 feet. Ascending the sub dome is essentially just a bunch of stairs with a bit of steep scramble to get all the way to the top. Once you crest the sub dome all that’s left is the cables. Below is a photo of the scrambling section of the sub dome.

There were gloves at the base of the cables but I was happy to have brought my own. They gripped the cables well, which was quite important as the rock of half dome is actually quite slick. When I got the top of the sub dome, I really didn’t want to even think and wait, I mainly just wanted to get it done. I shot up the cables quickly, we were going up at 10:00 AM on a Wednesday so it really wasn’t very busy. With both hands on the cables its quite easy to pull yourself up while finding a solid place to put your shoes and step up. There are wooden boards and areas where the rock juts out a bit for resting. There are a few very steep sections, but if you just focus on holding on and resting when needed its not that bad. I did the full cable section in what I think was < 10 minutes, but I didn’t time it nor was I really paying any attention to that.

Once we had all made it up, we walked around to both sides of the dome, took some photos and had a bite to eat. I was still a little nervous about going down, so I didn’t have much of a stomach to eat very much. After 4 hours of hiking to get to the top, I had probably only eaten 600 calories and drank about 2 liters of water. I had a good bit of adrenaline at this point and wasn’t really hungry at all. This was also the point where one of the guys I was hiking with turned to us and said he had no water left… this was bad because we were about 4 miles away from water I could filter and 7 miles away from a true source of drinking water. It was also probably going to take ~4 hours to descend so we’d be hiking down from 11:00 AM to 03:00 PM when it would be hot. This revelation made me even more nervous about getting down so after we took our last photos I started down the cables.

I was nervous for the descent but, again, it wasn’t bad. Two hands on the cables at all times, face the dome, don’t look out if you’re afraid of heights and rest on the boards if you need to. If someone needs to pass or you need to pass someone, coordinate so one person has a good place to rest comfortably and then grip one side of the cables with both hands and go around. Everyone was very nice and courteous, and most people were fine… a few were quite nervous. The biggest thing I could take away from this would be to avoid the crowds on the cables at all costs, it would have sucked if it was just straight lines of people all the way up with slow/nervous people causing problems. I also would never even think about going up if there was any inkling that it would rain or had just rained. The rock is extremely slick from so many people climbing it, if you have good upper body strength it’d probably not be a problem but it’d be a serious additional risk.

With three of us off the cables and on the sub-dome by ~11:30 AM, it became very apparent that it was going to be really hot, and we needed to get back to water before we all ran out with three people sharing the water of two people.

We pretty much had no reason to stop on the way down, about one mile out from Nevada falls the other guy I was with ran out of water and at that point I only had about a half a liter. When we got to Nevada Falls around 1:30 PM, we used my water filter to get a bit of water. It was quite slow and we filtered enough water for everyone to have a few drinks with the understanding we were just going to smash it down to water fountain that was approximately an hour away.

After sharing my water, I eventually ran out about 20 minutes away from drinking water. At this point we were booking it down the switchbacks of the John Muir Trail (We never used the mist trail, we took the JMT both ways). We got to drinking water and filled up our bottles with about 30 minutes of the hike remaining. At this time of day there are MANY poorly equipped day hikers on the lower parts of the trail.

Large groups, people arguing, people hiking in flip flops, people hiking with no water.
They have no concept of hiking etiquette, its hot, and we had to weave through this
unruly crowd with 16 miles in our legs.

We finished the hike around 3:00 PM; my girlfriend picked us up and had cold water and ice cream in the car for us, which was exactly what we needed. Total distance hiked was 19 miles, 5500 feet of elevation gain, a total hike time of 7 hours and 30 minutes, and a total time out on the trail of 9 hours.

Takeaways

I’d make sure to do this hike with people of adequate preparedness and fitness.
I would probably bring 4 to 4.5 liters of water, I probably would have been fine with three, but in reality I really should have been drinking more than I did.
I would try to eat a bit more and take in more sodium. I only ate ~1500 calories and was sweating profusely for the last 3 hours of the hike.
My calves were a bit sore the next day, but otherwise I was fine. Seemed as though the training a did was plenty, as I felt good during the hike as well.
I would consider doing the hike over-night and watching the sunrise on Half Dome, also breaking the hike up and camping (given the proper planning) would be a good idea. I don’t mind the way we did it, but I would like to take every opportunity to limit my exposure to heat and large crowds.
I would bring a water filter again just in case… some people were drinking the water directly from Merced River. I think this is highly stupid and a good way to give yourself the runs. The filter is light weight and will save you from butt pee.
I would freeze some water and keep it in your car or a cooler to have on the way back, something cold was nice.
I might take the mist trail on the way up and then the JMT on the way down. The mist trail offers a nice little mist from the falls, is more direct, and would still be a bit shaded in the morning despite not having a ton of tree cover. The JMT would be easier on the way down especially with tired legs and would offer more shade.
Trail running shoes or hiking boots would both have been fine.
Cables were not a big deal as long as its not busy. If you’re afraid of heights, just face the rock in front of you.
It is CRITICAL to take in salt, sugar, and a lot of water… also, probably try and eat ~200 calories an hour.

It was an excellent trip that I’d definitely do again.

[6 / 4 = 1 = 1.5]: Type Coercion and Precision in SQL

Sat, 06 Jul 2024 00:00:00 +0000

An overview of what’s to come:

I was originally writing this as a high-level introduction to send people when they encounter things like 6/4 returning 1 instead of 1.5 in some Database Management Systems (DBMS)… overtime its sprawled far beyond that. The points below serve as a TLDR, with a sprawling discussion expanding afterward.

Operations like division in SQL can be confusing for novices or anyone without a programming background
In some DBMS, 6/4 may equal 1 because, dividing two integers a.k.a. integer division returns an integer (the remainder is discarded)
```
 Casting one of the operands as a float or numeric/decimal will ensure floating point
 division. Be careful with the precision needed in your calculation as there are
 tradeoffs to using certain types.
```
Syntax and behavior will be different for various DBMS, but if you want 6 / 4 to equal 1.5 and not 1 then you can use something like:

PostgreSQL:
   SELECT 6::FLOAT / 4 AS result -- Type casting using "::" operator
   SELECT 6 / 4::FLOAT AS result -- Casting the divisor
   SELECT CAST(6 AS FLOAT) / 4 AS result -- Using the CAST function

MySQL:
   SELECT 6 / 4 AS result -- MySQL automatically performs type conversion
   SELECT 6 / CAST(4 AS DECIMAL(10,2)) AS result -- Explicit CAST to DECIMAL
   SELECT 6 / CONVERT(4, DECIMAL(10,2)) AS result -- Using the CONVERT function

SQLServer:
   SELECT 6.0 / 4 AS result -- Implicit conversion using decimal point
   SELECT CAST(6 AS FLOAT) / 4 AS result -- Explicit CAST to FLOAT
   SELECT CONVERT(FLOAT, 6) / 4 AS result -- Using the CONVERT function

Oracle:
   SELECT 6 / 4 AS result FROM DUAL -- Automatic type conversion
   SELECT 6 / TO_NUMBER(4) AS result FROM DUAL -- Using the TO_NUMBER function
   SELECT 6 / 4d AS result FROM DUAL -- Explicit type declaration

SQLite:
   SELECT CAST(6 AS REAL) / 4 AS result -- CAST to REAL
   SELECT 6.0 / 4 AS result -- Implicit conversion using decimal point
   SELECT ROUND(6.0 / 4, 2) AS result -- Throwing ROUND in there

-- Methods may have trade-offs in terms of precision, performance, and portability.
-- The choice depends on your specific requirements and the DBMS you're using.

This is NOT an exhaustive list of solutions/syntax for all DBMS as there are many DBMS

The idea here is that if you desire a specific output, explicitly casting an operand(s) is the safe bet. If you are the data producer, understanding the consumption patterns and applying the correct type will be critical for your consumers downstream
It’s crucial to understand the database management system (DBMS) and environment you’re working in before curating datasets, creating models, or exporting data for reporting

I think that’s a fair synopsis of what I’ve written below. If you’re interested, I’ll continue to cover division across a variety of DBMS, the behavior of data types/operations in SQL, Hotdogs, Docker, rug pulls and BIG DOGS!.

Observations about SQL Writers

Division is a frequent source of logical and syntactical mistakes for novice SQL users. SQL has some round edges for non-technical users like 1 based indexing. 1 based indexing is the SQL standard, and its more straight forward for people who would typically begin counting at 1 and have no familiarity with 0 based indexing found in languages like Python, Java, or C.

Division is more of a sharp edge. Not because it’s difficult to do division, but when you take someone who has always done =6/4 in Excel and received a result of 1.5 and now you show them that it evaluates to 1 when they write SQL against a database… they don’t anticipate that, nor would they have thought to check for it!

Some Typing and Coercion Background

Many users of SQL coming from a “business” background will have encountered different data types in Excel, but they will likely not have given any thought to Static vs Dynamically typed programming languages. I’m now going to ~~play with fire~~ lay out some definitions to provide some context to the previous sentence that are debatable, but I believe they provide a good mental model for SQL.

SQL is typically considered to be Statically typed, meaning:

Every data item has an associated data type defined at compile-time

SQL does have predefined data types for columns in table definitions, which are determined at “compile-time” (more accurately, at table creation time). However, there are dynamic aspects… SQL allows for implicit type conversions in many operations, and the actual type checking often occurs at runtime.

SQL is also most properly defined as Strongly Typed:

Every data item has an associated data type, defining its behavior and allowed usage

SQL uses a mix of static and dynamic typing. Data types are associated with column definitions at table creation time, but many type-related decisions and checks occur at runtime. SQL’s type system allows for implicit conversions and exhibits some characteristics of both strong and weak typing, with the specific behavior often depending on the particular SQL implementation.

Great, so when I think of SQL I’ll think Strong and Static, thanks! Well… no, like I mentioned, this behavior depends on the DBMS. SQLite uses a more general dynamic type system where the datatype of a value is associated with the value itself, not with the column it sits in. This behavior can be modified with the “STRICT” table option keyword.

An in-depth image for visual learners (I’m an aspiring artist):

Ok… that’s cool, but in SQL what happens when I divide an integer by a float or what if I evaluate an integer and a string for equality? In a more general sense we are asking: How do DBMS handle type coercion?

Type Coercion: is an implicit process where the language runtime automatically converts a value from one type to another. This is done to make operations between different data types possible without explicit instructions from the programmer.
Type Conversion: is an explicit process where the programmer manually converts a value from one type to another using specific functions or methods. This requires a deliberate action and is often clearer in terms of code readability and intent.

You’ll see that DBMS have extensive facilities for evaluating mixed-type expressions. I’m adding the Postgres docs here because they are good, but there are other docs for other DBMS.

Why should I care, and how about a few examples?

To make this implicit conversion explicitly clear, lets directly compare two values that are explicitly cast as different types:

SELECT CAST(5 AS INT) = CAST('5' AS VARCHAR(10))

Postgres: ERROR: operator does not exist: integer = character
SQLite: 1
Redshift: true

Some programming languages are quite strict when doing comparisons or operations on multiple types. Here is an example in Rust where someone has attempted to add an integer and a float and received an error.

fn main() {
    let float = 5.0;
    let integer = 4;
    let result = float + integer;

    println!("Result: {}", result);
}
//error[E0277]: cannot add an integer to a float
//     let result = float + integer;
//                          ^ no implementation for `{float} + {integer}`

Executing SELECT 5.0 + 4 or even SELECT '5' + 4 in most (maybe all?) DBMS returns 9.
So then what about SELECT CAST(5 AS INT) + CAST('4' AS VARCHAR(10))? This still returns 9 in some DBMS but NOT in Postgres:

In Postgres, an undecorated string literal (e.g., ‘4’) without an explicit type is treated as an unknown type. This is actually one of a few Pseudo-Types that allows Postgres to delay type resolution until more context is available. Here is a nice response from Tom Lane posted 13 years ago.

Using pg_typeof we can see that SELECT pg_typeof(CAST('4' AS VARCHAR(10))); returns character varying while SELECT pg_typeof('4'); returns unknown. Due to the specific casting of ‘4’, the second example errors (integer + character varying) while SELECT '5' + 4 evaluates an unknown + integer which Postgres implicitly coerces to an integer result of 9.

However, this is where you need to be careful. For instance, Redshift is a fork of Postgres, but… guess what: SELECT CAST(5 AS INT) + CAST('4' AS VARCHAR(10)) returns ‘54’. A few other interesting anecdotes from Redshift:

----------------------
-- Updates
----------------------
CREATE TEMP TABLE SOME_TUPLES (
    MY_NUMBER INT,
    MY_WORD VARCHAR(100)
);

INSERT INTO SOME_TUPLES(MY_NUMBER, MY_WORD) VALUES
(4, 'Hello'),
(6, 'Howdy');

-- Redshift has no problem updating the VARCHAR(100) MY_WORD column with a value I explicitly cast as an INT
UPDATE SOME_TUPLES SET MY_WORD = 4::INT
WHERE MY_NUMBER = 4;

-- How about a decimal instead? No problem
UPDATE SOME_TUPLES SET MY_WORD = 4::DECIMAL(38,0)
WHERE MY_NUMBER = 4;

----------------------
-- What about AVG()?
-- I see a lot of people use AVG in SQL that actually don't want the arithmetic mean
----------------------
WITH CHECK123 AS (
    SELECT 6 AS NUM
    UNION ALL
    SELECT 4 AS NUM
    UNION ALL
    SELECT 1 AS NUM
    )
SELECT AVG(NUM) FROM CHECK123 -- Results in 3

----------------------
-- Oh, Also
----------------------
-- Returns 2 in Postgres/Redshift
-- Returns 1 in SQL Server
SELECT CAST(1.99 AS INTEGER)

Ooooook so can we go back to Division now?

In some DBMS, SELECT 6 / 4; would result in 1 due to integer division, other DBMS will still return 1.5 even after being passed two “integers” to operate on.

Integer division is the division of two integers resulting in another integer. It discards the fractional part of the division and returns only the whole number quotient
In Python print(6/4) returns 1.5 (round edge); however, print(6//4) would return 1, discarding the remainder. We’ll see later on that DuckDB has similar syntax!

However, SELECT 6.0 / 4; or SELECT CAST(6 AS FLOAT) / 4; would result in 1.5 as the DBMS performs implicit type coercion and returns the result to the client as a float.

Floating-point division is an operation involving at least one floating-point number that results in a floating-point (decimal) number. Floating-point division is used when precision is important, and the fractional part of the result is significant.

Different DBMS have different defaults for this coercion behavior. Sometimes its: “Hey Big Dog looks like you gave me two integers two divide, I’m going to assume that the remainder is important to you and not return an integer” OR “Big DOG! You just served me up two integers and guess what… you are getting an integer back!”

SQL Standard is for the DBMS to refer to their users as Big Dog, just like Tiger.

I’m only talking about type coercion, but keep in mind that there are A LOT of DBMS and they do A LOT of things differently.

    Quick side note, in DuckDB there are two division operators: / and //.
    They are equivalent when at least one of the operands is a FLOAT or a DOUBLE.
    When both operands are integers, / performs floating point division (5 / 2 = 2.5)
    while // performs integer division (5 // 2 = 2).

On top of this, admin can make configuration changes to the DBMS you’re using, possibly making its behavior different than what I describe here! Developing a sound understanding of the environment you plan to write queries in before you or some text to SQL engine starts spinning off queries is important!

If you’d like to compare for yourself, SQL Fiddle is a nice online SQL compiler. You can also use docker. With docker installed, you can pull the official MySQL image and try for yourself:

# Start a new MySQL container named 'testsql' with the root password set
# The -d flag runs the container in detached mode (in the background)
$ docker run --name testsql -e MYSQL_ROOT_PASSWORD=ilovemysql -d mysql:latest

# List all running Docker containers
# This should show the 'testsql' container we just started
$ docker ps

# Start an interactive bash shell inside the running 'testsql' container
# The -it flags allow for an interactive terminal session
$ docker exec -it testsql bash

# Inside the container, start the MySQL client
# The -u flag specifies the user (root in this case)
# The -p flag prompts for the password we set earlier
$ mysql -u root -p

# Now we're in the MySQL prompt. We can run SQL commands here, for example:
SHOW DATABASES;
SELECT 6/4;  # This will return 1.5000 in MySQL

# To exit the MySQL client
EXIT;

# To exit the container's bash shell
$ exit

# Stop the 'testsql' container
$ docker stop testsql

# List running containers again to verify 'testsql' has stopped
$ docker ps

# List all containers, including stopped ones
# This will still show 'testsql', but in a stopped state
$ docker ps -a

In Practice

In my day job, 99% of the time people want floating point division, and I will advise them to explicitly cast one of the operands. For example in Redshift I would write: SELECT 6/4::FLOAT.

A good question might be: “Why bother casting if one or both of my columns is already a Decimal/Float, isn’t it irrelevant?”

In current state you would be 100% correct, but are you 100% sure nobody will alter the table and change that one column that was a Decimal to an INT…

Hey remember that old column we created as a Decimal type, we only write integer values between +/- 100000. We can save some space by storing it as an integer… nobody will have any problem with that! Sounds good: ALTER TABLE TBLONE ALTER COLUMN COLUMNONE TYPE INTEGER USING (NULLIF(COLUMNONE, ‘’)::INTEGER);
Hey that table we produce yeah we are actually deleting that column you use, but its ok because you can recalculate it using a JOIN on this other table to bring in another column… make sure it doesn’t break your transform… thanks! (We sent the email about the migration 3 months ago but must have missed you.. haha sorry lol… well we are making the change on Friday, thanks so much!)

I typically prefer writing defensive SQL and checking what I’m being defensive against rather than having the data producer pull the rug on me. Which is why I prefer to be the data producer, and why I prefer not to buy Meme Coins.

I’ll leave a final furthering thought for anyone interested. I’ve gone ahead and stolen this little exercise from the MySQL docs and re-written it for Redshift (should work identically in Postgres as well)

-- Three variables: i (an integer), d (a decimal with 10 digits total and 4 digits after the decimal point), and f (a floating point).
-- The WHILE loop runs 10,000 times, incrementing both d and f by 0.0001
-- After the loop, RAISE INFO prints the values of d and f.
CREATE OR REPLACE PROCEDURE test_precision()
LANGUAGE plpgsql
AS $$
DECLARE
    i INT := 0;
    d DECIMAL(10,4) := 0;
    f FLOAT := 0;
BEGIN
    WHILE i < 10000 LOOP
        d := d + 0.0001;
        f := f + 0.0001;
        i := i + 1;
    END LOOP;

    RAISE INFO 'DECIMAL result: %, FLOAT result: %', d, f;
END;
$$;

-- Call the stored procedure
CALL test_precision();

Here’s a hint if you’re curious about the output, or you can just smash this stored procedure into ChatGPT and it’ll rattle off a coherent answer the choice is yours.

NY Times Connections Solver PT. 1

Sun, 16 Jun 2024 00:00:00 +0000

Olive Oil:

During my daily morning ritual of waking up at 4:30 AM and immediately ingesting a ½ cup of [Trader Joes Extra Virgin Spanish Olive Oil] (Sponsored link), I decided I’d like to try and solve the NY Time’s Connections. So, with greasy fingers, lets begin.

This is part one, where I:

Review the rules of Connections
Gather historical Connections games into a structured dataset
Come up with some strategy for a program that’s somewhat capable of playing the game well

In part two, I’ll try and generate a solver for the game. I am not an expert on NLP, nor do I know much about word embeddings, nor do I know much about word embeddings given context, but I’m quite curious to learn. Here in part one, I’m going to nail out something semi-interesting and treat part two as a learning experience.

Overview/Definitions:

Connections is one of many NY Times games. It began in June 2023 and releases a puzzle daily. Outside of drinking straight oil, it’s one of my favorite things to do in the morning… so why not ruin it by figuring out a way to convert all historical games into a table and write SQL against it!

Connections is quite simple. You’re provided a 4x4 grid of word tiles. Words are grouped into sets of four based on a connecting category of varying difficulty. I’ll write about some patterns I discovered in the categories later.

Above is a picture containing the Connections game from 6/05/2024. Funny enough, this particular game so-happens to contain the most common word used across the history of connections which is Ring used, now, 11 times.

In this example, the four words “Sick”, “Nice”, “Cool”, and “Sweet” can be used to form the “Awesome!” category. This category was designated the “easiest” of all categories, although this definition differs from person to person as not everyone would associate Sick as a synonym for “Awesome!”. Personally, I associate sick with what happens after I drink my ½ cup of oil which, in my opinion, is pretty awesome… so, for me, it makes sense, but you do you. Rinse and repeat with the remaining 12 words, and that’s Connections.

Defining a few things to stay on the same page moving forward:

Category: A grouping of 4 words in a game
Words: A single tile in a game
Difficulty: How “Straightforward (easy)” or “Tricky (hard)” a category was in a game.

How does ChatGPT-4o Do?

I’ll start with a single days example from June 8th , 2024. This example aligns with a status quo Connections game, you’ll see why I say this in the analysis section.

**SPOILER ALERT** Below is the solution to the puzzle with the difficulty, category and words.

[Easiest] Grab One’s Attention: ABSORB, ENGROSS, HOLD, RIVET
[Easy] Document with Video: FILM, RECORD, SHOOT, TAPE
[Medium] ANAGRAMS: PASTEL, PETALS, PLEATS, STAPLE
[Hard] PRETTY ___: GOOD, PENNY, PLEASE, WOMAN

I made sure that the model is familiar with the game; however, at this point, I don’t offer it any previous connections data or answers to go off of.

Prompt:

"Solve today’s game, providing the category defining each word grouping.
Words: Pleats, Please, Hold, Record, Woman, Film, Good, Pastel, Penny,
Staple, Tape, Rivet, Absorb, Petals, Engross, Shoot."

Response:

Words related to movies: Shoot, Film, Tape, Record [CORRECT]
Words related to types of fasteners: Rivet, Staple, Pleats, Hold [WRONG], but I also considered this possibility
Words related to artistic or aesthetic qualities: Pastel, Penny, Woman, Good [WRONG] but close
Words related to engagement or involvement: Absorb, Please, Petals, Engross [WRONG]

When provided limited context or historical data, the LLM typically gets the two easier categories right (some cluster of closely related words). However, without any additional context, the model tends to miss the harder categories sometimes even letting mistakes spill back into the easier ones causing additional misses. This isn’t news to anybody, but illustrates that better prompts get better responses… if you’re going to go Oscar the Grouch mode (his birthday was June 1 BTW) expect a garbage in -> garbage out scenario.

Let’s provide an additional prompt denoting that the possibility of certain categories exists.

Prompt:

“These are incorrect, take into account that answers can involve grouping words
 that have similar meaning with the addition of a given prefix or suffix, homophones,
 and anagrams can all be at play to create a category. Please try again.”

Response:

Recording-related terms: Record, Tape, Film, Shoot [CORRECT]
Anagrams of ‘Staple’: Pleats, Pastel, Staple, Petals [CORRECT]
Words that can follow ‘Please’: Please, Hold, Absorb, Engross [WRONG]. The words almost form the easiest category, but it forces the prefix concept unnecessarily.
Terms associated with monetary or value: Penny, Good, Woman, Rivet (stretch, but possibly in terms of securing value) [WRONG] Switching “Rivet” and “Please” would yield a correct puzzle, but the categories are off significantly. It evens notes “Rivet” is a stretch.

While this attempt recognizes the anagram, the “Pretty ___ “ category seems to be fairly elusive for it and there is no idea of “Attention” in any category.

Final time… I provided a month’s worth of historical connections games with the categories, their associated difficulty and the four words. This, initially, caused it to regress significantly, providing an output that no longer followed the rules of the game. After re-orienting back to the task on hand, it butchered its previous progress and… umm… gave a response that added “Woman” to a category called “Objects”…

After providing thirty Connections puzzles, I found it was possible for the model to get them correct but only on easier days and after several prompts. If the model receives an extra hint, such as “I’m sure these three words together are in category “X”, but I don’t know the fourth word,” it usually manages to provide a correct answer.

There are probably better prompts, more prompts, or a curated LLM that could accurately solve most of these puzzles. There is certainly a good amount to explore in prompting an LLM and trying to get it to solve Connections.

Retrieving historical Connections games:

I figured a good place to start would be to create a table containing all historical connections games properly labeled with words, categories, difficulty, and date. This dataset would serve as the basis for understanding common patterns in Connections, as well as a clean set of labeled training data I could use for whatever solution I envisioned.

While the NY Times has a Wordle Archive, there is no such Connections archive. Luckily, there are a myriad of websites that host all historical connections games.

So, I LOCKED IN and spent ~5 days copying…

Nah… actually, I sent a couple cold emails to any address I could find on these sites asking if they would share a file containing the games and answers. No response. I did a quick search of Github and Reddit to see if I could uncover any leads, but no… ok sooooo web-scraping, Yay!

I wrote a quick script to scrape one of the historical Connections sites into a text file. It quickly loops through each month of connections data on the site, extracts the paragraphs with the games and writes them to a text file with a separator. I’m going to leave the URL out, but you can find the site quite easily if you look. After writing 335 connections games to the text file, I convert it to a csv and remove duplicates. Now, nobody needs to scrape. They can just steal from me.

This is probably a good time to point out that this whole process should be a DAG, but I dont care about robustness or repeatability right now so… shortcuts!

Analytics (Cleaning):

Cool, so now that we have our “Clean” CSV, we can copy it directly into a table using duckDB and explore all these historical games. With duckDB you can directly COPY a CSV from disk into a table in a database. Ten lines of code later, and we’ve got a table.

# duckdb.connect(dbname) creates a connection to a persistent database
conn = duckdb.connect("connections.db")

# Create DuckDB table
conn.sql(
"""
DROP TABLE IF EXISTS CONNECTIONS_DATA;
CREATE TABLE CONNECTIONS_DATA (
    GAME_DATE DATE,
    DIFFICULTY VARCHAR,
    CATEGORY VARCHAR,
    WORD VARCHAR
);
"""
)

# COPY CSV from disk
conn.sql(
f"""COPY CONNECTIONS_DATA FROM '{clean_csv_location}' (FORMAT CSV, HEADER);"""

In our new table:

Each game consists of 16 rows, one for each word
A category, on a particular day, can be described with four rows

I choose this because I didn’t want to deal with any unnesting logic for viewing individual words, which would be necessary if I collapsed a game into 4 rows and saved the words in an ARRAY/LIST or just LISTAGG’ed the strings together.

FYI, duckDB adopted the SQL standard of 1 based indexing a couple years ago.

You may have noticed I put “CLEAN” in quotes earlier. I’ve got some half-baked web-scraping and transform logic I cheffed up in ~ 1 hour, so I figured there was about a 1% chance I missed some stuff… and it turns out I did indeed miss some stuff.

Date-category combinations should have four rows. A count of rows (CNT) divided by four should return 1, and then, 1 mod 1 should equal zero. However, the below query returned six instances where CNT equaled 1.25. I discovered hyphenated words like “Yo-Yo” were written as two rows.

SELECT GAME_DATE
     , CATEGORY
     , COUNT(1) / 4::FLOAT AS CNT -- BECASUE THERE WILL BE 4 ROWS FOR EACH INSTANCE
FROM CONNECTIONS_DATA
GROUP BY 1,2
HAVING CNT % 1 <> 0
ORDER BY CNT DESC

Alright, probably good but let’s run some summary stats to be sure:

SELECT COUNT(1) AS TOTAL_WORDS
     , COUNT(1) / 4::FLOAT AS TOTAL_DISTINCT_CONNECTIONS
     , COUNT(1) / 16::FLOAT AS TOTAL_DISTINCT_DAYS
FROM CONNECTIONS_DATA

Oops, why is TOTAL_DISTINCT_DAYS 334.5?

I assumed every game was unique
There was some boiler plate text in the text file from the web scrape that was duplicated
So… why not remove rows in the text file that were duplicates of a previously seen row to clean it up?
Wrong! Connections reuses categories with identical words:
- August 30th, 2023, and May 30th, 2024: “Influence” category.
- January 11th, 2024, July 25th, 2023, and August 15th, 2023: “States of matter” category.

Instead of fixing this in the extract code, I just stripped them out of the table and called the clean_table funtion at the end of pipeline.py. Sweet, clean CSV, no quotes this time.

I also removed 4/1/2024, April Fools day, when all words were Emojis ¯\_(ツ)_/¯

Analytics (insights):

I hacked together a few queries in a Jupyter notebook style file, printing data frames to the console. Over the +/- 335 games in this dataset there were:

~3500 distinct words, not many words have spaces (fewer than 30 occurences), and most words are 4 to 5 letters.
Word length is consistent across difficulties on average 5.1 letters
Most commonly reused words: {RING:11, BALL:10, COPY:9, LEAD:9, WING:8, JACK:8, CUT:8, HEART:8}
Shortest words are individial letters like “X”, occuring only twelve times
Longest words include Peppermint Patty, Concentration, Mashed Potato, and Horsefeathers. Horsefeathers is synonymous with nonsense or rubbish (I could see myself yelling this in an argument)

Pivoting out difficulty by category type:

Unpacking a few things:

This is biased towards the “Hard” categories, describing ~70% of them. This is due to this category being pattern focused and based on challenging word play each day. The only way I’m going to be able to solve these is to reduce the degrees of freedom in selection or determine the pattern with high confidence.

Dunderscores are extremely common, describing ~40% of hard categories:
- ROCK ___: (Star, Candy, Bottom, Garden)
- ___ MOON: (Sailor, New, Blue, Harvest)

Homophones show up a few times, and I should probably just do a static check
for these against a database of English language homophones.

Anagrams should be checked similarly.

Prefix_String, Suffix_String, Minus, Added_Plus all denote occasions where the words are similar due to having some character(s) appended or removed.

WITH categories are a significant challenge. I’m not sure how to approach them yet.

Things With Trunks: (Cars, Elephants, Swimmers, Trees)
Tarot Carods With “THE”: (Fool, Lovers, Magician, Tower)

THINGS overlaps WITH 11 times in the data, so I could have included the intersection of the two: THINGS WITH ___. Instead, I’ve just pulled the intersections into the WITH category.
```
Things People Shake: (Hands, Maraca, Polaroid, Snowglobe)
Things to crack: (Egg, Knuckle, Smile, Window)
```
Medium is typically some niche knowledge like countries with red and white flags, weapons in the game Clue, or British cuisine. To some, this is akin to “Easy” if they are familiar with the terms. For the less familiar, these can be confounding and lead to misses.
Easiest/Easy difficulties are generally tightly related words without requiring a layer of context. However, they aren’t always so straightforward.
```
Connect: (Couple, Link, Tie, Join)
Attach with adhesive: (Adhere, Glue, Paste, Stick)
```

Filtering for the triple dunderscore categories was a bit tricky, with _ being reserved. In Redshift, I was able to do LIKE ‘%\\\\\_\_\_%’ , but this doesn’t seem to work in duckDB. Instead, I went with some length based logic checking for at least a difference of 2 characters once stripping them out:

LENGTH(CATEGORY) - LENGTH(REPLACE(CATEGORY, '\_', '')) >= 2

Final thoughts on a solution:

One advantage of this problem is that its surface area decreases exponentially with the reduction in the size of the initial set. Something I never explicitly provided Chat-GPT was information regarding which categories it had gotten right and what was wrong. Allowing it to build on top of past decisions, and reducing the size of the problem.

For combinations, the formula: C(n,r)=n!/r!(n-r)!

Evaluates the number of ways a sample of “r” elements can be obtained from a larger set of “n” distinguishable objects where order does not matter (unlike permutations) and repetitions are not allowed.

At the beginning of any Connections game, a player will choose 4 words from a set of 16 yielding 1820 possible combinations.

C(16,4)=16!/4!(16-4) = 1820

As categories are found and labeled as correct the set is reduced: 16->12->8->4

C(12,4)=12!/4!(12-4) = 495
C(8,4)=8!/4!(8-4) = 70
C(4,4)=4!/4!(4-4) = 1

Like I mentioned in the intro, I find NLP interesting, but I am by no means an expert. After my manual checks to catch some of the tricky categories in Connections, I’m not entirely sure what an optimal framework looks like for grouping four words and iterating over a solution:

Maybe I don’t even need to perform these manual checks if I provide the right context to the correct algorithm?
I’ve noted “confidence” several times in the diagram, but I’m not sure how to objectively arrive at this or if I’ll need to normalize “confidence” across different categories/difficulties?
I’ll want to operate on a reduced set of words given that a category has correctly been identified, but I’m not sure if I should throw out previous work to avoid any noise created by the larger set of words?
I’ll need to figure out how to handle Nouns, single letters, and words with spaces (Maybe this requires injecting a word classification layer between the raw set of words and the cached check for specific categories)
Not sure I’ll be able to get this category: Reads the Same Rotated 180 degrees: 96, MOW, NOON, SIS

Anyway, that’s part one. Who knows what part two will be, stay greasy!

Temporary Tables

Sun, 05 May 2024 00:00:00 +0000

The database management system (DBMS) I’ll reference today is Redshift (AWS Cloud Datawarehouse offering) which is based on Postgres. I’ll dive into some of the anti-patterns around Temporary Tables I’ve seen abused during my work experience, why they don’t make sense, and how to fix them.

Lets get it out of the way… Temporary Tables are not “better” than Common Table Expressions (CTEs) and CTEs are not “better” than temporary tables. It is entirely dependent on the context, the data, the DBMS, the query, upstream tables, etc… because at the end of the day the stakeholder wants the 1MM row output in excel so they can make a pivot table. They don’t care if it was procured via CTEs, TEMP Tables, or that you put another hole in the home-office dry wall.

A few other notes:

In Redshift (and many other DBMS for that matter) there are plenty of excellent use cases for Temp Tables (I’ll provide a few at the end)
Be wary of Premature Optimization. If analytics workloads are running in an acceptable amount of time and not incurring unacceptable expenses, then effort is probably better spent elsewhere

Read, Write, Read, Write:

Internally, most analytics queries we run scan TBs of data (Redshift Spectrum costs $5 per TB of data scanned from S3, but hey… screw it SELECT * it’s a business expense, Right!, Right?). Most commonly, large Transform or Extract jobs are a series of Temporary tables, there is nothing ~~necessarily~~ wrong with this. The issues come about when large chunks of unfiltered data are passed through this series of Temporary tables, being read and written over and over again.

SELECT statements encased in a CREATE TEMP TABLE AS will read data from the upstream source and write it to disk.
In a particular session these tables can be seen in STV_TBL_PERM, they will be collected after the session is closed (obviously permanent tables will not).
Repeatedly reading and writing data too and from temp tables throughout the course of a query can be costly both monetarily and temporally.

Issue 1: Create a temp table just to SELECT everything out of it

Uselessly writing the entire result set to disk just to read it all back out. Just run the final SELECT, there is no need for the temp table. This also presents an additional opportunity for a statement to get stuck in the que behind other workloads.

------------------------------------------
-- NOT GREAT, I SEE THIS WAY TOO OFTEN
------------------------------------------
CREATE TEMP TABLE FINAL AS (
SELECT EVERYTHING
FROM SOME_SCHEMA.BIG_TABLE
);

SELECT *
FROM FINAL

------------------
-- BETTER
------------------
SELECT EVERYTHING
FROM SOME_SCHEMA.BIG_TABLE

Issue 2: Unnecessary data and losing optimizations

Redshift is a columnar database. Each column is stored separately, meaning if you don’t need a column Redshift can ignore scanning the data entirely. Yes, if you're using Spectrum for your OLAP queries and your data is sitting in S3 in abunch of .csv, Redshift will have to scan the whole file... but its probably time for Parquet? In the below SQL:

Redshift is forced to read and store columns BT.COL4, BT.COL5, BT.COL6 , BT.COL7 in TEMP_1 even though they are never referenced again in the rest of the query.
The query optimizer can’t parse further down in the query that all that was wanted were some fairly specific predicates; it could have skipped many blocks of data. Each of the select statements is on its own island in a way. Redshift stores data in 1MB blocks, with additional metadata about these blocks. This meta data (a.k.a zone maps) is what allows the DBMS to avoid scanning irrelevant blocks.

CREATE TEMP TABLE TEMP_1 AS (
SELECT BT.COL1
            , BT.COL2
            , BT.COL3
            , BT.COL4
            , BT.COL5
            , BT.COL6
            , BT.COL7
FROM THIS_IS_A.BIG_TABLE BT
);

-----------------------------------------------------------------------------
-- Now that we’ve read everything and written to disk lets read it back out
-----------------------------------------------------------------------------
CREATE TEMP TABLE TEMP_2 AS (
SELECT BT.COL1
            , BT.COL2
            , BT.COL3
            , ST.COL9
FROM THIS_IS_A.BIG_TABLE BT
	INNER JOIN TINY.SMALL_TABLE ST ON BT.COL1 = ST.COL1 AND BT.COL2 = ST.COL2
);

------------------------------------------------------------------------------------------
-- FINAL SELECT, lets read everything back out, after creating TEMP_2 for… no reason?
------------------------------------------------------------------------------------------
SELECT *
FROM TEMP_2
WHERE COL1 = ‘I Should Have’
    AND COL2 = ‘Applied these predicates’
    AND COL3 = ‘Earlier

SQL is a declarative language, you indicate to the DBMS what you want and the DBMS goes off and determines the best way to retrieve that for you. The series of TEMP TABLES above augments this process and confines the DBMS to only performing rudimentary optimizations without broader context.

These optimizations can be implemented manually by the author of the query, but would otherwise be done automatically by the DBMS. Its like paying your chauffer a bunch of money to drive you to the airport, but, instead, you wrestle him for the keys, tie him up in the back seat of the car, turn on Maroon 5 (You love Maroon 5), and drive to the airport yourself? Why do that?

Issue 3: Sort Keys, Multiple Sort Keys???

Something that seems to be fairly common is defining a SORT KEY within the CTAS syntax when the column used in the sort key should already have been filtered on in the first place during the creation of the Temp Table. This is essentially identical to my point above about filtering after the Temp Table Creation. Defining multiple sort keys at this point just makes things worse.

The kicker is that Redshift will attempt to determine the optimal SORY KEY and DIST KEY for any Temp Table based on the query plan. Manually specifying it in the temp table only to filter it out later is just a bit painful to see and flies in the face of aggregate early and often. More info can be found in the advanced table design playbook. This is probably a good time to mention that its important to make sure table statistics are up to date.

------------------------------------------------------------------------------------
-- THE ADDITIONAL SORTING MAKES THE WRITE MORE EXPENSIVE WITH NO DOWNSTREAM BENEFIT
-- WE'VE DISTRIBUTED THE DATA ACROSS THE NODES IN THE CLUSTER USING COL4
------------------------------------------------------------------------------------
CREATE TEMP TABLE TEMP_1 SORTKEY(COL1, COL2, COL3) DISTKEY(COL4) AS (
SELECT BT.COL1
     , BT.COL2
     , BT.COL3
     , BT.COL4
     , BT.COL5
     , BT.COL6
     , BT.COL7
     , …
FROM THIS_IS_A.BIG_TABLE BT
);

SELECT BT.*
FROM TEMP_1_SORT BTA
        INNER JOIN ANOTHER_TABLE ATA ON BTA.COL4 = ATA.COL4 -- AT LEAST WE COLLOCATED THE JOIN COL...
WHERE BTA.COL1 = ‘Yikes’ -- SHOULD HAVE FILTERED ABOVE
;

In regard to distributing on COL4 someone may think:

“I want to save data in a Temp Table and specify a distribution key, so that I can improve my JOIN later on”

There is certainly some thought going into this argument, but there is no free lunch.

The redistributed data may improve the later join by collocating data across the cluster’s nodes, however, all that shuffling across different nodes still had to occur when the temp table was created!
While this is a valid argument, it would be important to benchmark the run times to actually determine what is most optimal here.
If the current query patterns don’t utilize the table’s partitions (sort keys) and rarely join on the column its distributed on… it might be time to think about adjusting the upstream table rather than having all subsequent queries cache the table in a temp table just to sort and redistribute it before further processing.

When are temp tables a good choice?

If the temporary table is going to be read from numerous times, it may be best to filter once and read several times from the aggregated data. This was especially handy when Redshift used to lack ROLLUP, CUBE, and GROUPING SETS
Using Temp Tables to chunk up pieces of a query can be very handy in development, as they are persisted for the duration of the session, things like tests, logical checks, and quick aggregations can be run against them
Using a series of CTEs within a temp table can be a happy medium, doing more work and writing out less often
Honestly, I’d take 100 temp tables over having to parse through some nested sub query mess

SELECT FROM(
    SELECT FROM(
        SELECT FROM(
            SELECT
            FROM
            WHERE (COL1 IN (SELECT FROM))

        )
    )
)

Blindness and a Chess Puzzle

Thu, 25 Jan 2024 00:00:00 +0000

A Chess Puzzle

I recently encountered a chess puzzle that revolved around capturing my opponents material due to a forced mate in one threat. While exploring various permutations of moves, I struggled to choose the right piece (between the bishop and the knight) to simultaneously block my opponent’s mate in one threat while allowing me to freely capture a few of his minor pieces. After a brief though I played Knight C3 -> D5, blocking my opponents mate threat on g2 and adding an additional attacker to the black knight on F6. I completely missed that by moving the knight I was unveiling my opponents Queen to be able to attack my queen. Just a single move, only 64 squares, and I was entirely blind to the undermining of my idea. After black captures on D5 with the pawn and I take the knight on F6 with my bishop, my opponent can simply capture my queen, negating my mate threat and forcing me to respond by taking back with my bishop.

To my surprise, the solution was quite simple: I moved the rook one square up instead of to the right. I had been fixated on the idea of the rook moving horizontally, a pattern I was familiar with in trapping queens. Faced with this new scenario, I failed to reevaluate the rook’s positioning and didn’t realize that a slight vertical shift would solve the puzzle.

I had been trapped in my accustomed patterns and couldn’t see the obvious queen attack that should have been my initial consideration.

Inattentional Blindness?

It’s interesting how we can become blind in opposite degrees of experience. When we’re beginners, we struggle to discern crucial elements in a sea of information, unable to recognize common patterns and solutions. Our thoughts are malleable, as we miss many things but discover new patterns. As we become more seasoned, the details start to fade into the background, and a sort of “flow state” takes over, making everything seem almost automatic and fast. Taking a familiar walk around the block can feel like teleportation, with the journey happening almost unnoticed. Conversely, exploring a new place can overload our senses, making it easy to miss small details in the midst of absorbing so much new information.

There’s some fine balance between routine and new. New is good, it seems to slow down time and deepens perspective. However, routine seems to be critical to making considerable progress on activities I’d like to be good at, it removes the newness and mental overhead. I can focus on getting better or ‘teleport’ my way through difficult moments I might otherwise struggle to make it through.

Redlining

Sat, 18 Nov 2023 00:00:00 +0000

For some people, not red lining all the time is a skill in and of itself. Why wait until you have an injury or a mental health crisis to take a breath?

Reflection

I was brushing my teeth recently, paused, turned to my girlfriend and proclaimed that I wasted all four years of college on two degrees that were not challenging. I said that I should’ve majored in something “more difficult” like Engineering, Math or Computer Science because “I could have” and people wouldn’t “default to questioning my abilities”. The constant grappling with “I could have done more” or “I could have challenged myself” is probably good in some regard, but I believe the above scenario is unhealthy and bleeds into why I have found myself redlining ~~for no reason~~ at work over the last year or so.

Redlining: drive with (~~a car engine~~ the human brain) at or above its rated maximum rpm.

My girlfriend refuted the above points pretty quickly. Reflecting on the above, I realized I was being quite harsh on an 18 year old that had no idea what he wanted to do. I was being insecure about my background and over indexing on some irrational fear of my inability to learn and produce good work. I was discounting the fact that I have had and will have future opportunities to pursue furthering education in a variety of fields. I understand that there is some amount of life I can’t control, responding erratically and sprinting as fast as I can toward certain goals is suboptimal.

Consistency is efficiency, going 100 mph in most vehicles is not as fuel efficient as going 55 mph. You do not need to impress everyone at all time. Although I am not a car or a transformer, I think this applies.

At Work

At work, I would find myself trying to accommodate all types of requests as quickly as possible. There is a consistent sense of urgency about most things that people want done. I think the urgency is caused by people wanting to believe that what they are working on is urgent/important or being told that it’s urgent/important. In addition, others may also want to impress their peers/boss and the urgency surrounding that is subsequently passed on to the person they require something from. The presence of urgency in combination with the desire to help/impress coworkers, fuels a sense of wanting to get things done as quickly as possible redlining and stressing to deliver.

Most things are not important, if ten people come to you in a day telling you they have a hugely important task think twice
When evaluating a large project don’t assume that its value add, has been properly vetted, and the timelines make sense. Default to being skeptical
If extremely value add work could be done easily/quickly it would probably already be done, approach these types of request cautiously
Be flexible on implementation, the hot topic changes, do not take a change of direction to heart and pivot as necessary

Refrain from doing as much as possible: If you get a ton of cool stuff done but have no time to document it, iterate on it, and tell people about it… it will go unknown. If you work on a ton of stuff and get it all done fast most of that stuff is probably not useful, will require corrections, and be prone to going unnoticed

Spending more time investigating and figuring out why something is needed ~~is this actually valuable~~ is worth way more in the long run than solving useless problems. You can spin your wheels all day, redlining, while only having a small portion of your work be actually useful or… spend the extra time investigating to avoid useless work in the first place.

Investigating is not as mentally taxing. Working on a difficult problem, writing a bunch of code, figuring out why the code you wrote is wrong and doesn’t work is stressful and difficult. Asking people to provide you better inputs as to what they want, asking why they need what they asked for, asking how much benefit is to be derived, openly asking a plethora of questions to verify exactly what is needed all increases the quality of your work while reducing the mental bandwidth required in producing a useful artifact.

At the end of the day:

Most things will become out of date unless they are maintained, nobody will take the time to maintain them unless they are useful.
Nobody will have time to maintain many useful things if no-one uses them.

I also think this opens the door to more interesting job and role opportunities. Sure, you may not have produced as many artifacts as someone else but the depth and usefulness of the projects you did complete should outshine in comparison. Going deep on a few topics, creating something with tangible benefit, and then adequately documenting and sharing that thing scales exponentially while the ability to produce more and more “Stuff” scales linearly with the amount of time you want to sink into it.

On Stress

Tue, 27 Jun 2023 00:00:00 +0000

Stress

I’ve been meaning to jot down my thoughts on this for a while, but I keep running into things that tweak my mental model about stress.

Recently, I watched some old home videos. Twenty years ago, my parents moved us into the home we still live in today. Around this time, my dad filmed a video of my brother and I walking the quarter mile from our house to a canal that runs through metropolitan Phoenix. It’s about a ten-minute walk that probably took us thirty as we stopped for a snack on a small berm of grass, waddled around with empty backpacks that hit the backs of our knees and ran at my dad fists clenched screaming “Power!” for some reason.

Recently

Over the last few years, I’ve done this walk, probably, several thousand times. Walking, which I previously thought was only for old folks or people with nothing better to do, is something I now cherish and serves as my focal point for both learning and introspection. The walk to the canal hasn’t changed much over the last two decades, in fact, barring new people moving into the neighborhood the most drastic change would be me. Now an unrecognizable twenty-four year old strolling down the side-walk, sometimes still with my father.

I’ve become quite accustomed to this walk; it’s a routine for me, and for better or worse I like routines. Sometimes I switch it up, try a different path, walk to a different neighborhood; I like to pick out things that I’ve never noticed before each time to keep myself present. I’m comfortable; I find myself slipping into this state of comfortability, time passes by quickly. Walking in a new place alters time for me, my normal walk feels smooth I know where to go, I don’t even think, and before I know it I’m home again. A new place ‘slows’ down time.

Panic

Over the last year, I’ve suddenly begun to struggle with new places. I’ve done a fair bit of traveling in this period and, for the first time in my life, primarily by myself. Instead of taking on the whimsical nature of four-year-old me carelessly walking to the canal, I’ve become rigid. I have an underlying level of stress I can’t shake. Dealing with crowded places, meeting new people, taking in new sights; in my head I’m certain these are valuable moments. Yet I can’t shake the tension in the back of my neck, the nervous thoughts of what if I get sick, what if a lose my passport, what if I twist my ankle. These thoughts culminate in a panic that ebbs and flows coming to peaks in enclosed spaces where I have to sit for a while (restaurants, airplanes, etc…). Instead of enjoying these experiences, I usually just want to go home where I feel safe.

My initial reaction was that I had some kind of disease. The nervousness, loss of appetite and sweaty palms felt like I had a stomach flu. Yet the consistent pattern these symptoms lead me to believe I was just panicking. My concern for my health quickly turned to anger, why are you ruining this for yourself? Why can’t you just sit through a nice dinner with your family? Why are you shaking in the Uber to the airport? Just snap out of it. Shaming myself was not a successful strategy and did nothing to address the root cause of the problem.

Root Causing

Part of the reason I’ve struggled to write this is because I’m still working through it. I have a better understanding of what’s going on, and I’m much kinder to myself and can calm myself down. However, I’m still unnerved when I’m traveling and I’m still figuring out a root cause(s).

I’m doing a lot of new things fully on my own and assuming the risk and responsibility in doing so.
I think I’m social and outgoing, but I did spend 18 months between 2020 and 2021 almost completely isolated from social interaction outside of my parents.
During those 18 months, I would rarely leave home outside of going for my walks or bike rides, spending the rest of the time studying.
I put immense pressure on myself at work, currently, my work is one of the few avenues I have to get approval from other people I respect, and I value approval/recognition more than my monetary compensation.
A sense of urgency around staying up to date with new technologies and everything that’s going on in the world; I feel that taking time to relax is causing me to regress and will be detrimental to my long-term happiness (I know this is wrong but struggling to shake it).

I’m trying to find a middle ground between the happy, curious, unknowing toddler and the ridged, white knuckled, and loss-averse adult that I’ve found myself becoming.

Business Intelligence Engineer

Fri, 12 May 2023 00:00:00 +0000

Two months as a BIE

Promoted to BIE on the same team I have been on since September.

Worked hard during my first four months on the team, put together a doc with my boss and got approved starting about two months ago. Since Septmeber I’ve done a lot of stuff:

A little Background

- Redshift Cluster Maintenance and Query optimization of our longest running jobs
- Working toward establishing code reviews and a central code repository
- Created new production tables for partner teams
- Made adjustments/additions to some of our most critical tables that buoy all reports

How to improve

Looking back, I am very happy with my effort, but I want to do two things better.

ONE) I have not asked “WHY?” enough. I often get a request and my mind instantly goes into solve mode: how do I do this, how do I do this optimally, how quickly can I do this, etc… I need to clarify the ask and get a better understanding of if they EVEN need what they are asking for.

“Hey can you add this to the output of the model?”
“Why do you want that?”
“I’m not sure someone else just mentioned it”
“Ok let me know what the use case is, and I can see what I can do” _ Never hear about it again _

This helps to avoid useless work…

TWO) I’m struggling to detach from work. I just took two full weeks off; it was nice, I went to Europe, hung out with my girlfriend, saw family… but I also wanted to be working. I get way to attached to work, and I feel like my greatest source of satisfaction right now is recognition/ appeciation from people I work with.

I’ve recognized that my level of committment is 24/7 and my derivation of self-worth is much too entwined with my job. I am grateful to be passionate about what I do, and I am not saying that I want to de-commit, care less, and avoid work… I need an off button. I will work 7AM to 7PM no problem, when I’m done working I will go to the gym and think about how to solve work problems, and when I get home I will check my computer to see if any emails came in. I need to learn that throwing more effort, more concern, and more intent at a problem isn’t always the way… sometimes you need to close your copmuter at 5PM and leave it shut until the next day. I’m trying.

Forward Looking

One thing that might help with my above problem is I’m being forced into an office 3 days a week starting July 1. None of my team is in the same state is I am, but I will have to drive 30 minutes to plug my computer into a different monitor to take calls and write code.

On learning, I’ve been deep diving DBMS internals mostly through Andy Pavlo’s CMU lectures (so good). I have been tinkering around with different AWS resoruces, GIT, read a book on SPARK, learned about a python library called Polars, and continue to just consume a ton of information on data engineering tools/topics.

I love what I do everyday, I’m having a great time learning new technologies, and I’m grateful I get to help others with my learnings.

4 Months as an Analyst

Sat, 28 Jan 2023 00:00:00 +0000

Four Months as an Analyst at Amazon

My perspective switching to the Supply Chain BI team.

A little Background

The first four months have been interesting. Last week, Amazon laid off 18,000 people. Fortunately, nobody on my team or, I think, my whole org got the axe. This really hit home the fact that jobs come and go, and that I should be prioritizing employment that allows me to work on what I’m interested in and gain meaningful skills.

I’ve been working ~60-70 hours a week, mostly because I want to be good (I want to be the best) at my job, I’m learning a lot, and I’m about to transition to a BIE (given that 10 different people and our VP signs off that I am capable/deserve it). I have a deep desire to fulfill every request of me as quickly and accurately as possible which is both a good/bad trait. I’m learning how to prioritize my work and focus on one thing rather than bouncing around.

While I would be lying if promotions or wide-spread recognition isn’t nice, I’ve found that individuals reaching out to tell me they appreciate my work and my attentiveness to detail means the most to me. I also derive a lot of meaning from the 1:1 conversations I have with the people I view as mentors.

-Personally, I’ve been enjoying spending time with friends (Even reaching out and
getting together with people I haven’t seen in years). I’ve also been actively
organizing and spending time with my family; this is something I didn’t do in
high school/college outside of major holidays, now in my early 20’s,
I’ve attempted to rectify this before it becomes a major regret in my 30s/40s+.

-I’ve been studying poker, I played for the first time at a casino recently:
my hands were shaking the entire time, and I lost ~$300 over the course of
3 hours. I got an espresso machine, and I’ve enjoyed using it
(I am not a coffee snob!).

-One of my child-hood friends got married. It was surreal remembering
playing pool basketball and watching Iron Man at my 4th grade
birthday party/sleepover, now he has a wife.

-I’ve been continuing to run my book club at work, this month we are reading
Talking to Strangers by Malcolm Gladwell.

-I’m working on being the best son, brother, friend, peer, and partner I can be.

What has been not so appealing/difficult?

Creating reports, models, and queries for a new part of the supply chain that I’m less familiar with has been challenging. There are a variety of teams that create complex models that generate complex outputs to optimize different parts of our network. We consume these outputs into our Datawarehouse. I must interpret those results, figure out if they make sense, figure out if they make sense for our specific network, and then create visualizations that make those results easily accessible to PMs/Leadership.
Supply Chain Program Managers and their bosses each have a unique set of jargon and metrics for their silo/role in the supply chain. Understanding what everything means and what it means in the context of other jargon is quite complex!
We have close to 500 tables containing over a Petabyte of data on our Redshift Custer (Datawarehouse). Figuring out what tables are useful and accurate for my queries has taken months to develop. I now use a subset of 30-40 tables for almost all asks. 100/500 tables cover almost all asks from supply chain teams, the other 400 tables are used for the mist niche of requests and queried a handful of times per year. However, those tables still had to have long initial syncs and then subsequent data syncs (daily) that use up valuable/limited cluster resources.
I don’t like when other people’s reports break or when the numbers “aren’t quite right” for a dashboard. This usually entails digging through a ton of SQL (Sometimes poorly written) or trekking through all the steps in someone’s data pipeline, or worst of all digging into their scrappy solution where we are web-scraping into a an excel file and reformatting with VBA. At the end of the day, this is an important part of the job, but I feel like its thankless/non-value add work.
Dealing with issues that have been kicked down the road: Poor organization of our tableau dashboards, scrappy process’ that few people actually know about/can fix, POOR DOCUMENTATION (Please add comments in your queries, an info tab in your dashboards, and links to your ETL process), poor data validation/inadequate testing, and improper error handling/alert systems/code reviews.

What have I enjoyed?

I’m quite passionate about writing complex and performant SQL and helping to educate others do the same. I’ve written a couple documents about optimizing queries to scan less data and run faster, how to use the explain plan, and reached out to folks to help them write better queries. I’m going to be the mentor for one of our new interns starting next week, so I’ll be learning what the best way to get these concepts across quickly is. I’ve also written some stored procedures for database admin tasks, identified and fixed tables with improper sort/dist keys and retention policies, and adjusted poor performing ETL jobs.
I’ve gotten to work on some fascinating projects. How do we position/spread inventory as close to customer demand as possible? How do we improve our buying/forecasting systems and our responses to overbuying of inventory? Automating hundreds of hours of manual data gathering and reporting into a succinct/performant report. Why are we sending more inventory to warehouses that already have a significant % of our on hand quantity of that item? Building on Native AWS using CDK (Infrastructure as Code) and maintaining the with a CI/CD pipeline. Using AWS Sagemaker, can we predict how often a particular item will be within X miles of our customer at the time of order? What factors are driving our fulfillment optimization system to plan specific paths that will be used to fulfill customer demand for a particular fulfillment set?
I work with some incredibly intelligent folks who I’ve learned a lot from.
I have very few meetings each week, and the meetings I do have are generally quite interesting; I can actively contribute.
I’ve never been told not to pursue something I’m interested in. I am constantly presented the opportunity to learn new things; whenever we are looking into a new project that I’m interested in working on (Not time sensitive) or I have an idea, I don’t get the responses: “Let’s leave that to the data science team”, “Lets let the Sr. BIE handle this one”. It’s more along the lines of “Yeah go for it, reach out to _ for help, and don’t delete any of our AWS resources!”. At this point in my career, I’m very grateful for this.

Overall

-I’d like to start writing on here more. I’m going to maintain these longer career posts
at their current cadence 1-2x per year, but I think I’ll start writing some
shorter technical pieces, maybe some about supply chain optimization problems,
and some personal thoughts I can look back on and see what I was thinking
as a 23 year old.

Year 1

Thu, 25 Aug 2022 00:00:00 +0000

Year 1 at Amazon

My perspective working at Amazon as my first job out of college.

Background

After graduating college in May of 2021, I took a job at Amazon. I also had offers from PepsiCo and a small logistics company, but I felt I would learn the most, have the biggest impact, and gain the most opportunities from working at Amazon.

When people hear you work at Amazon, they think that you’re a part of everything. I’m not. Specifically, I work for AMAZON XL (AMXL). We are the heavy bulky/large parcel arm of our transportation network, delivering things like TVs, Treadmills, Mattresses, etc… (The big stuff).

My Job

My role involved helping launch new warehouses, new connections between warehouses, changing times trucks would depart warehouses, other supply chain related changes, automating tasks for partner teams (Python, VBA, SQL), writing queries for partner teams (ETL, SQL, VBA), etc…

Accomplishments

Automated the analysis of launching new lanes between warehouses based on distance and number of packages that would be added to those trucks.

Implementing a directed graph in Python, I evaluated every opportunity to connect an unconnected Origin-Destination pair given current network constraints.

Saved my team and partner teams thousands of hours of work annually through a variety of automation projects and queries.

I’ve done other cool things, but it’s difficult to describe some of them in a way that would make sense in the context of our supply chain in less than 400 words…

What I dislike about Amazon

My Value:

-I feel like some of the best data visualizations, automation projects, and queries I’ve built are underappreciated/underused

-Disheartens me from continuing to go above and beyond for any other reason
than my own learning

-Maybe this is my own doing: failure to distribute, educate, and advertise

There’s a lot of people:

-Many upstream/downstream stake holders must sign off on every change making things slow and difficult to approve (Not necessarily anyone’s fault, probably unavoidable with the size of the org)

-It's difficult to loop all relevant stakeholders in on a change that will affect them

Organizational Silos:

-People are good at what they do but often have no clue what most others due

-This leaves large gaps when people leave that may or may not have documentation to
clarify things with new hires

-This makes explaining things to other people cumbersome and time consuming

-It can be difficult to explain your wins/accomplishments to hiring managers/partner teams who aren’t deeply familiar with what you do

What I like about Amazon

There are lots of smart people to meet:

-Most people genuinely try to do a decent job. The distribution is skewed toward smart/hardworking employees

-The vast majority of people who have made it decently high up work hard, are smart, and
genuinely care about what they do.

Interesting projects/initiatives:

-There are always interesting/high impact projects that you learn about and get involved in if you poke around enough

-It's not very difficult or time consuming to develop a reputation as a hardworking
individual contributor

-Your large wins are broadcast to hundreds/thousands of people

Career:

-You have access to a variety of different jobs/orgs that you can switch to based on your interests/career trajectory

The Future

Overall, I’ve had a positive experience. After a little over a year, I’m being promoted to a business-analyst with the hopes of becoming a BIE or Data Scientist. I’ve opted for more of a tech path, but feel like I could transfer back into a supply chain management role should I want to… I don’t know if I could do the other way around, thus, I’ve chosen this path to keep my options open.

I’m starting to understand that I know little in a large pool of people, working on a variety of different goals, with different ambitions, and an underlying directive to meet pre-defined metrics.

About

Mon, 01 Jan 0001 00:00:00 +0000

Hi there

My name is Sam, and, sometimes, I like to write.

My hobbies are chess, poker, riding bikes, hiking, computer science, and being a good person.

I am currently working as a Data Engineer at Amazon.

My work includes:

AWS (Redshift, SNS/SQS, Glue, EMR, Lambda, etc..) managed via CDK
Database Administration/Maintenance (Multiple Redshift Clusters, RDS, and Dynamo DB)
I read/write a lot of SQL (mostly in Redshift which is a fork of Postgres or EMR using the Spark SQL API) also Python, TypeScript, and Java
Creating batch & streaming data pipelines, data modeling
Develop LP models, simulations, etc… to optimize small aspects of our supply chain

What I’m learning about/deep diving right now:

Writing/Understanding OCaml
AWS Advanced Networking
Trying to finish The Count of Monte Cristo
My stand up comedy routine

Relevant Links:

“I cannot remember the books I’ve read any more than the meals I have eaten; even so, they have made me.” ― Ralph Waldo Emerson

i	rowcount
0	1
1	2
2	3
3	4
4	5
5	6
6	7
7	8
8	9
9	10
10	10
11	10
12	10
13	10
14	10
15	10
16	10
17	10
18	10
19	10
20	10
21	10

i	row_count	source_d	row_num
1	1	1	1
2	2	1	1
2	2	2	2
3	3	1	1
3	3	2	2
3	3	3	3
4	3	1	1
4	3	2	2
4	3	3	3
5	3	1	1
5	3	2	2
5	3	3	3
6	3	1	1
6	3	2	2
6	3	3	3
7	3	1	1
7	3	2	2
7	3	3	3
8	3	1	1
8	3	2	2
8	3	3	3
9	3	1	1
9	3	2	2
9	3	3	3
10	3	1	1
10	3	2	2
10	3	3	3

i	rowcount
0	1
1	2
2	3
3	4
4	5
5	6
6	7
7	8
8	9
9	10
10	10
11	10
12	10
13	10
14	10
15	10
16	10
17	10
18	10
19	10
20	10
21	10

i	row_count	source_d	row_num
1	1	1	1
2	2	1	1
2	2	2	2
3	3	1	1
3	3	2	2
3	3	3	3
4	3	1	1
4	3	2	2
4	3	3	3
5	3	1	1
5	3	2	2
5	3	3	3
6	3	1	1
6	3	2	2
6	3	3	3
7	3	1	1
7	3	2	2
7	3	3	3
8	3	1	1
8	3	2	2
8	3	3	3
9	3	1	1
9	3	2	2
9	3	3	3
10	3	1	1
10	3	2	2
10	3	3	3

i	rowcount
0	1
1	2
2	3
3	4
4	5
5	6
6	7
7	8
8	9
9	10
10	10
11	10
12	10
13	10
14	10
15	10
16	10
17	10
18	10
19	10
20	10
21	10

i	row_count	source_d	row_num
1	1	1	1
2	2	1	1
2	2	2	2
3	3	1	1
3	3	2	2
3	3	3	3
4	3	1	1
4	3	2	2
4	3	3	3
5	3	1	1
5	3	2	2
5	3	3	3
6	3	1	1
6	3	2	2
6	3	3	3
7	3	1	1
7	3	2	2
7	3	3	3
8	3	1	1
8	3	2	2
8	3	3	3
9	3	1	1
9	3	2	2
9	3	3	3
10	3	1	1
10	3	2	2
10	3	3	3