Posted 09-Aug-2026 in Software Development tagged AI

Reverse Engineering a PostHog SuperDay

I'm not here to bullshit anybody, I have never been through a PostHog SuperDay. Like many humans, I've read about it on their website, as they are a company that over-communicates just about everything.

This tale started when I got curious about their onboarding wizard. Their pivot from analytics to self-driving systems mirrors what I've been working on for the last couple of years, which has accelerated greatly over the last 3 months. Figured there was probably something I could learn and possibly bite off to include in my own system.

Figuring out what to do

I've been around open source software (OSS) for a good long while now. Even though I've had some strong opinions about the state of things over the years, I still believe it's the way. If not for OSS, I wouldn't be able to say things like "I wrote a Lorem Ipsum generator that's somehow been installed over 1 million times."

These days, most projects have a label on their issues that indicate what's good for a new contributor to pick on. PostHog's repos are no different, except the repo I was looking at didn't have any issues tagged with the good first issue tag.

Not a big deal, with 100+ open issues I figured I'd be able to find something I could pick up to give me a chance to play with the codebase. I did find a handful of things, I even started to chase down one of them. As I did I realized I kept talking myself out of each issue because what appeared to be low hanging fruit was stuff that I could defend deprioritizing.

A pattern started to emerge

Along this journey of trying to find an issue worth fighting for, I started to notice a pattern of somewhat disparate issues and PRs opened by the same contributor. I also noticed that core team members (read: employees) were commenting on these issues and PRs and not in the usual "yea or nay" fashion that I see.

The comments were constructive in a way that you'd be with a more junior teammate that you're attempting to mentor and train up. Not a bad thing, but the feedback was more open ended rather than "hey I think this will cause an issue, go fix it" that you usually run into.

With my hamster wheel turning, it started to seem like these issues and PRs weren't from outside contributors as much as they were from actual candidates. Didn't hurt that the username of one of the users actually had the word "candidate" in it.

Even if my theory is incorrect, the feedback from the core team members seemed valuable, and I used some of what was out there to analyze some of my own work to see what could be learned and improved upon.

The simulation

At this point I was pretty convinced I was watching SuperDays play out in real time on the repo. Having talked myself out of a handful of issues to work, I decided it was time to send one of my robot friends to analyze the situation.

The results? After analyzing the data that seemed to be from candidates, cross referencing the team and their openly documented goals, I was feeling an awful lot like Charlie Day:

I was also feeling like I'd found an issue to work on. It was filed by an actual employee and not a potential candidate. The ticket was decently scoped, even though I pursued a quicker fix than what was defined.

It was backed by a report from an actual user. It was also an issue that I was eyeing already because it was something I had run into while setting things up myself.

Best part, it seemed to line up pretty nicely with one of their published quarterly goals on their website. The wizard team's TUI revamp explicitly states "users discover and run wizard programs without the exact CLI command".

Most importantly, it was currently unassigned so ideally I wasn't stepping on any toes.

Are you going to tell us?

Yes, I'm going to tell you which issue I picked up. But the whole point of this post is less about the what and more about the how and the why. So the issue I had picked up was related to running PostHog's wizard more than once on a repo.

As mentioned, I ran into the issue when running it myself. Even though the project does "2 hours of work in 8 minutes", if you don't remember if you already ran it, or worse, ran into something that forces you to run it again, you're paying round trip prices for each subsequent run.

My thought was that there was an easy win in there to do some light detective work and then raise some awareness of what else the script can do. I scope intentionally thin for a few reasons.

First, oftentimes PRs are completely ignored and never merged. Zero reason to boil the ocean when your contribution will be negated. Second, smaller scope makes it easier to get something into production. Ship fast, get feedback fast. Iterate from there.

Claude as my navigator

Similar to how I never denied using Google or Stack Overflow to research stuff, I'm not going to act like I don't use the heck out of AI. I play with different harnesses, and models. Claude's still my ride or die at the moment so of course I leveraged it to get up to speed on the codebase.

Since I wanted to actually learn the code a bit, and knock some ring rust off, I thought that I'd leverage my friendly robot more like a mentor than a subordinate. I did have to keep an eye on which mode I'd left it in. Usually I don't juggle between modes, but I wanted to be a bit more explicit here allowing for correct attribution.

It worked out surprisingly well, as I mostly kept Claude Code in plan mode so it didn't make any code changes. I used it to learn the flow of the system and figure out where I needed to make the changes I wanted to make.

I like to follow a strict test-driven workflow. Tests and business logic never to be committed at the same time. It works out most of the time, allowing me to have a clean history and helps keep the robots honest so they aren't munging tests to make their code seem functional.

Other than that, it was a pretty boring process. Pair programming with the direction of Claude Code. Figuring out how to test things and run them against my project code. Figuring out conventions, in some cases, the lack thereof.

The result of an afternoon (and part of an evening) of effort is living in this PR. Both the PR and the comment on the issue are unacknowledged at the time of this writing.

Not even mad

I'm no stranger to open source contributions being flat out ignored. I have a PR in flight on eleventy-plugin-rss to fix a bug I ran into a while back. The times I've actually cared, I come in super hot, fork the project and start to operate like a project lead rather than an unpaid intern.

That all said, the process was fun, and I learned quite a bit. Including but not limited to playing with pnpm for the first time and having a TIL moment with tsdown.

The bigger code takeaways were related to an adjacent repo that PostHog uses to manage their context files, and the friendly reminder that I should be open sourcing more of my code.

Over this weekend I actually put my Igor agent out in the open and started a "context mill" inspired project I'm calling the Distillery to serve as a shared library of my context files and skills.

Both are part of my journey to crack the nut of complete automation and are licensed under the GNU Public License Version 3 (GPLv3). You can thank this post for that.

Where I probably missed the mark

Let's say I did reverse a small bit of PostHog's SuperDay. If I had to judge the work I did against what I think their expectations are, I'd honestly say I probably missed the mark.

Even though I ganked the idea of the "context mill" for my own gains, I actually didn't bother doing much with the repo. I acknowledged its existence, I learned a small bit about it, but I didn't bother to see if there was anything out there that could have been a more impactful task. Nor did I identify a task that could potentially touch both repos.

I'm a builder through and through, so I've always defaulted to code. Code's just a smaller part of the equation in the generative AI age. I did my default here, but also used it as a chance to pick apart myself a bit.

Probably going to save that for another post, but all in all I would say that I enjoyed this process. I definitely have some new tricks to working with legacy codebases, which I have done regularly throughout my career. I have a few action items to work on for myself, and of course it's always great when you learn something new and implement it into your own project.

:wq