I Cut My Test Suite Time in Half by Slicing Up a CSV File
I made a confession recently on LinkedIn:
I've been running Holiday API for the last decade on spreadsheets.
Not one of those fictional accounts that so many people post, I actually did run everything in a spreadsheet. It allowed me to have an admin interface without building one, and the friction was minimal but not zero. For the most part, CSV files are easy enough to parse and extract data from, even when they are massive.
Easy to work with isn't the same as fast though. I'm a caching wizard on the live site, but in my test code, I like to skip the cache so tests are run from a cold start. In this scenario, no one test can influence the results of another, as it should be.
My test suite was already plenty fast, clocking in just north of 10 minutes. Years ago it was taking 15+ minutes so I did some performance tuning and always felt like it was in a sufficient place. Keep in mind we're talking about over 1.2 million tests, mostly related to data regression.
Part of my recent confession was because I was finally moving away from using spreadsheets, but keeping the CSVs since it ain't broke. I do my best to have some decent file hygiene, maintaining smaller files with the exception of one monolith file.
This big bad CSV file was 557 columns and topped 45k cells total. I ended up breaking it down into 70 files, the largest of which was only 5% the size of the original file.
Thanks to my massive test suite, the code changes to support this restructure proved I didn't completely mess anything up.
Since the monolithic file was being loaded with just about every test, it was adding a significant amount of time to the test builds. The rework cut the test suite down by over 50% and got things down to less than 5 minutes.
The wildest part is that I've since added another 300k regression tests against the data and it hasn't had much impact on the run time.
Even though a 10-minute test build isn't that crazy at all, cutting things down to less than 5 minutes has been very noticeable. I can go from push to production in roughly 10 minutes, with the tests running as part of the GitLab MR and then again before deploy because I'm paranoid like that.
Before, it would actually take closer to 30-40 minutes due to a flaky test and something wonky in the way the Redis container works on GitLab. Somewhat ironic that my tests don't use the cache, but my tests related to caching were partly to blame. Fortunately, since speeding up the test suite, the flaky test has calmed down.
If you're looking for somebody who genuinely loves optimizing CI/CD pipelines for your next project, feel free to drop me a line and tell me what you're working on.
:wq
Like this drivel? There's a whole RSS feed of it, or subscribe via email.