At Baseshift we ❤️ Django, but it comes with some challenges. As we increasingly switched to the software factory paradigm, we started noticing our CI unit tests take over 100 seconds just to run the migrations on an empty database. This setup time increases as migrations get piled on, both in our apps and third-party ones like dj-stripe. Using brancher, we brought that time down to about 6 seconds.
When migrations are infrequent, this is resolvable with a cached pg_dump or by squashing migrations, but both fall apart at high velocity:
First, when multiple agents are pushing both third-party updates and new features that require repeatedly testing out migrations, CI runs require careful specification of what pg_dump output to cache such that a run can start from a common ancestor, and that common ancestor is a moving target.
Second, squashing migrations is easier said than done when your project includes third-party apps; even more so when the third-party app is referenced. An old Django ticket shows one reason why. It’s not impossible, but it is cumbersome.
Combine both issues and it gets worse, because concurrent efforts need to coordinate the squash.
The ideal solution would have the following properties:
- Works with standard databases, e.g. regular PostgreSQL, MySQL, etc., and supports the same extensions
There are lookalikes (DuckDB, Dolt[gres], etc.), but that creates risk due to small differences between the production and dev/test databases. - Cacheable, typically as a directory with files
- Works natively in a CI job
- Supports branching, like git
brancher solves this problem and has all the above properties. brancher adds branching functionality to native PostgreSQL, MySQL, and MongoDB. It has the added benefit of turning the database state into versioned artifacts that can be cached by CI.
Here’s how it works:
Using brancher, the state of the Postgres database used for unit tests is a set of files on disk. The state of the database is branchable, like git: each time the db starts up, all changes are stored in separate files. The files are referenced by their branch, such that each migration is like a commit in git, with files that reference the previous state. The files include the full database state, including schema and table content. When the whole set of files is cached by the CI job, the whole lineage is saved, so it’s possible to branch out of any state in the lineage. This way, jobs continuing the main branch can leverage its latest state, while branches in development can leverage the lineage up to the last common ancestor. A garbage collection job can clean up orphaned states to clear up the cache.
Our Django plugin, django-baseshift-brancher, seamlessly integrates Django test frameworks (Django unit test and pytest-django) with brancher to detect when a migration has occurred, and which database states are relevant to the latest migration state of the Django project.
Here’s a comparison of the same run running migrations without brancher and with brancher:
When no migrations are necessary, the test saves almost 100 seconds of startup time and instantly has a PostgreSQL instance available for spin up; it just checks that there are no migrations to run:
When a migration is necessary, it can be applied to the last common ancestor’s state:
We have 3,000+ tests, all of which use the database, so we use pytest-xdist to run 4 workers concurrently. Each worker attaches to an ephemeral database clone, started once before pytest from the database state with all migrations applied. These clone instances are intentionally configured to be lightweight and short-lived: fsync() is not applied, and some state is not persisted to disk in real time to save CPU and I/O resources.
When the job completes, if there is a new migration applied, its state is uploaded to the cache as files to be used in subsequent CI runs.
Try it on your Django project
Paste this into your coding agent:
brancher runs on Linux (glibc 2.39+) and macOS on Apple silicon. Learn more at baseshift.com/brancher, and find the plugin on PyPI.