At Baseshift we routinely take snapshots of databases for developers and agents, typically output as Docker images, with block-level copy-on-write. To do this in a portable and database-agnostic manner, we chose an architecture that works with unmodified database binaries without requiring a special file system, driver, or storage subsystem: a bespoke libc for databases.
This architecture has the benefit of affording us the freedom to innovate and rapidly iterate on the components of the software stack, storage in particular, that affect performance, cost, and functionality.
Today we're proud to announce Database Compression V1 (DC1), our first such innovation. On real-world databases, DC1 compresses PostgreSQL data well beyond what block-level codecs achieve today, while maintaining random read access to 8 KB blocks with decode latencies below those of cloud storage such as AWS EBS and S3. On OpenStreetMap, it even outperforms whole-file compression with Zstandard, xz and zpaq.
To make that concrete: the full United States OpenStreetMap database goes from 242 GB to 28.7 GB, a ratio of 8.4×, while Zstandard -19 with trained dictionaries, on the same 8 KB blocks, produces twice that size. A DC1 block decodes in about 0.18 ms on average, on one CPU core.
DC1 is a research preview, and you can try it today: the Docker images are further down.
The trade-off breakthrough
Block compression for a database trades three things against each other: the compression ratio, how long a read takes to decode, and the block size (the smallest piece you can read on its own). Larger blocks compress better, but every read then decodes more data.
In this benchmark DC1 averages a compression ratio of 6.32× at 8 KB. At 8 KB, Zstandard -19 reaches 3.44× without a dictionary and 4.29× with one, while xz reaches 3.74×. xz has to grow its blocks to 8 MB to match DC1's ratio, and a single read then costs 72 ms instead of 0.18 ms.
How it works
In 2016, Yann Collet and Chip Turner at Meta showed that Zstandard with a dictionary trained on sample data makes small-block compression far better, improving the ratio of the github-users sample set from 2.86× to 6.9×, a ~2.4 fold gain. DC1 takes that idea a step further.
With Zstandard, a trained dictionary lets each block reference common byte sequences. But a table’s values can have higher order structure, across columns and from row to row, and that structure differs from table to table. DC1’s approach is to model the structure of a whole block with one of a family of codecs, favoring computation that can run in parallel.
Each block is stored with whichever codec encodes it smallest. Some codecs are bespoke, built for structures we know well, such as indexes and geometry. DC1 also introduces neural codecs, which learn the structure from each table’s own blocks and store only what they fail to predict. Their encoding is deliberately designed for concurrent decoding, to ride a hardware trend: for two decades, chips from CPUs to GPUs have grown wider rather than faster at running a single thread. So the neural codecs decode a block in a few passes, each predicting many values at once. Below we show the gain from AVX-512 VNNI.
All of this runs inside the libc layer described at the top, so PostgreSQL itself stays unmodified.
How it compares
Against block-level compression
Baseshift snapshots already compress every block with Zstandard -3 (the default), reaching 2.5:1 to 4:1. In the comparison below, every method compresses the same 8 KB blocks independently. We measured on held-out blocks, which the models and dictionaries never saw during training, and counted everything DC1 ships in its size.
Across six databases (OpenStreetMap US and New York, MusicBrainz, and three sizes of a synthetic benchmark, 610 GB in total), DC1's output is:
- 57% smaller than Zstandard -3
- 48% smaller than Brotli (quality 9)
- 38% smaller than Zstandard -19 with trained dictionaries
The margin over Zstandard -19 with dictionaries is widest on OpenStreetMap (51%), where we’ve experimented with fine-tuned codecs.
Against whole-file compression, for reference
Whole-file compressors are not an option for a live database. To read one block from a 1 GB database file (compressed as one stream with no seek points) you decompress, on average, half of it: about half a second with Zstandard, three seconds with xz, and about fifteen minutes with zpaq at its strongest setting. They are still a useful yardstick, because they see the entire file at once and show how much redundancy is there to find.
On OpenStreetMap, DC1 beats all of them, one block at a time. On the US database its output is 46% smaller than whole-file Zstandard -22, 31% smaller than xz -9e, and 15% smaller than zpaq -m5.
On MusicBrainz and the synthetic benchmark, xz and zpaq stay ahead. Across the whole suite DC1 and xz are roughly level, and zpaq -m5 leads.
Index compression
In the US OpenStreetMap database, the primary-key index of the nodes table compresses 101.8:1 and that of the ways table 53.3:1. All B-tree indexes in that database, taken together, compress 60.7:1. We attribute these results to sorted keys and row pointers that advance almost in lockstep, so a purpose-built codec stores little beyond the irregular steps. Indexes on random keys, such as UUIDs, gain much less.
Decode time in context
DC1's mean decode time is about 0.18 ms per block. The neural codecs are the slowest, typically 0.15 to 0.45 ms per block on one x86 core with AVX-512 VNNI; small next to a read from network storage. On local NVMe, where a read takes tens of microseconds, it is noticeable.
This is the part we expect to improve. The neural decoders are small integer matrix arithmetic, the kind of work that processors keep getting better at. Averaged over the benchmark's reads, decoding takes about 1.46 ms with portable C and 0.18 ms with AVX-512 VNNI kernels: an 8× speed-up with no change to the format.
Try it
We've published two OpenStreetMap databases as Docker images, with PostgreSQL 17 and PostGIS, already encoded with DC1. They run natively on both amd64 and arm64.
Sizes are the database payload, including every model, table and dictionary needed to decode it. The Docker images also contain PostgreSQL, extensions and runtime files.
Start with New York:
Once PostgreSQL is up, connect with psql, no password needed.
The full United States is at baseshift/db-osm-us.
Map data © OpenStreetMap contributors, available under the Open Database License.
What’s next
DC1 is a proving ground for what’s possible: a showcase of experiments leveraging our libc-based storage architecture exploring a variety of ideas. We’re excited about building on these results: providing more research preview databases (more OpenStreetMap countries and US states coming soon!), iterating with DC2 based on lessons learned, and integrating codecs ready for market into Baseshift’s snapshot workflows.
Put Baseshift to work for you
Baseshift gives developers, CI pipelines and coding agents isolated, masked database environments built from the databases their teams already run: realistic data for testing, debugging, migrations and agent workflows.
Try Baseshift, point your coding agent at baseshift.com/auth.md, or contact us for a demo.
