Announcing Diagonal Datasets

For sixty years the field has recognized two orientations. We are formally proposing a third, and we have already registered the domain, which we understand is the hard part.

Every data system ever shipped has asked you the same question, and it has only ever offered you two answers.

Row-oriented? Or column-oriented?

That's it. That's the whole menu. Postgres, Parquet, Cassandra, ClickHouse, that thing your staff engineer wrote in 2019 that everyone is afraid to delete — all of them, without exception, storing data along one of exactly two axes. Zero degrees or ninety degrees. Flat or upright.

We think this is insane.

The unexamined right angle

Consider what you are actually being asked. You have a two-dimensional structure. Two dimensions admit infinitely many directions. And the industry has, with total consensus and zero recorded debate, chosen the two most boring ones.

Nobody voted on this. There was no RFC. There is no paper titled Why Only Two that anyone can produce when asked, and we have asked, repeatedly, sometimes at volume, at three separate conferences.

The right angle got in early and it has been coasting ever since.

What we are proposing

A diagonal dataset stores values along a band oriented at 45 degrees to both the row axis and the column axis. Cells that fall inside the band are retained. Cells that fall outside the band are discarded — not deleted, not archived, discarded, because they were never diagonal in the first place and their presence in your warehouse was always a kind of accident.

The immediate consequences:

• Your storage footprint drops by roughly 1 - 2b/n for a band of half-width b over an n × n matrix. At b = 1 and n = 4096 that is a 99.95% reduction. We are aware of what we just said.
• You partition by key and by time in a single cut, because the diagonal is the only line that moves through both simultaneously.
• Your storage layer becomes visually interesting for the first time in the history of the discipline.

Anticipated objections

"You are just describing a banded matrix." Banded matrices are a numerical linear algebra technique from the 1950s for solving systems of equations. We are describing a storage orientation and a worldview. These are completely different and we would appreciate it if you stopped bringing this up.

"You lost 99.95% of the data." We retained 100% of the diagonal.

"Which 0.05% did you keep?" The correct 0.05%.

What happens next

We are publishing our notes here as we go. There is a specification, which is unfinished in the way that all important specifications are unfinished. There are benchmarks, which are extremely favorable, in part because we wrote both the benchmark and the thing being benchmarked. There is a lab where you can diagonalize a matrix in your browser and watch most of it disappear.

The data was always there. We just needed the right angle.