Column-Family Stores

You're building a messaging platform. Each user has a mailbox. Some users have 10 messages. Power users have 500,000. Each message has different metadata, some have attachments, some have read receipts, some have forwarding chains.

A relational database could handle this, but at the scale of billions of users and trillions of messages, you hit walls. Joins between tables become impossibly expensive. Sharding by user requires careful coordination. Schema changes on a multi-petabyte table can take days.

Facebook faced this exact problem in 2008 with their inbox search. They built a system on top of HBase, a column-family store, that could handle their scale. Apple uses Cassandra (another column-family store) to store over 10 petabytes of data powering iCloud services. Netflix, Instagram, and Discord all rely on Cassandra for workloads that relational databases couldn't handle at their scale.

Column-family stores sit in a unique spot: they're not relational databases, they're not simple key-value stores, and despite the name, they're not the same thing as columnar databases. That last point trips people up constantly. Let's clear it up.

A column-family store organizes data into rows and column families, where each row can have different columns within a family. Think of it as a two-dimensional map: the first key is the row key, the second key is the column name, and the value is the cell.

Here's what a user inbox might look like in Cassandra:

Row Key: "user:alice"
 Column Family: "messages"
 "msg:001" → { from: "bob", body: "Hey!", timestamp: 1711234000 }
 "msg:002" → { from: "carol", body: "Meeting at 3pm", timestamp: 1711234100, attachment: "agenda.pdf" }
 "msg:003" → { from: "dave", body: "Got it", timestamp: 1711234200 }

Row Key: "user:bob"
 Column Family: "messages"
 "msg:001" → { from: "alice", body: "Thanks!", timestamp: 1711234300 }

Notice: Alice has 3 messages, Bob has 1. Alice's msg:002 has an attachment field that others don't. This is fine, each row can have different columns. No NULL-filled sparse table like in a relational database.

How this differs from columnar storage:

• Columnar storage (BigQuery, Redshift) stores each column separately on disk for analytical scans
• Column-family stores group columns into families, and store each family together per row, optimized for reading a row's data within a family

The "column" in the name refers to the data model (flexible columns per row), not the storage layout.