We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
System Design and AI Engineering, by Roni Das. 984 interactive lessons.
Start LearningIntermediate · Database Types & Storage
A NoSQL database that groups columns into families, optimized for reading and writing large amounts of data across many machines. Cassandra and HBase use this model.
In short
A column-family store is a NoSQL database that groups related columns into units called column families and stores all the data for a row together on disk, partitioned by a row key across many machines. It is built for very high write throughput and for reading wide rows fast, and it is the model behind Apache Cassandra, HBase, and Google Bigtable.
In short
A column-family store is a NoSQL database that groups related columns into units called column families and stores all the data for a row together on disk, partitioned by a row key across many machines. It is built for very high write throughput and for reading wide rows fast, and it is the model behind Apache Cassandra, HBase, and Google Bigtable.
Learn it in the course
This page is the free part.
₹1,111 in India$25 everywhere elseonce, for the whole course
The System Design course covers Column-Family Store across a run of lessons, not one page. These 4 alone are about 156 minutes of step-by-step reading, 1 of them with code you run in the browser, all with a quiz.
Wide-column databases like Cassandra and HBase, built for massive scale with flexible schemas.
How Apache Ignite runs SQL on a partitioned cache. Measured tests show a query with the key stays fast, and one without it slows as the cluster grows.
Database model comparison, relational vs non-relational data stores.
Delta Lake, Apache Iceberg and Apache Hudi: what is actually in the metadata? I wrote the same 11,198,026 NYC taxi trips into all three, three commits each, and opened every metadata file. Delta kept 3 JSON commits, Iceberg 10 files in a four-level tree, Hudi a timeline and a metadata table. All three returned January's exact 3,475,226 rows when asked for the first version. After 120 small appends, an Iceberg planner opened 122 metadata files and a Delta reader 22; Hudi packed the rows into 1 file group instead of 120 files. Compaction fixed the count, and cleanup ended time travel.
all part of
System Design Masterclass
770 lessons · about 287 hours · 18 free to read
₹1,111in India, by UPI
$25everywhere else, by PayPal
It is really nice and good Please join the course
You own this course
Continue with Column-Family Storespay once,
yours for life
A column-family store keeps data in rows, but those rows are far more flexible than a relational table. Every row has a unique row key, and under that key the data is organized into column families. A column family is a named group of related columns that get stored together on disk. Inside a family, each row can carry a different set of columns, and rows can hold thousands or even millions of columns. There is no fixed schema forcing every row to look the same.
A concrete way to picture it: think of a giant sorted map. The outer key is the row key. The value is itself a map of column families, and each family is a map of column name to a value with a timestamp. Cassandra and HBase both build on this idea, which comes straight from Google's 2006 Bigtable paper. The term wide-column store is used for the same thing because rows can be extremely wide.
This is not the same as a columnar analytics database like ClickHouse or Redshift, even though the names sound alike. Those split every column into its own file to scan billions of rows fast. A column-family store instead clusters a row's data together so it can fetch or update one entity by key with low latency.
The write path is the reason these systems are fast. A write first lands in an append-only commit log on disk for durability, then goes into an in-memory structure called a memtable. The client gets an acknowledgment almost immediately. When a memtable fills up, it is flushed to disk as an immutable file called an SSTable. Because SSTables are never modified after they are written, there is no expensive read-modify-write on the disk.
Over time a single row's data ends up spread across the memtable and several SSTables. A background process called compaction periodically merges SSTables, drops deleted entries, and keeps the newest value for each column based on its timestamp. Reads use a Bloom filter and an index per SSTable to skip files that cannot contain the requested key, so a read touches as few files as possible.
Data is spread across the cluster by hashing or ordering the row key. In Cassandra every node is equal and uses consistent hashing on a token ring, so there is no master to become a bottleneck or a single point of failure. HBase instead splits the key space into regions managed by region servers on top of HDFS. Both let you tune how many replicas must respond to a read or write, trading consistency against availability and latency.
Reach for a column-family store when you have a huge volume of writes, a key you can query by, and a need to scale horizontally across many cheap machines. Time-series data, event logs, sensor readings, user activity feeds, and chat messages all fit well. If your write rate is hundreds of thousands of operations per second and you need predictable low latency, this model holds up where a single relational server would fall over.
The cost is that you give up the conveniences of SQL. There are no joins, foreign keys, or ad-hoc queries across arbitrary columns. You design tables around the exact queries your application runs, which usually means duplicating the same data into several differently shaped tables. Get the access pattern wrong and you may have to remodel and rewrite a lot of data.
Consistency is also a choice you make per operation rather than a guarantee you get for free. With tunable consistency you can read stale data if you favor availability, and features like counters or lightweight transactions are limited compared to a relational engine. If you need complex transactions, rich querying, or strong consistency by default, a relational database such as PostgreSQL is the better tool.
Discord stores chat messages this way. The natural query is give me the messages in this channel around this time, so the data is modeled with a partition key built from the channel id plus a time bucket, and a clustering key on the message id so messages stay sorted by time inside each partition. Reading a page of chat history becomes a single lookup of one contiguous partition rather than a scan.
At that scale, which grew into trillions of messages, Discord first ran on Cassandra and later moved to ScyllaDB, a C++ rewrite of the same model, to cut tail latency and reduce the number of nodes. The schema barely changed because both speak the same column-family data model. That portability is one practical benefit of designing around the row-key-and-column-family shape rather than a specific product.
Apache Cassandra
The reference open-source column-family store; powers write-heavy workloads at Netflix, Apple, and Instagram with no single master node.
Apache HBase
Column-family store built on HDFS that backs large-scale Hadoop pipelines and once stored Facebook Messages.
Google Bigtable
The original system this model is based on; runs Google Search indexing, Maps, and Gmail metadata.
Discord
Stores trillions of chat messages keyed by channel and time bucket, originally on Cassandra and now on ScyllaDB.
No. A columnar database like ClickHouse or Amazon Redshift physically stores each column separately to speed up analytical scans and aggregations. A column-family store groups columns by row key and stores a row's data together, optimized for key-based reads and high write throughput, not full-table analytics.
The row key uniquely identifies and partitions a row across the cluster. A column family is a named group of related columns stored together on disk. A column is a single name-value pair with a timestamp that lives inside a column family. You define families up front but can add columns freely per row.
No. There are no SQL-style joins. You model data by duplicating it into tables shaped around each query you need, and you read by a known row or partition key. If your application needs ad-hoc joins and flexible filtering, a relational database fits better.
Writes append to an in-memory memtable and a commit log, then later flush to immutable SSTable files. Nothing is overwritten in place, so there is no slow read-modify-write on disk. Cleanup happens in the background through compaction, which keeps the write path cheap.
Choose Cassandra when you want a masterless, multi-datacenter setup with tunable consistency and simpler operations. Choose HBase when you are already in the Hadoop and HDFS ecosystem, need strong single-row consistency, and want tight integration with MapReduce or Spark batch jobs.
NoSQL
Databases that don't use traditional table-based relational models. Includes document stores, key-value, graph, and column-family databases.
Document Database
A NoSQL database that stores data as flexible JSON-like documents. MongoDB and CouchDB let each document have a different structure.
Key-Value Store
The simplest NoSQL model: store data as key-value pairs. Blazing fast lookups by key. Redis, DynamoDB, and etcd are key-value stores.
Redis
An in-memory data store used as a cache, message broker, and database. Blazing fast because everything lives in RAM.
Memcached
A simple, high-performance distributed memory caching system. Stores key-value pairs in RAM. Simpler than Redis but less feature-rich.
Graph Database
A database built for storing and querying relationships between entities. Nodes are entities, edges are relationships. Neo4j is the most popular.
This page explains the idea. The "Column-Family Stores" lesson builds it up step by step, with diagrams. It is part of the System Design Masterclass course, which has 770 lessons. 18 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Column-Family Stores lessoncourse 1
From absolute beginner to principal engineer, drawn step by step.
Asked about Column-Family Store in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (268 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
