Elasticsearch

Elasticsearch is a document database with a search engine bolted into its core. You send it JSON documents over a REST API, it stores them, and it builds an inverted index so that queries like "find every document containing the word laptop" return in milliseconds instead of scanning every row.

The thing that makes it different from a normal database is the inverted index. A relational database is optimized to fetch a row by its primary key. Elasticsearch is optimized to answer "which documents contain this term", which is exactly the question full-text search asks. It also handles fuzzy matching, relevance ranking, and synonyms, none of which a SQL LIKE query does well.

It is distributed by design. A single logical index is split into pieces called shards, and those shards are copied into replicas and spread across machines. That is what lets companies run Elasticsearch over hundreds of terabytes while still returning a query in under 100 milliseconds.

Each Elasticsearch index is divided into shards. A shard is a complete, self contained Lucene index. When you index a document, Elasticsearch hashes the document id to pick a shard, so writes spread evenly across nodes. When you search, the query fans out to every shard in parallel, each shard returns its top matches, and a coordinating node merges the results. This scatter gather pattern is why search stays fast as data grows: you add nodes and shards rather than making one machine bigger.

The inverted index is the heart of it. For every term in your text, Lucene keeps a sorted list of which documents contain it and where. Searching for a word becomes a lookup in that list rather than a scan. Numeric and date fields use a different structure called BKD trees for range queries, and exact match fields use doc values stored column wise for fast sorting and aggregation.

Writes are not immediately searchable. New documents go into an in memory buffer plus a translog for durability, and Elasticsearch flushes them into a new searchable segment on a refresh interval, which defaults to one second. This is called near real time search. Segments are immutable, so deletes are just tombstones and updates are delete plus reindex, with a background merge process compacting small segments into larger ones.

Replicas serve two jobs. They protect against a node dying, and they also serve read traffic, so adding replicas increases search throughput. A write must succeed on the primary shard and is then copied to its replicas before it is acknowledged.

Reach for Elasticsearch when search relevance or text matching matters: an e commerce product catalog, a documentation search box, autocomplete, or matching across messy free text. It is also the standard tool for log and metrics analysis, where you ingest huge volumes of events and slice them with filters and aggregations in tools like Kibana.

It is the wrong tool as your only system of record. Elasticsearch has no real transactions, no joins across documents, and no foreign keys. If you need to atomically transfer money between two accounts, that belongs in Postgres or another transactional database. The common pattern is to keep the source of truth in a relational store and stream data into Elasticsearch as a denormalized search copy.

The other trade-offs are operational. Near real time means a one second lag between writing and seeing data by default. Memory and JVM heap tuning matters, and a badly chosen shard count is hard to change later because the shard count of an index is fixed at creation. Mapping changes often force a reindex. Running a healthy production cluster takes real attention to node roles, heap size, and shard sizing, usually keeping shards in the tens of gigabytes range.

Imagine a store with 50 million products. A user types "wireless noise canceling headphones" and expects results in under a tenth of a second, ranked so the most relevant products show first, with filters for brand and price on the side.

You index each product as a JSON document with fields for title, description, brand, and price. Elasticsearch builds an inverted index over the text fields and stores brand and price as keyword and numeric fields. The search query matches the text terms and scores each product using a relevance algorithm called BM25, which rewards documents where the search terms are frequent and rare across the catalog.

At the same time, a single request can run aggregations that count how many matching products fall under each brand and each price bucket, which is how the faceted filters on the left of a search page get their numbers. Because the work is split across shards on many nodes and run in parallel, the whole thing comes back in tens of milliseconds even over tens of millions of documents.