Earlybird — Twitter Search Engine

Earlybird — Twitter Search Engine

How Twitter built Earlybird, its custom real-time distributed search engine — from MySQL LIKE queries to inverted indexes, Lucene, and a distributed Lucene index tuned for tweet search at scale.

Search Engines

Tweet 1: "I love Java"

Tweet 2: "Java virtual threads are interesting"

Tweet 3: "Learning Python today"

Tweet 4: "System design interview tomorrow"

Simple definition A search engine is a system that takes a user's query and finds the most relevant data from a large collection of data.

Functional Requirements

Non-Functional Requirements

1.Search Tweet by text 2. Return relevant results 3. support filters 4. Pagination

1.Low Latency 2.High Availability 3.Scalability 4.Freshness : example: if post at 10:01 -> available at 10:01 5.Relevance. Example: "iphone"

Approach - 1 : MySQL + LIKE

tweets id | text ---------------------------- 1 | I love Java 2 | Learning Python 3 | Java is fast

WHERE text LIKE '%java%';

No new system required

Fine for small amount of data

Cons For large data, we may scan a huge number of rows.

Approach 2 — Normal Database Index

CREATE INDEX idx_text ON tweets(text);

user_id = 123 created_at > some_date name starts with "java"

A normal B-Tree index is not designed for this type of full-text search.

Database already supports it

Very fast for exact values/ranges/prefixes

No separate infrastructure

Not good for arbitrary words inside large text