AI for oil and gas: how predictive maintenance systems are built
A pump at a remote oil field that fails without warning can stop production for days. Predictive maintenance tries to catch the failure early, from the way the pump vibrates and heats up in the hours before it breaks.
People often think the model is the clever part. In practice most of the work is the data platform underneath it, and that is a system design problem you can learn. This guide walks through the whole pipeline.
How does AI predict that an oil pump will fail?
Sensors on the pump measure vibration and temperature many times a minute. A data pipeline collects those readings, groups them into time windows for each pump, and turns them into features. A model trained on past failures scores each pump, and a high score alerts an engineer before the pump stops. Most of the engineering is in the pipeline: readings from remote sites arrive late, out of order and sometimes twice.
Edge first
A gateway at the site stores readings while the network link is down.
Keep the order
Kafka keyed by pump id keeps each pump's readings in one partition.
Measured time
Group readings by when they were measured, not when they arrived.
Then the model
A model can only be as good as the windows it is given.
If this helped, the full course goes much deeper.
In it, you design real systems step by step, from queues and stream processing to partitioning, storage and failover, with a diagram for every step.
From the Gulf you pay in US dollars, by card or PayPal.
Why this matters in the Gulf
The region's largest energy companies already measure what AI earns them, and maintenance is a big part of it.
Aramco reports $2.6 billion of AI-driven value in 2025, with 442 AI applications mapped and more than 200 already deployed.
Source: Aramco: Building the AI future, read 19 November 2026Aramco says it has realised more than $6 billion in business value from AI, including less maintenance downtime and faster inspections, and has trained more than 6,000 of its experts in AI.
Source: Aramco newsroom, 27 January 2026Aramco unveiled its own generative AI model, Aramco Metabrain AI, at LEAP in Riyadh in March 2024, with 250 billion parameters trained on data from more than 90 years of company history.
Source: Offshore Technology, 7 March 2024ADNOC generated $500 million of value in 2023 from more than 30 AI tools across its business.
Source: The National, 5 March 2024ADNOC and AIQ launched ENERGYai, an agentic AI system trained on 80 years of ADNOC data, built with G42 and Microsoft.
Source: World Oil, 4 November 2024
The pipeline, step by step
First, sensors on each pump publish readings over MQTT, a lightweight messaging protocol made for devices on weak networks, to an edge gateway at the site.
Second, the gateway forwards readings to the data centre. Remote links drop, so the gateway buffers readings on local disk and sends them when the link returns. This is called store and forward, and without it every outage leaves a hole in the data.
Third, readings land in Kafka, keyed by pump id. Kafka keeps the order of messages within one partition, and the same key always goes to the same partition, so each pump's readings stay in the order the gateway sent them.
Fourth, a stream processor such as Apache Flink groups each pump's readings into time windows and turns them into features, for example the average vibration over the last ten minutes. Fifth, a model scores each pump, and a high score alerts a field engineer. Kafka also writes every raw reading to a time-series database for dashboards and for training the next model.

The part most designs get wrong: late readings
When a site's link comes back after an hour, the gateway sends an hour of old readings at once, mixed in with live ones. If your windows count readings by the time they arrived, that hour of data lands in the wrong window, and the model sees a pump that was quiet for an hour and then suddenly very busy.
The fix is to window by event time, the timestamp the sensor wrote when it took the reading, and to use a watermark, a rule that says how long to wait for late readings before a window is closed. Add a deduplication step too, because MQTT's common delivery setting promises at least once, so a reading can arrive twice.
The chart below, from our course, shows the difference on forty events: counted by arrival, two events fall in a window in which nothing actually happened.

If this helped, the full course goes much deeper.
In it, you design real systems step by step, from queues and stream processing to partitioning, storage and failover, with a diagram for every step.
From the Gulf you pay in US dollars, by card or PayPal.
Where the readings live
Sensor data is time-series data: many small readings, always appended, usually read by time range. A time-series database stores it compactly, partitions it by time, and makes questions like the average temperature per hour for one pump last week fast.
Keep raw readings for a fixed period, then keep only hourly or daily summaries for the long term. Decide that period with the people who train the models, because they need enough history around every past failure.
Practise it as an interview question
Try this one: design a predictive maintenance system for 10,000 pumps across several remote sites. Start with the numbers. If each pump has two sensors that send one reading a second, that is 20,000 readings a second. At about 100 bytes a reading that is 2 MB a second, or about 173 GB a day of raw data.
Those numbers tell you a lot. One Kafka cluster handles that rate easily, so the real risks are the remote links, the late data and the storage bill. Say so out loud, then design for them: store and forward at the edge, event-time windows with a watermark, deduplication, and a retention plan.
Interviewers care less about the tool names and more about whether you noticed the late readings. Mention them before you are asked.
Questions people ask
Do I need to know machine learning to work on predictive maintenance?
Not to start. Most of the engineering is the data platform: collecting, ordering, storing and serving the readings. The System Design course teaches that part. Add the AI Engineering course later if you want to build the models too.
Why Kafka and not a simple queue?
Kafka keeps messages in order within a partition and keeps them for a set time, so several systems can read the same readings independently and a new model can be trained by replaying history. A simple queue usually deletes a message once it is read.
What is a digital twin?
A digital twin is a live software model of a physical asset, such as a pump or a whole plant, kept up to date by its sensor data. It sits on the same kind of pipeline described on this page.
Every source on this page was read on 19 November 2026. Plans, rules and numbers change, so check the linked page before you rely on one.
If this helped, the full course goes much deeper.
In it, you design real systems step by step, from queues and stream processing to partitioning, storage and failover, with a diagram for every step.
From the Gulf you pay in US dollars, by card or PayPal.