Forward Deployed Engineer interview question · Integrating with messy enterprise data · mid level
Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry?
Free to read in full. This is 1 of the 100 questions in the Forward Deployed Engineer interview pack, with 127 diagrams. Get the pack, ₹999 · $20
Why it is asked
Retries are unavoidable over enterprise networks. The interviewer checks that you know a timeout is not a failure and how idempotency fixes duplicates.
The answer
Make every write idempotent, which means doing it twice has the same effect as doing it once. A timeout does not mean the first request failed; often it succeeded and only the reply was lost.
Three ways, best first:
1. Idempotency key. Before the first attempt, create one key per logical operation, ideally derived from the source event ID, and send the same key on every retry, usually in an Idempotency-Key header (Stripe's API is the well-known example). The server stores key and result; a repeat returns the stored result instead of creating a new ticket. The key must not change between retries.
2. Upsert on a natural key. If we write to a database, put a unique constraint on the external ID and use INSERT ... ON CONFLICT in Postgres or MERGE in Snowflake. A repeat updates the same row.
3. If the target API supports neither, store our source ID in a custom field on the ticket, look it up before creating, and run a single writer per ticket so two workers cannot race between the lookup and the create. Keep our own ledger of what we wrote.
This is the usual pairing: delivery is at-least-once (messages can repeat), and the consumer is idempotent, so the outcome is as if each change happened exactly once.
I also add a guard against old data: only overwrite when the incoming updated_at is newer, so a late retry cannot replace a fresher status.
What the interviewer listens for
- A timeout is unknown outcome, not failure.
- Idempotency key fixed before the first attempt and reused on every retry.
- Upsert on a unique external ID (ON CONFLICT / MERGE).
- At-least-once delivery plus idempotent consumer gives exactly-once effect.
- Guard against older data overwriting newer data.
Follow-ups the interviewer may ask
- How long should the server keep idempotency keys? Hint: longer than your longest retry window, often 24 hours or more.
- What if the same key arrives with a different body? Hint: reject it with an error; it signals a client bug.
- How do you clean up the duplicates already created? Hint: group by source ID, keep the earliest, merge or close the rest with the customer's approval.
Common mistakes
- Generating a new idempotency key on each retry.
- Turning retries off, which turns duplicates into data loss.
- Search-then-create with several parallel workers, which still races.
Written by Roni Das, who builds distributed systems and AI systems for a living.
One question is not enough for the interview.
This is 1 of the 100 questions in the Forward Deployed Engineer pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (127 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- A customer says: "We want AI for our support team." What do you do in the first meeting? (mid)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (junior)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (senior)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (staff)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)