Forward Deployed Engineer interview question · Mini system-design cases · staff level
One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy?
Free to read in full. This is 1 of the 100 questions in the Forward Deployed Engineer interview pack, with 127 diagrams. Get the pack, ₹999 · $20
Why it is asked
Large enterprises want one platform but strict separation between units. The interviewer checks tenant identity, data isolation tiers, encryption, fair quotas and how you prove isolation.
The answer
I would run one shared platform with strong logical isolation: a tenant ID from the identity provider on every request, data separated per tenant, a key per tenant, and quotas so no unit starves the others.
Identity: users sign in through Entra ID or Okta. The token carries the business unit as a claim (a field inside the token), and the gateway turns it into a tenant ID. Services never trust a tenant ID sent by the client.
Data isolation, in tiers:
- Default: shared Postgres with a tenant_id column and row-level security (the database itself hides other tenants' rows). The app connects as a role that does not own the tables, so the policy cannot be skipped.
- Sensitive units, such as HR or legal: their own schema or database.
- Vector search: a separate namespace (a walled-off section of the index) per tenant. Inside a tenant, per-user document permissions still use metadata filters, as in a permission-aware RAG design. The namespace is an extra wall between business units, so one missing filter cannot leak across tenants.
Encryption: one KMS key per tenant. Offboarding disables the key at once and schedules it for deletion after the agreed retention period, which makes the remaining data unreadable.
Quotas: per-tenant rate limits and token budgets at the gateway. If the shared LLM limit is 2,000,000 tokens per minute and we keep 20% headroom, 2,000,000 × 0.8 = 1,600,000 remain, so each of the 50 units gets a guaranteed 1,600,000 / 50 = 32,000 tokens per minute and can burst into spare capacity.
Operations: per-tenant metering and audit logs, and release tests that try to read tenant B's data as tenant A.
What the interviewer listens for
- Tenant ID from the token, never from the client.
- Isolation tiers: RLS, own schema or database, per-tenant vector namespace.
- Per-tenant KMS keys.
- Guaranteed quota plus burst, computed.
- Cross-tenant access tests on every release.
Follow-ups the interviewer may ask
- Why not one deployment per unit? Hint: 50 stacks to patch and run; reserve dedicated stacks for the few units that need them.
- How do you handle a user in two units? Hint: the token lists both; the user picks one per session, and every request carries exactly one.
- How do you bill units fairly? Hint: meter tokens, storage and requests per tenant and report them monthly.
Common mistakes
- Taking the tenant ID from a request header the client controls.
- One shared vector index across business units, walled only by a metadata filter.
- No per-tenant quota, so one batch job starves 49 units.
Written by Roni Das, who builds distributed systems and AI systems for a living.
One question is not enough for the interview.
This is 1 of the 100 questions in the Forward Deployed Engineer pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (127 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- A customer says: "We want AI for our support team." What do you do in the first meeting? (mid)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (junior)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (mid)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (senior)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)