Read this once. Then sit the case. You do not need to train a model. You need to talk about one without bluffing.
Time: about 12 minutes. Then: Clearclaim Mutual, a 20-minute AI case.
The 90-second map
Five words get mixed in every interview. Keep them separate.
| Word | In one sentence | Coffee-shop version |
|---|---|---|
| AI | Software that does a task people used to need judgment for | The umbrella |
| Machine learning | The software learned from examples, nobody wrote the full rulebook | Teaching by flashcards, not by a manual |
| Generative AI | The output is new text, images, code, or audio | It writes, it does not only classify |
| LLM | A generative model that predicts the next word, at huge scale | A very well-read intern who has never worked at the client |
| Agent | An LLM plus tools, memory, and a loop so it can do multi-step work | An intern you allow to open the filing cabinet and send the email |
If you remember nothing else: the model is the intern. The harness is the manager. Interviewers at McKinsey, BCG, and Bain now listen for whether you know the difference.
How a model learns
A machine-learning model is a function with a lot of knobs (weights).
- You show it examples: “this claim is fraud / this claim is not.”
- It guesses.
- You tell it how wrong it was.
- It nudges the knobs.
- Repeat until the guesses are useful enough.
That loop is training. Nobody sat down and wrote “if the payout is > $8,000 and the garage is in this ZIP, flag it.” The model found patterns in the history. That is the power and the risk. It will also find patterns you did not want: a biased ZIP code, a leaked target, last year’s process, not the real world.
In a case, say: “We are not buying magic. We are buying a function fitted on this data, for this decision, with this error cost.”
Training vs inference
- Training is cooking the recipe. Slow, expensive, needs a pile of labelled examples (or a pile of text, for an LLM).
- Inference is using the recipe. Fast, you pay per call, this is what the client runs in production.
A partner who says “just retrain it every night” is usually wrong on cost, data freshness, and risk. Most enterprise wins are better inference (how you ask, what you retrieve, what a human checks) not a new training run.
Three kinds of machine learning
| Kind | What you give it | What you get | Client example |
|---|---|---|---|
| Supervised | Examples with answers | A prediction or a class | Will this customer churn? Is this claim high-risk? |
| Unsupervised | Examples with no answers | Groups or structure | Segment stores that behave alike |
| Reinforcement | A reward for good actions | A policy for what to do next | A bidding agent; a robot that scores the game, not the slide |
Most generalist cases are supervised or generative. If you hear “the model will learn from its mistakes in production,” ask how the reward is defined. A claims agent rewarded for “speed” will pay junk claims faster.
What “good” means (accuracy is a trap)
Accuracy is “how often was it right?” That is a terrible score when 98% of claims are fine and 2% are fraud. A model that always says “pay it” is 98% accurate and useless.
Use the language interviewers want:
- False positive: we flagged a good claim as bad. Cost = delay, customer anger, extra adjuster minutes.
- False negative: we missed a bad claim. Cost = leakage, fraud, a wrong payout.
- Precision: of the ones we flagged, how many were actually bad?
- Recall: of the actually-bad ones, how many did we catch?
- Latency and cost: seconds and $ per call. A brilliant model that takes 40 seconds will not sit in a call center.
In a case, say: “I need the cost of each error, not a leaderboard score. Then I decide how much human review is worth.”
Generative AI and LLMs, with no mysticism
A large language model does one thing: guess the next token (a shard of a word) given the tokens so far. Do that a thousand times and you get a memo. It is not a database. It is not a rules engine. It has read a huge slice of the public internet plus licensed books, code, and (if the vendor is sloppy) things it should not have.
That is why it:
- Sounds confident when it is wrong (hallucination)
- Is great at drafts, summaries, and boilerplate
- Is dangerous when the answer must be this policy, this number, this date
Context window = how much you can stuff in the prompt (the question + the documents). Bigger windows help. They do not replace a filing system.
Four ways to make an LLM useful
Cheapest first. Do not skip steps because the vendor demo skipped them.
| Move | What it is | When it wins | When it loses |
|---|---|---|---|
| Prompting | Better instructions, examples, a role, an output format | Style, structure, a first draft | Facts that are not in the prompt |
| RAG | Retrieval-augmented generation: look up the client’s docs, then write | Policies, contracts, knowledge bases that change | If search is bad, the answer is bad with footnotes |
| Fine-tuning | Extra training on your examples so the model sounds like you | Tone, format, a narrow task with stable rules | Last week’s policy change; anything that must be cited |
| Tools / agents | The model can call systems: search, calculator, CRM, email | Multi-step work with a clear stop rule | Unbounded loops, write-access, no evals |
RAG vs fine-tuning is the question that now shows up in generalist cases. The consulting answer is almost always: RAG for facts that change; fine-tune only if you have a stable task and a pile of graded examples. Fine-tuning does not “teach the model your SharePoint.” RAG does.
In a case, say: “I would not fine-tune first. I would retrieve the policy, generate a draft, and keep a human on the payout button until the error cost is proven.”
Agents and harnesses
A chatbot answers one question and waits.
An agent can plan a few steps: look up the policy, pull the claim history, draft a payout, ask a checker, stop.
The harness is everything around the model that makes that safe:
- Tools it is allowed to call (and tools it is not)
- Memory (what it may remember about this customer)
- Evals (a frozen set of real cases you score every time you change a prompt)
- Guardrails (PII, jurisdiction, “never pay above $X without a human”)
- Human checkpoints (where a person must click)
- Logging (so you can explain the decision to a regulator)
Vendors sell “agentic AI.” Partners should buy a harness with an intern inside, not an intern with admin rights.
In a case, say: “The model is cheap. The operating system around it is the project: permissions, evals, and who gets fired when it is wrong.”
What is changing at the frontier (enough to sound current)
You do not need a research blog. You need four shifts that change a recommendation.
- Reasoning models. Some models spend more compute thinking before they talk. Better at math and multi-step plans. Slower and dearer per call. Use them for the hard 10%, not the 10,000 daily emails.
- Tool-using agents. The model can search, calculate, and call APIs. Value shows up when the workflow is real, not when the demo is pretty.
- Evals over vibes. Serious teams keep a scored set of tasks. If you cannot say how you will know the pilot worked on Monday, you do not have a pilot.
- Smaller, specialized models. Not every task needs the biggest model. A cheap classifier plus a bigger model on exceptions is often the architecture.
Multimodal (images, voice) is real. In a case it is a data question: do we have the photos, are they labelled, who owns consent?
How to open an AI case
Steal this tree. Customize the leaves.
- Decision and metric. What does the CEO need to say yes to, in what units, by when?
- Value pool. Time saved, leakage avoided, conversion, risk reduced. Size it before you name a model.
- Data and workflow. What exists, what is labelled, who does the work today, where would the model sit?
- Method. Prompt, RAG, fine-tune, classifier, agent — cheapest that works.
- Error economics and harness. Cost of each miss, human in the loop, evals, kill criteria.
- Pilot then scale. One workflow, one metric, a date you will stop if it is not working.
The common fail is naming GPT and sitting down. The common pass is a number, a method, and a kill switch.
What the firms are actually testing
This is not only QuantumBlack, BCG X, or Bain Vector. Generalist interviewers now drop an AI layer into an ordinary profitability or operations case and watch whether you:
- Separate demo accuracy from error cost
- Know RAG vs fine-tune without a TED talk
- Refuse write-access agents until the harness exists
- Still do the case: structure, math, so-what, a close a partner could take to dinner
Technical specialist loops (QuantumBlack, BCG X engineering, Bain AIS) go deeper: metrics, leakage, training data, monitoring. The data science case guide is for that path. This module is the generalist bar: how it works, in English, with a $ attached.
When you can say the intern/manager line, pick RAG for changing facts, and size the cost of a wrong payout, you are ready. Sit Clearclaim Mutual.
If you want the same ideas aimed at a search query: RAG vs fine-tuning, agent vs chatbot, LLM interview questions, or the McKinsey / BCG / Bain versions.