Skip to main content
HireInterviewAIHireInterviewAI
ProductAI & MLProctoringPricingSkillsBlogDevelopers
Log inBook a Demo
  1. Home
  3. Blog
  5. How to interview an AI Engineer: a concept-by-concept guide

Role Guide

How to interview an AI Engineer: a concept-by-concept guide

How to interview an AI engineer concept by concept — what temperature really changes, chunking as a quality lever, LLM-as-judge biases, RAG vs long context vs fine-tuning, LoRA/QLoRA mechanics, and the memory-bound KV cache — to measure real depth, not model-card recall.

HireInterviewAI Team·July 14, 2026·9 min read
An interviewer testing an AI engineer with sampling, RAG chunking, LLM-as-judge, adaptation-decision, LoRA, and KV-cache probes
On this page
  • Why "the RAG demo works" isn't AI-engineering knowledge
  • The concepts that separate real AI engineers
  • 1. Temperature & the sampling stack
  • 2. Chunking — a quality lever, not plumbing
  • 3. LLM-as-judge & its biases
  • 4. RAG vs long context vs fine-tuning (the adaptation decision)
  • 5. LoRA, QLoRA, and what actually trains
  • 6. The KV cache & why decode is memory-bound (the ceiling probe)
  • What an AI-engineering depth report looks like
  • Mastery vs bluffing: the pattern

On this page

  • Why "the RAG demo works" isn't AI-engineering knowledge
  • The concepts that separate real AI engineers
  • 1. Temperature & the sampling stack
  • 2. Chunking — a quality lever, not plumbing
  • 3. LLM-as-judge & its biases
  • 4. RAG vs long context vs fine-tuning (the adaptation decision)
  • 5. LoRA, QLoRA, and what actually trains
  • 6. The KV cache & why decode is memory-bound (the ceiling probe)
  • What an AI-engineering depth report looks like
  • Mastery vs bluffing: the pattern
HireInterviewAI Team

Written by

HireInterviewAI Team

AI Interview Research

The HireInterviewAI team builds adaptive AI technical interviews that probe candidates concept by concept and report exactly which topics they understand at depth.

hireinterviewai.com

HireInterviewAI

See what HireInterviewAI's per-concept interviews reveal

Stop hiring on a single fuzzy score. Run a live, adaptive AI technical interview that probes each concept to its ceiling and reports exactly which topics a candidate understands at depth.

See what HireInterviewAI's per-concept interviews revealExplore the developer API

Related reading

  • Role Guide

    How to interview an AI Agent engineer: a concept-by-concept guide

    How to interview an AI agent engineer concept by concept — the agentic loop's exit semantics, tool description contracts, the agent-vs-workflow decision, context rot and compaction, trajectory evaluation, and agent security — to measure real depth, not framework-quickstart recall.

    Read
  • Role Guide

    How to interview a FastAPI developer: a concept-by-concept guide

    How to interview a FastAPI developer concept by concept — dependency injection, Pydantic response boundaries, yield dependencies, the OAuth2/JWT flow, and the blocked event loop — to measure real depth, not tutorial recall.

    Read
  • Role Guide

    How to interview a Python developer: a concept-by-concept guide

    How to interview a Python developer concept by concept — the object model, the data model, generators, decorators, EAFP, and the GIL — to measure real depth, not syntax recall.

    Read
HireInterviewAIHireInterviewAI

AI-powered technical interviews that help engineering teams hire smarter, faster, and without bias.

Product

  • Features
  • Pricing
  • Security
  • Changelog

Company

  • About
  • Blog
  • Careers
  • Contact

Resources

  • Documentation
  • API Reference
  • Skill assessments
  • Status

Legal

  • Privacy Policy
  • Terms of Service
  • Cookie Policy
  • GDPR

© 2026 HireInterviewAI, Inc. All rights reserved.

Built for engineers who deserve better interviews

Key takeaways
  • The LLM API surface makes shallow understanding look deep — a working RAG demo tells you the tutorial was good, not whether the candidate understands retrieval quality, what temperature changes, or how an evaluation gets fooled.
  • Probe six concepts: temperature & sampling, chunking as a quality lever, LLM-as-judge biases, the RAG-vs-long-context-vs-fine-tune decision, LoRA/QLoRA mechanics, and the memory-bound KV cache. Each has a clean "understands it" vs "pattern-matched it" tell.
  • Mastery shows up as reasoning about what actually executes — what the softmax sees, which chunk strands its context, why the judge preferred the longer answer, what QLoRA actually freezes, why decode is memory-bound — not reciting that "RAG grounds the model" and "temperature adds creativity".
  • Adaptive, per-concept probing reveals each candidate's real ceiling on each concept instead of one misleading score.

To interview an AI engineer well, you have to get past the surface fluency — "RAG grounds the model," "temperature controls creativity," "LoRA makes fine-tuning cheap" — that anyone who has skimmed a launch blog can recite, and probe whether they can reason about what actually happens: what the softmax sees after temperature rescales the logits, which chunk strands the context an answer needs, why an LLM judge quietly prefers the longer response, and what a QLoRA run actually trains versus freezes. This guide gives you specific probes for six concepts and, for each, the difference between an engineer who understands LLM systems and one who has memorized their vocabulary.

Why "the RAG demo works" isn't AI-engineering knowledge

AI engineering is the easiest field to look competent in right now. The API does the hard part, a RAG chatbot is an afternoon of copy-adaptation, and the demo always works on the three questions you tried. That's precisely the trap: the stack does so much that a candidate can ship a working pipeline while holding a wrong mental model of every mechanism underneath — and those wrong models don't fail in a demo. They fail as an ungrounded answer in production, an evaluation that rewards verbosity, a fine-tune that "taught the docs" and taught nothing, or a serving bill that doubled because someone added GPUs to a memory-bound decode path. The interview can't be "build a RAG bot." It has to be "what actually happens when this runs, and why?"

The concepts that separate real AI engineers

1. Temperature & the sampling stack

Probe: "A teammate sets temperature to 0 to make the model 'deterministic' and to 1.5 for 'more creativity.' What does temperature actually change, is 0 truly deterministic, and what tends to break at 1.5?"

  • Mastery: Temperature only rescales the logits before the softmax — it changes token selection after the probabilities are computed, not how the model "thinks." Higher flattens the distribution, lower sharpens it; temp 0 routes to greedy argmax (effectively, not bit-exact — infra and floating point still wobble). Past ~1 you get an error-mimicry spiral: once an unlikely token is sampled, continuing the mistake is locally more probable than recovering. Knows logits aren't probabilities until the softmax, and that top-k / top-p / min-p sit on top of temperature rather than being the same knob.
  • Bluffing: "Temperature changes how the model reasons"; "temp 0 is fully deterministic"; "higher temperature is always more creative"; treats temperature and top-p as interchangeable dials.

2. Chunking — a quality lever, not plumbing

Probe: "Your RAG system retrieves the right documents but the answers are still vague and miss context. Someone says 'chunking is just splitting text, it's plumbing.' Push back — how is chunk strategy a quality lever?"

  • Mastery: Chunk size is a precision-vs-context tradeoff: small chunks retrieve precisely but strand the surrounding context the answer needs and raise per-query cost; large chunks carry context but dilute the embedding. Reaches for overlap, recursive/token-aware splitting to the model's limit, and contextual retrieval (prepend an LLM-written summary of the parent document to each chunk — shown to cut incorrect retrievals ~67%). Knows the query must be embedded with the exact same model and version used at ingestion, that RAG reduces but never eliminates hallucination, and that under ~200K tokens the right move is often to skip RAG and just stuff the context.
  • Bluffing: "Any chunk size works"; "irrelevant retrieved context is harmless, the model ignores it" (it doesn't — it's a loaded gun in the prompt); thinks retrieval is solved the moment the vector DB returns something.

3. LLM-as-judge & its biases

Probe: "You're using GPT-4 as a judge to pick between two model versions. It consistently prefers version B — which happens to produce longer answers and shares a base model with the judge. What could be wrong, and how do you harden the eval?"

  • Mastery: Names the biases by mechanism: verbosity bias (judges reward length), position/first bias (the order of A vs B changes the verdict — so swap positions and re-run), and self-bias / self-preference (a model favors its own outputs — the effect is real and sizeable, e.g. one model showing a ~+25% win-rate judging itself). Knows criteria like "faithfulness" are not standardized across Ragas / MLflow / LlamaIndex, that an unpinned judge model or prompt makes a useless longitudinal benchmark, and sizes the eval set to the effect being measured (detecting a 30% effect needs ~10 samples; a 3% effect needs ~1,000) instead of eyeballing a handful of outputs.
  • Bluffing: "Just ask GPT to grade it"; no notion of position, verbosity, or self bias; treats a five-output spot check as an evaluation; assumes two eval tools' "faithfulness" scores are comparable.

4. RAG vs long context vs fine-tuning (the adaptation decision)

Probe: "A stakeholder wants the model to 'know our internal docs.' Walk me through RAG vs stuffing everything into a long context vs fine-tuning — when each, and what does fine-tuning actually buy you here?"

  • Mastery: Fine-tuning changes behavior and form, not cheaply new knowledge — LoRA mostly can't teach facts. RAG is how you inject knowledge that grows and changes, with citations, and is retrieval-as-feature- engineering for the model. Long-context stuffing is fine for a small, static corpus (the sub-200K-token "skip RAG" guidance). Frames the ladder as prompt/RAG first (10 examples and a weekend), fine-tune only when form, latency, or cost demands it (1M examples and months). Knows long context doesn't kill RAG: data keeps growing, cost/latency scale per token, and models use very long contexts poorly.
  • Bluffing: "Fine-tune it on the docs so it memorizes them"; treats fine-tuning and RAG as interchangeable; "big context windows made RAG obsolete."

5. LoRA, QLoRA, and what actually trains

Probe: "In LoRA, why is one of the two adapter matrices initialized to zero? And a candidate tells you 'QLoRA trains the model in 4-bit.' Correct them."

  • Mastery: LoRA adds W + AB; B is zero-initialized and A is random, so at step zero the adapted model is identical to the base — randomly initializing B instead corrupts the starting point. QLoRA does not train 4-bit weights: the base is frozen in 4-bit NF4 and the trainable LoRA adapters live in higher precision on top, buying a big memory saving for a ~30% speed penalty. Doesn't confuse NF4 with plain INT4. Bonus: the distributive-law trick (x(W+AB) = xW + xAB) lets adapters be hot-swapped without merging into the base.
  • Bluffing: "QLoRA trains 4-bit weights"; can't say why B starts at zero; thinks LoRA cheaply teaches large new knowledge; uses NF4 and INT4 interchangeably.

6. The KV cache & why decode is memory-bound (the ceiling probe)

Probe: "Enabling the KV cache barely speeds up a 20-token completion but hugely speeds up a 2,000-token one, and your max batch size drops as context grows. Explain the cache — and why prefill and decode behave so differently."

  • Mastery: The KV cache stores each token's key/value projections so each decode step computes K/V for only the newest token instead of re-attending the whole prefix — the win scales with sequence length, which is why it's invisible at 20 tokens and huge at 2,000. Prefill is parallel (the whole prompt processed at once, compute-bound); decode is sequential, one token at a time, and memory-bandwidth-bound — you're streaming weights and cache, not saturating FLOPs. Cache size grows with tokens × layers × heads × head-dim × bytes, so it eats the memory budget and caps batch size (a 7B fp16 model can exceed 2GB of cache past ~2,048 tokens). Knows continuous batching and PagedAttention exist precisely to reclaim that memory.
  • Bluffing: "The KV cache stores tokens/logits"; can't connect cache size to the batch-size limit; thinks decode is compute-bound like prefill; "just add more GPUs."

What an AI-engineering depth report looks like

Interviewed this way, a candidate doesn't get one AI-engineering score — they get a profile that tells you exactly what to trust and what to verify in onboarding:

Concept depth report

AI Engineering · 'senior' candidate, depth view

RAG systems8/10
Evaluation & testing7/10
Prompt engineering7/10
Inference optimization5/10
Fine-tuning & alignment3/10

This candidate would sail through most "build a RAG chatbot" take-homes — their retrieval, prompting, and evaluation instincts are genuinely strong. The depth view surfaces the risk you'd otherwise find later: shallow serving mechanics and only surface knowledge of fine-tuning, exactly the gap that tries to "fine-tune the docs in" when RAG was the answer, or adds GPUs to a memory-bound decode path. That's a confident hire for an applied LLM-app role and a known risk for a model or platform role — not a guess. A single "AI engineering: 6.5/10" would have hidden both the strength and the risk. (Here's why one number is useless.)

Mastery vs bluffing: the pattern

Across all six concepts the tell is the same. Bluffers describe what things are ("RAG grounds the model," "temperature adds creativity," "LoRA makes fine-tuning cheap"). Engineers who actually understand LLM systems describe what happens and why — the logits rescaled before the softmax, the chunk that strands its context, the judge that rewarded length, the base frozen in 4-bit while the adapters train, the decode step bound by memory bandwidth — and how they'd catch it. Mechanism reasoning is the signal. Vocabulary is noise.

That's also why a fixed question can't carry the interview: a strong engineer clears your first probe instantly and you learn nothing about their ceiling. You have to keep raising difficulty — which is precisely what an adaptive interview does automatically, finding each candidate's real boundary on each concept instead of stopping at the first correct answer.

HireInterviewAI probes concepts like these adaptively and reports the per-concept depth above instead of a pass/fail. See the AI Engineering assessment, the sibling guide to interviewing a Python developer, or the framework for how to evaluate developer skills. If the role ships autonomous, tool-using systems, that is a separate skill — see the AI Agents assessment.

Frequently asked questions

What is the single best question to interview an AI engineer?
The adaptation decision — RAG vs long context vs fine-tuning for "make the model know our docs." It instantly separates engineers who know fine-tuning changes behavior rather than cheaply adding knowledge, and that RAG is for knowledge that grows, from those who reach for "fine-tune it on the docs." It opens naturally into chunking, evaluation, and cost. A close second is the LLM-as-judge biases question, which exposes whether someone has actually shipped an evaluation.
Should I interview AI engineering separately from Python?
Yes, as a skill track on top of the language track. Python probes the data model, generators, typing, and the GIL; AI-engineering probes LLM-system judgment — retrieval quality, evaluation design, sampling and decoding, defensive prompting, and serving mechanics. A candidate can write clean Python and still hold a wrong model of what temperature changes or why a judge prefers longer answers, and the hire decision needs both signals.
Do I need to test fine-tuning and inference for every role?
No — weight the concepts to the role. Applied LLM-app roles are decided by RAG, prompt engineering, and evaluation; fine-tuning, dataset engineering, and inference optimization matter most for model and platform roles at senior and staff level, and even there they are best probed as judgment ("what does QLoRA freeze, when does LoRA earn its cost") rather than a live training run. A strong applied engineer who has never trained a model can still be an excellent hire.
How is this different from interviewing for AI Agents?
AI Agents is a separate, adjacent skill that presumes AI-engineering fluency. AI engineering owns model internals, RAG, evaluation, prompting, fine-tuning, and serving; agents add the agentic loop, tool and function contracts, agent memory, multi-agent orchestration, and trajectory evaluation — a question that pure prompting and RAG knowledge can answer isn't really an agent question. If the role builds autonomous, tool-using systems, run the separate [AI Agents assessment](/skills/ai-agents/) alongside this one.

Interview for what actually executes when the pipeline runs, not what they can recite — and report it per concept so you know exactly what you're hiring. See how HireInterviewAI does it on the AI Engineering assessment, or check pricing.