# LLMInterview synthetic candidate-risk benchmark

Version 1.0 · https://llminterview.com/benchmark

All fixtures are synthetic. No real candidate, recruiter, resume or company interview process is represented, and no formal academic validation is claimed.

## What it tests

- detection of a planted risk in a candidate work history
- interpretation of the risk into interviewer probes
- propagation of the risk into preparation output
- confidence calibration: confirmed versus inferred versus unknown

It does **not** test prediction of any company's real interview process.

## Files

- `https://llminterview.com/benchmark/fixtures.json` — machine-readable fixtures
- `https://llminterview.com/benchmark/README.md` — this file

## Fixture shape

Each fixture plants one subtle weakness in a fictional work history and states
the ground truth: `planted_risk`, `expected_detection_terms`,
`expected_probes`, `expected_propagation`, and `confidence_expectations`
separating rounds a fictional recruiter confirmed from rounds that may only be
inferred or must stay unknown.

A system passes a fixture when it (1) surfaces the planted risk using the
expected detection terms, (2) turns it into probes an interviewer could ask,
(3) changes its preparation output because of it, and (4) does not promote an
inferred or unknown round to confirmed.

## Fixtures

### spellbrush-late-creative-alignment

- Risk class: Late creative alignment
- Role (synthetic): LLM Engineer, generative game content (synthetic evaluation against a public job description)
- Buried line: Rewrote the character-dialogue prompt architecture end to end and shipped it, then walked the art and narrative leads through the new behaviour once it was stable.
- Planted risk: Major prompt architecture changes are completed independently and creative partners are involved only after the behaviour is already set.
- Detection terms: creative, late, alignment
- Confirmed rounds: Technical screen, Project deep dive
- Must not be confirmed: Creative collaboration probe (likely), Behavioural round (likely), Design workshop with creative leads (unknown)

### metric-without-measurement

- Risk class: Impressive metric, weak measurement substantiation
- Role (synthetic): Applied AI Engineer, retrieval systems (synthetic)
- Buried line: Improved answer accuracy by 40% after rebuilding the retrieval pipeline.
- Planted risk: A headline number with no stated baseline, evaluation set, or measurement method behind it.
- Detection terms: baseline, measurement, evaluation
- Confirmed rounds: AI system design, Hiring manager conversation
- Must not be confirmed: Evaluation and reliability probe (likely), Standalone evals round (unknown)

### depth-without-production

- Risk class: Strong technical depth, unclear production deployment
- Role (synthetic): LLM Engineer, platform (synthetic)
- Buried line: Built and benchmarked three fine-tuned summarisation models; the internal pilot ran for two months.
- Planted risk: Deep modelling work with no evidence that anything reached and survived production traffic.
- Detection terms: production, pilot, rollback
- Confirmed rounds: Coding round, System design round
- Must not be confirmed: Production operations probe (likely), On-call / incident round (unknown)

### strong-resume-unknown-coding-round

- Risk class: Strong resume, unknown coding round
- Role (synthetic): Senior Machine Learning Engineer (synthetic)
- Buried line: Eight years shipping ML systems; most recent coding was review and design, not implementation.
- Planted risk: Seniority has moved the candidate away from hands-on implementation while the coding format stays unknown.
- Detection terms: coding, unknown, hands-on
- Confirmed rounds: Hiring manager screen
- Must not be confirmed: Practical coding round (unknown), Code reading exercise (likely)

## Usage

You may use these synthetic fixtures to test interview-prep or candidate-analysis systems. Attribution to LLMInterview.com is appreciated but not required.