How to Interview Engineers on AI Agent Proficiency

Why the Hiring Signal Has Moved

In 2022, a technical interview measured something real: could this person write correct code without help? That question made sense when the job was to write correct code without help.

In 2026, that is not the job. The job is to steer AI agents toward a correct result, verify what they produce, reject what's wrong, and ship code you can defend line by line. Closed-book LeetCode selects for the wrong skill while actively screening out the candidate's actual workflow.

The companies that figured this out first — Meta, Google, Canva, Shopify, Sierra, Rippling — restructured at least one interview round around AI-assisted work. They didn't do it to be generous to candidates. They did it because the alternative is hiring the wrong person.

What to Actually Measure

The evaluation target shifts entirely when you allow AI tools. You're no longer asking "can this person write code?" You're asking: can this person produce a trustworthy result using AI, and do they know when to trust it?

This breaks into five observable behaviors:

1. Scoping before prompting. Strong candidates read the codebase, understand the task, and write a mental (or literal) spec before they type a single prompt. Weak candidates paste the problem statement directly into Claude Code and accept whatever comes back.
2. Prompt precision. The quality of the prompt predicts the quality of the output. A precise prompt with constraints, context, and explicit scope gets a usable result in one or two turns. A vague prompt produces vague output that requires 8 more turns to become usable — and usually still has bugs.
3. Verification discipline. Every model produces wrong output sometimes. The question is whether the candidate notices. Do they read the diff? Do they run a test? Do they ask "why did it choose this approach?" Strong candidates write a failing test before fixing anything. They treat model confidence as a hypothesis, not a fact.
4. Orchestration judgment. Can they decompose a task correctly — breaking it into atomic subtasks the agent can execute well? Do they know when to keep the agent in the loop versus when to take the wheel? Can they fan out parallel agents when appropriate and stay sequential when the task has dependencies?
5. Ownership. This is the final gate. Remove the AI. Ask them to explain any line in the diff. A strong candidate can. A weak one says "the AI wrote that part." Ownership doesn't mean writing every line — it means understanding every line you're merging.

The Scoring Rubric (4 Dimensions)

This is the rubric Altor uses when conducting AI agent interviews on behalf of engineering teams. It's adapted from rubrics used at Meta and Canva's AI-enabled interview rounds, refined for evaluating tool-specific fluency.

Three Interview Formats That Work

Red Flags and Green Flags

Token Efficiency as a Hiring Signal

This is the dimension that nobody else is measuring — and it's one of the strongest predictors of real-world productivity.

Token efficiency means: how many prompts and messages does a candidate need to accomplish a defined task? Every additional turn is a signal. A candidate who gets to a working, verified result in 2 prompts thinks more clearly than one who needs 12. The difference is not typing speed. It's cognitive clarity about what they want before they ask for it.

You measure it by reviewing the AI session transcript after the interview. Claude Code keeps session logs. Cursor Composer shows the full history. Copilot Chat has a session view. Three metrics matter:

• Turns to first usable output: Fewer is better. A precise prompt gets usable output in one turn. A vague prompt requires 3–5 clarifying back-and-forth turns before the model produces something worth reading.
• Rejection rate: What percentage of model suggestions did they reject, and why? Zero rejections is a red flag — it means they accepted everything. Rejecting one in four suggestions is healthy. Always rejecting suggests they don't know how to prompt well. The ratio matters less than the reason for each rejection.
• Context window management: Did they provide sufficient context upfront, or did they start a new conversation mid-task because the model "forgot" what they were doing? Strong candidates front-load context. Weak candidates restart constantly.

How to Review AI Session Transcripts

The session transcript is the most underutilized interview artifact. Here's how to read it:

What to look for in Claude Code transcripts

• Plan mode usage: Did they hit /plan before implementing? Using plan mode before touching code is the single strongest signal of disciplined AI-native engineering. It shows they think about scope before execution.
• Tool call quality: Did they use targeted tools (Read specific files, Grep for patterns) or did they ask the agent to "look at everything"? Targeted tool use means lower token waste and faster results.
• Rejection messages: When the agent proposed something wrong, what did the candidate say? "No, that's wrong because the race condition occurs at the point of write, not read" is a 5/5 response. "Try again" is a 2/5.
• Checkpoint behavior: Did they stop and review at natural breakpoints, or did they let the agent run continuously and review at the end? Mid-task verification is better — errors compound when caught late.

What to look for in Cursor transcripts

• Composer vs. Chat usage: Composer handles multi-file changes. Chat handles single-file edits and questions. Appropriate tool selection for the task type is a signal.
• Diff acceptance rate: Accepting all diffs without inspection is a red flag. A strong candidate inspects, selects, and sometimes modifies before accepting.
• Prompt length distribution: A few long, precise prompts is better than many short, vague ones. The shape of the conversation tells you how the candidate thinks.

What Meta, Google, Shopify, and Sierra Actually Do

The pattern is consistent: allow AI tools, observe the interaction, score judgment not output. No company in this list is running a closed-book whiteboard in 2026. The ones still running LeetCode are having retention problems — they hired the 2022 shape of engineer at 2026 prices and are wondering why productivity is flat.

The CLAUDE.md Signal

One of the sharpest pre-interview signals has nothing to do with the interview itself. Ask candidates to share their personal CLAUDE.md or AGENTS.md file. These files encode how an engineer works with AI agents — their personal conventions, project rules, and operational expectations for AI collaborators. A well-structured CLAUDE.md is a higher-signal artifact than a portfolio, a degree, or any certification. A missing one, or one that says "be helpful", tells you everything you need to know.

Frequently Asked Questions

Should you allow AI tools in every technical interview?

Yes, for most engineering roles in 2026. The exception is when you specifically need to see unaided reasoning — for example, a system design discussion where you want to probe first-principles thinking without scaffolding. But for any role where the daily workflow involves shipping with AI in the loop, testing without AI tests a fiction. Reserve AI-free segments for specific moments, not entire rounds.

How do you prevent cheating if AI is allowed?

Live screen-sharing with mandatory camera. The interview isn't about whether the candidate uses AI — it's about how they use it. You're watching the session in real time. After the session, you ask them to explain every decision. Cheating via AI requires real-time help from a human co-pilot; that's hard to hide under live questioning. The ownership walkthrough ("explain this line to me") breaks hidden-assistance strategies quickly.

What role seniority does this format work for?

All of them, with different calibration. For L3/L4, you're checking baseline verification discipline and prompt fundamentals. For L5+, you're checking orchestration judgment, architectural decisions made while directing agents, and the ability to scope and decompose large problems. Senior candidates should be able to run four or five parallel agent tasks while reviewing the output of a fifth — and explain every decision in each.

How long does an AI agent interview take?

A full format takes 90 minutes: 60 minutes live session + 30 minutes transcript review and defense. A streamlined version is 60 minutes. Unlike traditional interviews, you get an artifact (the AI session transcript) that you can study after — which means you don't have to make all your evaluation decisions in the room.

What's the biggest mistake companies make in AI agent interviewing?

Scoring on output quality instead of process quality. Two candidates can produce identical-looking diffs. The one who got there in 2 precise prompts with a failing test catching a model error is a completely different engineer than the one who got there in 14 turns of vague iteration with no tests and a diff they can't explain. The transcript tells the story. The diff alone does not.

Related reading: AI Agent Interview Service — how Altor runs it · Download the scoring rubric · AI strategy consulting vs. AI implementation