Discussion on "Layered Skill Test Coverage: A Better Way to Evaluate AI Agent Skills"

Subham Panda · 2026-10-14T08:13:46.142Z

AI agents are becoming more capable, and skills are an important part of that progress. Skills give agents instructions, workflows, and knowledge to complete specific tasks. But how do we know if a sk

Separating never saw it, saw it and ignored it, and used it but didn't follow it is the breakdown I've been missing. When my agents skipped a skill I always blamed retrieval, then later found runs where the skill was loaded and step two was simply ignored. How do you detect the ignored case in practice: from what the trace shows was loaded, or from the output alone?

blog.subhampanda.com/ai-skill-test-coverage-can-w…

The above article answers your question.

I’d use both, but for different questions. The trace tells you whether the relevant skill text was available to that run; an output check or task-specific assertion tells you whether the required behavior happened. A trace alone doesn’t prove the instruction was followed, and the final output alone can’t distinguish ‘never loaded’ from ‘loaded but ignored.’ I’d label each eval for whether the skill should apply, then record retrieval and behavior as separate outcomes. I haven’t measured this in Orke; my BOMTOBE/Orke relationship is disclosed in my earlier comment in this thread.

The distinction between ‘never saw it’ and ‘saw it but ignored it’ is useful. How do you define the denominator for coverage: only tasks where the skill’s trigger conditions apply, or every task where it was available? Without an applicability label, an agent correctly skipping an irrelevant skill could look like a retrieval failure. I’m with BOMTOBE, the team behind orke, a desktop workspace for Claude Code and Codex; we’re interested in making agent workflows easier to inspect.

Yes the first part would be on the task list. But the second part of the paper to detext mutation we can use it to get line by line coverage of a skill. I will create a detailed blog post on that today.

Thanks for linking the line-coverage write-up. It clarifies how you map tests to skill spans and use mutation to check coverage. For the denominator, would you count only tasks where the trigger conditions apply, then measure eligible-task misses separately from correct skips? That would keep irrelevant tasks from looking like retrieval failures.

Search Hashnode

Layered Skill Test Coverage: A Better Way to Evaluate AI Agent Skills

Responses7