Before getting deep into the above topic, we need to understand more on the skill lift. Attaching a link to the article on skill lift. Simple in one line, there might be n tests, run those with skill
Markdown supported · ⌘↵ to post
One guardrail: treat line coverage as explanatory metadata, not the eval score. Keep the task set and scoring fixed, then compare the full skill with targeted section ablations on held-out tasks; otherwise coverage can rise while actual outcomes regress. Disclosure: I work on Orke at BOMTOBE.
Kartik N V J K
AI Developer | Making AI reliable, trustworthy & accessible to everyone | Active community contributor
Tagging character ranges like {15,28} to each eval is a neat way to make ablation granular, and I like the rule that the test must fail once those lines are removed. My worry is overlap, since two ranges can each look redundant on their own and still matter together. Does your python script ever remove pairs of ranges to catch that?