Three teams plugged MCP servers into our agent. Security asked who checks the permissions.Our internal agent started small, with one MCP server my team wrote. Within a quarter, two more teams had connected their own servers to it. One exposed billing lookups and one could open tickets in a3d ago 路 5 min readkartiknvjk.hashnode.dev
My agent said the refund went through. The ticket went to the one team that could not fix it.My refund agent told a customer the refund was processed. It wasn't. The payment tool had timed out, the harness retried twice, the third attempt came back with an empty body, and the output parser tu1MOct 13 路 5 min readkartiknvjk.hashnode.dev
I captured a hundred agent failures a week and fixed three. The bottleneck was curation.I captured a hundred agent failures a week and fixed three. The bottleneck was curation. My agent had the kind of observability setup I used to brag about. Every model call, every tool call, every retOct 8 路 5 min readkartiknvjk.hashnode.dev
Eval the Signature, Not the Program: How I Fixed My DSPy EvaluationLast month I shipped a DSPy RAG pipeline that looked done. I compiled it with MIPRO, the hold-out numbers were solid, CI was green. Two days into real traffic the quality complaints started. RetrievalOct 6 路 5 min readkartiknvjk.hashnode.dev
My CI evals were green. A regression still paged me at 3 AM.The eval that finally caught a bad regression for me was not the one in CI. CI had run twelve hours earlier, against a frozen test set, and passed. The regression was sitting in live production traffiSep 17 路 4 min readkartiknvjk.hashnode.dev