An agent that worked, until it didn’t
Northpeak sells billing software to 40,000 small businesses. In early 2025 the team launched a support agent that could look up charges, explain invoices and issue refunds up to $500. Within a month it resolved 58% of billing tickets without a human.
Then refund complaints started climbing. Customers reported refunds that never arrived, or arrived twice. The agent’s logs showed successful API calls. The support team had screenshots of unhappy customers. Nobody could connect the two.
Finding the failure in a single afternoon
Priya Raman’s team installed the Sightline SDK on a Tuesday. By Wednesday they had 90,000 traces and a clear pattern: when a customer mentioned a duplicate charge, the agent refunded the most recent charge without checking whether a refund already existed. One sentence in the system prompt caused 71% of all refund errors.
We went from “the agent is acting weird” to the exact span and the exact prompt line in under a minute.
Proving the fix before shipping it
Instead of deploying a new prompt and waiting for complaints, the team used Replay to run the new version against the 412 traces that had failed. Tool responses came from the recording, so no real refunds were issued. Policy adherence rose from 0.72 to 0.91 and the refund-amount check passed on 99.4% of cases.
Guardrails as a safety net
Northpeak now runs three guardrails on the refund agent:
- A $500 per-refund limit that asks a human to approve anything larger.
- A loop breaker that stops a run after five identical tool calls.
- An hourly spend cap of $120 across all agents.
In the last quarter the limit rule triggered 1,204 times. Every one of those refunds was reviewed by a person before any money moved.
What’s next
The team is moving its eval suite into CI, so a pull request that lowers any score by more than 2 points cannot be merged. They’re also giving support leads direct access to traces, so escalations arrive with the root cause already attached.