Know exactly why your agent did that.
Sightline traces every step your AI agents take in production: each model call, tool call and retrieval. Then it scores the output, so you find the regression before a customer does.
refund-agent · “Charged twice for my annual plan”
Teams shipping agents to millions of users trust Sightline
Every decision, laid out on one timeline
Agents fail in the gaps between steps: a stale retrieval, a tool call with the wrong ID, a retry loop that burns $40. Sightline records all of it as one nested trace you can read like a stack trace.
Nested spans with full payloads
Prompts, completions, tool arguments and responses, captured automatically and redacted for PII before they leave your VPC.
Cost per outcome
See spend per resolved ticket, not just per token.
Replay any trace against a new prompt
Pick a production failure, change the prompt or model, and re-run it with the original tool responses mocked in.
I’ll refund the second charge now.
I’ll check refund status before refunding.
Search that understands agents
Query traces by tool, eval score, cost, user or any attribute you attach.
AND eval.policy < 0.5 SINCE 7d
Ship prompt changes with evidence
Score every production trace with LLM judges, code checks or human review. Then compare versions side by side before a change reaches 100% of traffic.
refund-agent · prompt v12 → v13
2,400 replayed traces| Evaluator | Type | v12 | v13 | Change | Score distribution (v13) |
|---|---|---|---|---|---|
| Policy adherence | LLM judge | 0.72 | 0.91 | +0.19 | |
| Groundedness | LLM judge | 0.86 | 0.89 | +0.03 | |
| Refund amount correct | Code check | 96.1% | 99.4% | +3.3% | |
| Tone | LLM judge | 0.93 | 0.92 | −0.01 | |
| p95 latency | Metric | 6.4s | 5.1s | −1.3s |
Judges you can trust
Calibrate every LLM judge against 50 human labels. Sightline shows you its agreement rate before you rely on it.
Datasets from real traffic
One click turns a failing trace into a regression test. Your eval set grows with every incident.
Gates in CI
Block a merge when a score drops. sightline eval --gate runs in GitHub Actions in about 90 seconds.
Stop the bad step before it runs
Guardrails run inline, in under 20ms. They can block a tool call, rewrite a response or hand the conversation to a human. Every intervention is logged on the trace.
- Spend limits per tool, per user and per hour
- PII and secret detection on inputs and outputs
- Loop detection that stops a run after repeated identical calls
- Custom rules written in Python or TypeScript
Blocked refund over limit
refund-agent tried to refund $2,340. The per-call limit is $500. Handed to a human.
Loop stopped after 6 retries
triage-router called crm.lookup with the same arguments 6 times in 9s.
Redacted card number
A user pasted a card number. It was masked before reaching the model.
Rewrite: removed legal promise
kb-answerer said “guaranteed”. Rewritten to match support policy §4.2.
Two lines to instrument. No lock-in.
Built on OpenTelemetry, so your traces stay portable. Works with every major model provider and agent framework, including the one you wrote yourself.
import sightline # auto-instruments model and tool calls sightline.init(project="acme-support") @sightline.agent(name="refund-agent") def handle(ticket): plan = llm.create(model="claude-sonnet-5-5", ...) charges = stripe.Charge.list(customer=ticket.customer) return resolve(plan, charges) sightline.evaluate("policy_adherence", sample=0.2)
import { init, agent } from "@sightline/sdk"; // auto-instruments model and tool calls init({ project: "acme-support" }); export const handle = agent("refund-agent", async (ticket) => { const plan = await llm.create({ model: "claude-sonnet-5-5" }); const charges = await stripe.charges.list({ customer: ticket.customer }); return resolve(plan, charges); });
We went from “the agent is acting weird” to the exact span and the exact prompt line in under a minute. Sightline is the first tool our support engineers open every morning.
See what your agents actually do
Free for 50,000 spans a month. No credit card.