Introduction
Kalytera catches the mistakes your AI agents make in production, and stops the costly ones before they happen.
What Kalytera does
- Reads every conversation from Langfuse or LangSmith (no code), or from the SDK
- Judges each whole conversation, including what every action did and returned
- Groups the failures into problems, named in plain English, and tells you in Slack
- Turns a problem into a rule; kalytera.init() blocks the action before it runs
- Shows failed conversations before and after a rule started blocking
Two ways in
Connect Langfuse or LangSmith from the dashboard to find problems, with no code. Add one line to your agent to also stop costly actions. Many teams start with the first and add the second once they've seen their problems.
Connect Langfuse or LangSmith
No code. Kalytera reads the last 30 days right away, then new conversations every 15 minutes.
1. Get read keys
Langfuse: Project settings → API keys → create a key pair (a public key pk-lf-… and a secret key sk-lf-…). Note your region (EU or US), or your URL if you self-host.
LangSmith: Settings → API keys → create a key (lsv2_…).
2. Connect
Open app.kalytera.dev, sign in with your Kalytera key, click + Connect an agent, choose Langfuse or LangSmith, name the agent and paste the keys. Problems appear as soon as judging finishes; the dashboard shows its progress.
Your keys
They're stored encrypted, only ever used to read conversations, and deleted when you remove the connection. Conversations newer than 5 minutes are left for the next read, so they're complete when judged.
Add to your code
Records every conversation, and checks every action against your rules before it runs.
1. Install
2. Add one line
agent.py
# export KALYTERA_API_KEY=kly_live_...
import kalytera
kalytera.init() # before your agents run
That's all for the OpenAI Agents SDK and LangChain / LangGraph: every run is recorded and judged, every tool call is checked against your rules, and each agent appears under its own name.
3. Your own agent loop? Two more lines
agent.py
@kalytera.tool # checks your rules before each call
def refund_api(order_id, amount): ...
with kalytera.session(conversation_id, input=message) as s:
s.output = run_my_agent(message)
Kalytera never breaks your agent
Recording happens in the background. If Kalytera is unreachable, your agent keeps running and its actions still run. Only a rule you've switched to blocking stops an action.
Try it with no agent of your own: pip install kalytera, then KALYTERA_API_KEY=kly_live_… kalytera quickstart, runs a sample refund agent, finds its problem, writes the rule and blocks the next bad refund.
Slack alerts
A message right away when there's a serious mistake (even just one) or a reply was stopped before a customer saw it, and when a smaller problem starts repeating or gets worse. Each links straight to the conversation.
Set up
- Go to api.slack.com/apps → Create New App → From scratch, and pick your workspace
- Incoming Webhooks → turn on → Add New Webhook to Workspace → choose a channel → copy the URL
- In the dashboard: + Connect an agent → Alerts in Slack → paste the URL
Kalytera posts a test message right away. Problems found before you turned alerts on stay in the dashboard; Slack hears about new ones from then on.
How conversations are judged
A judge model reads each finished conversation as a whole: what the customer asked, what the agent said, and what each action was given and returned.
Six scores
accuracy
0–100
Was it correct, given what the actions returned?
goal_alignment
0–100
Did it serve what the customer actually needed?
decision_quality
0–100
Were the right actions taken, with the right inputs, in the right order?
completeness
0–100
Was the request fully resolved?
helpfulness
0–100
Could the customer act on the answer?
factuality
0–100
Was every claim backed by what the actions returned?
A conversation passes at 70 or more. In the dashboard (Overview → Scoring configuration) you can pick weights for your kind of agent, change the pass mark, and add your own checks. A failure also gets one of seven types: wrong_answer, tool_failure, goal_drift, incomplete, hallucination, context_loss, loop.
When the judge is wrong
Open the conversation and click 👍 Not a failure or 👎 This failed. Your verdict replaces the judge's everywhere, and problems are regrouped right away. Judge models make mistakes; this is how you correct them.
Problems
A problem is the same mistake repeating: at least 5 failed conversations that went wrong in the same action or step.
Failures are grouped by the action they were about (refund_api, track_order), or by step or type of failure when no action is involved. Each problem is named from all of its conversations, shows how many it affected and whether it's getting worse week over week, and lists the conversations. Problems update every hour, and right after each new read from Langfuse or LangSmith.
Rules
A rule checks an action before your agent takes it: for example, a refund over $250 needs a person to approve it.
- Starter rules, before anything goes wrong. Kalytera looks at the actions your agent takes and suggests "a person approves it first" for the risky ones: commands that can wipe files or databases (like rm -rf or DROP TABLE), turning off monitoring or security, deleting things, moving money above a limit, and changing accounts
- Suggested rules come from your problems, with what they would have caught: failed conversations stopped, and good ones touched
- New rules only watch. They record what they would have blocked, so trying one is safe. Switch a rule to blocking when you trust it.
- Blocking needs the SDK (kalytera.init(), or @kalytera.tool on your own tools). A blocked action doesn't run. Your agent gets the rule's reason instead, for example "Refund needs a person to approve it when the amount is over 250".
agent.py
@kalytera.tool(on_block=lambda d: f"I can't do that automatically: {d.reason}")
def refund_api(order_id, amount): ...
Without on_block, a blocked call raises kalytera.Blocked. If Kalytera can't be reached, actions run.
Check replies
An agent that says "your refund has been processed" when the refund failed does as much harm as a wrong refund. check_reply() stops that reply before the customer sees it.
agent.py
with kalytera.session(conversation_id, input=message) as s:
reply = run_agent(message)
check = kalytera.check_reply(reply) # the original, or a safe message if it was held
s.output = check.reply
send_to_customer(check.reply)
- Always on, no delay: a reply that says something was done (refunded, cancelled, deleted, sent) is held when that action failed, was blocked, or never ran in the conversation. This is plain code, not AI.
- AI review, if you want it: check_reply(reply, review=True, policy=YOUR_POLICY) also has Kalytera's AI reviewer read replies about money, policy, accounts or legal matters before they're sent, and hold promises your policy doesn't make. It adds about 1–3 seconds to those replies only. If the reviewer doesn't answer in time, the reply goes out.
- You hear about it: every held reply goes to Slack right away and shows in the Failure Feed, with what the agent wanted to send. The customer gets a safe message instead; set your own with safe_reply="…".
Memory and sharing
Kalytera remembers at three levels: the agent, your account, and anonymously across companies.
- The agent: its problems and rules
- Your account: rules from your other agents are offered to each agent, with what they would have caught there
- Across companies: tell Kalytera what the agent does (industry and kind of agent) and it warns you about mistakes agents like it make elsewhere
What's shared is anonymous and aggregated: kinds of mistakes and the shapes of rules that stopped them. Conversations, names and your limits are never shared, and a pattern appears only once at least 5 companies share it. Sharing is on by default. Turn it off in the dashboard (Overview → Data & retention); your contributions are removed right away. Data & security →
API reference
kalytera.init()
Call once at startup, before your agents run. Returns immediately.
kalytera.init(
api_key: str = "", # or KALYTERA_API_KEY
agent_id: str = "", # or KALYTERA_AGENT_ID
redact: bool = False, # or KALYTERA_REDACT=1
) -> None
api_key
str
Your Kalytera API key (kly_live_…).
agent_id
str
Report everything under one name. By default each framework agent appears under its own name.
redact
bool
Mask email addresses, phone numbers and card numbers in everything sent to Kalytera. Rules still see the real values.