Five tabs at 2 a.m.
Sentry, Datadog, CloudWatch, GitHub and the database are all open, and none of them talk to each other.
From the alert to the root cause.
When something breaks, TraceX lines up the errors, the logs and the deploys, finds the change that caused it, and writes the fix, while you're still reading the alert.
The problem
Sound familiar?
Which deploy was it?
Matching the first error to the change that caused it is slow, manual work.
The fix waits for morning
Even once the cause is clear, someone still has to write and test the change.
Errors
Every error traced to the change behind it.
TraceX reads the Sentry issue and its stack trace, finds the line in your code, and matches the first occurrence to the deploy that shipped it.
- Stack traces mapped to your code
- Matched to deploys and commits
- Read-only, from the first minute
checkout-web· issuesErrorTypeError: Cannot read properties of undefined (reading 'amount')
chargeCard · first seen Oct 22, 14:02
node_modules/stripe/lib/charges.jscreatesrc/checkout/payment.ts:88chargeCardsrc/checkout/submit.ts:41onSubmitIt started with deploy a1c9e2: on a retry the payment intent is undefined.
Logs
The lines that explain it, picked out.
TraceX searches Datadog and CloudWatch around the moment it started and highlights the log lines that show what went wrong.
- Datadog and CloudWatch Logs Insights
- Queries written for you
- The relevant lines highlighted
Logs· service:checkout14:01:52INFOPOST /charge 200 order=48188
14:02:03WARNcard_error retry=1 order=48190
14:02:04INFOPOST /charge 200 order=48190
14:02:04INFOPOST /charge 200 order=48190
14:02:31WARNcard_error retry=1 order=48197
14:02:32INFOPOST /charge 200 order=48197
Each card error retried the charge without an idempotency key, so 6 orders were charged twice.
The fix
A pull request, ready to review.
TraceX writes the smallest change that fixes the cause, with its risk and how to validate it. You review it and merge it.
- You approve every change
- Validation steps included
- Draft pull requests if you prefer
app/workers/refunds.py+1 −1
def pending_refunds(db):
− return db.refunds.filter(state="pending")
+ return db.refunds.filter(state="awaiting_payout")
Validate run the refund worker against staging; 34 refunds move to payout
Timeline
The incident, in order.
TraceX lines up the deploy, the first error, the alert and the fix on one timeline, ready for the postmortem.
- Deploys, errors and alerts in one place
- Times taken from your own systems
- Ready to paste into the postmortem
14:00Deploy a1c9e2
Retry card errors in checkout
14:02First error
TypeError in chargeCard, 128 events
14:04Alert fires
Checkout error rate above threshold
14:07Root cause found
Retries run without an idempotency key
14:11Fix opened
Pull request #322, ready for review
How it fits
Set up once. Useful every day.
Connect your signals
Sentry, Datadog or CloudWatch, GitHub, and a read-only database.
Ask when it breaks
Ask in the dashboard, in Slack or with tracex -c from your terminal.
Review and ship
Check the root cause and its evidence, then review and merge the fix.
What you get
Built for how you work.
Signals in one place
Errors, logs, deploys and data, lined up on one timeline instead of five tools.
The cause, not the symptom
TraceX follows the evidence back to the change that started it.
Shorter incidents
The investigation runs in parallel with you, from the moment you ask.
Ask follow-ups
Who was affected? Since when? Ask in plain English and get cited answers.
From the terminal too
Run tracex -c and ask without leaving your shell.
No risk to production
Read-only database access, and write queries refused before they run.
FAQ
Questions
Does TraceX replace our alerting?
No. Your alerts still tell you something is wrong; TraceX helps you find out why, and fix it.
Can it roll back a deploy?
No. TraceX never changes production. It tells you which change caused the problem and writes a fix for review.
Does it work from the terminal during an incident?
Yes. tracex -c gives the same answers, with the same evidence, without leaving your shell.
Find the root cause.
Ship the fix.
TraceX is free during the private beta. Connect a database and a repository and ask your first question in minutes.