Runs list/logs degraded
# Postmortem: Missing task logs and traces Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered ## Summary Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand. ## What customers experienced Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeed
The latest update from Trigger.dev.
State
Resolved
Started
60 d ago
2026-09-03 14:56 UTC
Lasted
1 h 24 min
until 2026-09-03 16:20 UTC
Worst impact
Degraded
Timeline
5 updatesDegraded
1 h 24 min 100%
Updates
Times in UTCAs posted by Trigger.dev
# Postmortem: Missing task logs and traces Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered ## Summary Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand. ## What customers experienced Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeed
The ingestion delay is down to normal levels. A post-mortem will follow.
Run logs/traces are inserting and the ingestion delays for are coming down. We have fixed the underlying issue. We will resolve this issue with more details when the ingestion delays are back to normal.
We have identified the root cause of the issue and are working towards mitigating the issue. Currently the run list dashboard and API are fine, but we aren't able to insert new run logs and trace data.
The insert delay for the runs list and logs is longer than usual. This impacts the dashboard and APIs associated with listing runs and viewing run logs. This is being caused by higher load and slow auto-scaling in our ClickHouse cluster.
Affected components
3Trigger.dev in the last 90 days
Full historyIncidents in 90 days
4
Days with an incident
4
Median duration
57 min
This incident
1.5× the median
Other Trigger.dev incidents
All incidentsTrigger.dev·Security Disclosure: Sunday November 1st
There has been a security incident at Trigger.dev. We've blocked the exploit used in this attack but our investigation has identified exposure across a subset of organizations. The attacker gained unauthorized access to environment variables for these organizations. We have emailed all impacted account owners with steps to mitigate. We will follow up with more details.
Trigger.dev·Some sensitive environment variables synced from Vercel were cleared
We stopped the issue at 15:10 UTC and no environment variables have been cleared since. A permanent fix is in progress. Variables that were cleared between 12:21 and 15:10 UTC are still empty and will not come back on redeploy. We have contacted every affected customer directly. Re-entering them in the Trigger.dev dashboard under Environment Variables fixes the affected project right away. We're still working with affected customers to restore their values, so if you need help or have questions, reach out to us at support or reply to our email.
Trigger.dev·Run logs are being dropped
Logs are being fully ingested again. The issue was with the internal workers in our shared OpenTelemetry service. They became unhealthy but weren't rolled. We're looking into why and how to prevent this from happening again.
Follow Trigger.dev
Your coding agent knows when Trigger.dev is down, and tells you when a failure is theirs, not yours. You can also get Trigger.dev incidents in a feed reader or in Slack.
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.
Add it as a remote MCP server named "outages" at https://outages.live/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.
Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "triggerdev" and tell me if Trigger.dev has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.Paste this into any coding agent. It adds the server with its own MCP setup.
https://outages.live/feed.xml?providers=triggerdev/feed subscribe https://outages.live/feed.xml?providers=triggerdevFAQ
Go from tracking outages to fixing production issues with Polylane.
Try Polylane