Hosted agent builds failing in US East
### Summary An issue with the image registry which powers our hosted agents offering temporarily caused new hosted agent builds in US East to fail. Existing agent workloads were not affected, and no sessions or calls were impacted. ### Timeline \(UTC\) * 19:54 - Hosted agent builds begin failing in US East * 20:48 - Our team noticed the increased failure rate and began investigating * 21:04 - First mitigation deployed * 21:08 - Most builds succeeding * 22:14 - Second mitigation deployed, covering the remaining affected infrastructure * 22:24 - All builds succeeding ### Scope of Impact Only new hosted agent builds and deploys in US East were affected. Agents already running were unaffected, as were all sessions, calls, and other regions. Customers who were impacted would have seen their deploy fail with “unable to deploy agent: an error occurred while building your agent, please try again”. ### Mitigations and Follow-ups * We are improving alerting on build failures. * We are improving
The latest update from LiveKit.
State
Resolved
Started
79 d ago
2026-08-14 21:10 UTC
Lasted
1 h 51 min
until 2026-08-14 23:01 UTC
Worst impact
Degraded
Timeline
5 updatesIdentified
5 min 5%
Monitoring
1 h 46 min 95%
Updates
Times in UTCAs posted by LiveKit
### Summary An issue with the image registry which powers our hosted agents offering temporarily caused new hosted agent builds in US East to fail. Existing agent workloads were not affected, and no sessions or calls were impacted. ### Timeline \(UTC\) * 19:54 - Hosted agent builds begin failing in US East * 20:48 - Our team noticed the increased failure rate and began investigating * 21:04 - First mitigation deployed * 21:08 - Most builds succeeding * 22:14 - Second mitigation deployed, covering the remaining affected infrastructure * 22:24 - All builds succeeding ### Scope of Impact Only new hosted agent builds and deploys in US East were affected. Agents already running were unaffected, as were all sessions, calls, and other regions. Customers who were impacted would have seen their deploy fail with “unable to deploy agent: an error occurred while building your agent, please try again”. ### Mitigations and Follow-ups * We are improving alerting on build failures. * We are improving
We pushed a second mitigation at 22:14 UTC and all US East builds have been successful since 22:25 UTC. We will follow up with a postmortem describing the timeline, root cause, and follow ups as soon as possible.
The total build error rate has decreased significantly due to the mitigation, but a small percentage of builds in US East are still failing. We are continuing to monitor and will follow up with another update as soon as possible.
We have deployed a mitigation and builds are now successful. We will continue monitoring while we collect any further details to share.
New hosted agent builds are failing in US East. Existing agent workloads are not affected. We are currently deploying a mitigation and will follow up with more information shortly.
Affected components
1LiveKit in the last 90 days
Full historyIncidents in 90 days
27
Days with an incident
24
Median duration
1 h 26 min
This incident
1.3× the median
Other LiveKit incidents
All incidentsLiveKit·Disconnections and failed connections in multiple regions
This incident has been resolved. Between 07:10 and 07:43 UTC on October 31, a software bug caused media servers in several regions to restart repeatedly. During this window: - Participants connected to affected servers were disconnected. Impact was heaviest in US East, US Central, US West, Canada and Mexico, where about 18% of participants were disconnected at least once. Many were disconnected more than once, as automatic reconnects landed on servers that then restarted as well. A small share of participants in Europe Central, Europe South, UAE, Saudi Arabia and Israel were also disconnected. Most disconnected participants reconnected automatically, about half within 5 seconds, but each reconnect started a new session rather than resuming the existing one, so participants may have seen repeated audio and video interruptions. Some participants did not reconnect. Participants joining during this window may also have taken longer to connect or failed to join. - Most agents reconnected an
LiveKit·Investigating failing calls and RoomService API failures in US East and dashboard issues
## Summary On October 28, 2026, a network disruption at our cloud provider affected our US East data center. Starting at 23:28 UTC, services in US East lost the ability to reach our database and certain external services, such as model providers. Connections to end users and to our other regions kept working, so realtime media continued to flow. This caused widespread failures across SIP calls, Room Service, Egress and Ingress APIs, and Cloud Agents in US East. A related issue affected our Mexico region for longer. Connectivity in US East came back at around 23:58 UTC. Mexico recovered at 01:01 UTC after we moved its traffic to nearby regions. Our automatic failover should have moved traffic away from US East, and it didn't. That's the main thing we're fixing. We sincerely apologize to customers whose calls and agents were disrupted. ## Impact **US East \(23:28 to 23:58 UTC\)** * **SIP inbound:** a large share of inbound calls failed, rejected with 500 or 503 responses. Answered inboun
LiveKit·Increased join latency in US East and US Central
Between 21:30 and 23:17 UTC, roughly 10% of all WebRTC connections in our US East and US Central regions took 5 seconds or longer to establish. We applied a mitigation at 23:15 UTC, and join latencies have returned to normal by 23:17 UTC. We will follow up with more details on the root cause and preventive actions in a postmortem. We apologize for the impact and appreciate your patience.
LiveKit·Investigating reports of elevated API failures in Chicago region
## Summary On October 19, 2026, between 16:34 and 17:03 UTC and again between 17:47 and 17:54 UTC, some room management, agent dispatch and SIP API requests failed for rooms hosted in our US Central \(Chicago\) region. Requests from other regions that targeted those rooms also failed. During these windows some rooms could not be created or joined, and some agents were not dispatched. The cause was CPU exhaustion on the node running one of our message bus instances in US Central. A recently deployed overlay network used more CPU than it requested, and because our monitoring tracked CPU per pod rather than per node, we didn't see the node run out. That message bus instance began dropping messages, and a room-coordination service that depended on it stopped completing requests. We have moved the message bus in US Central onto larger nodes and resized other regions at similar risk. Node-level CPU alerting and automatic recovery for the coordination service are in progress. ## Impact Betwee
LiveKit·Degraded SIP Performance in US West
**Root Cause** On 2026-10-16, a subset of SIP calls in our US West region failed. Two of our SIP servers advertised IPv6 addresses for signaling and media, which many carrier providers don't support. Carriers rejected outbound calls routed through these servers, and inbound calls were answered but dropped almost immediately without audio. We had recently started running SIP in a US West cluster that uses both IPv4 and IPv6 networking and our SIP servers didn't enforce IPv4 addresses. **Technical Details** When a SIP server starts, it determines its public address by querying an external address-lookup service. On the two affected servers, our primary lookup service didn't respond in time so they fell back to a backup service that returned their IPv6 address instead. Nothing in the SIP server checked that the address it advertised was IPv4. **Timeline \(UTC\)** 2026-10-16 14:49 - First affected SIP server started with an IPv6 address; call failures began 2026-10-16 15:41 - Second affect
Follow LiveKit
Your coding agent knows when LiveKit is down, and tells you when a failure is theirs, not yours. You can also get LiveKit incidents in a feed reader or in Slack.
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.
Add it as a remote MCP server named "outages" at https://outages.live/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.
Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "livekit" and tell me if LiveKit has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.Paste this into any coding agent. It adds the server with its own MCP setup.
https://outages.live/feed.xml?providers=livekit/feed subscribe https://outages.live/feed.xml?providers=livekitFAQ
Go from tracking outages to fixing production issues with Polylane.
Try Polylane