Live

Increased Error Rates in SFO3

0

Increased error rates affected multiple services in the SFO3 region, including Networking in SFO3, from 02:53 UTC to 04:32 UTC. DigitalOcean reported that the incident is resolved and services have been stable since mitigation. Operations that failed during the incident may need to be retried.

State

Resolved

Started

Lasted

13 h 27 min

Worst impact

Degraded

Timeline

4 updates
04:3707:5911:2114:4218:04
DegradedUpdate postedTimes in UTC

Investigating

7 min 1%

Monitoring

13 h 19 min 99%

Updates

Times in UTC

As posted by DigitalOcean

Resolved

Our Engineering team has confirmed that the increased error rates in the SFO3 region are resolved. From 02:53 UTC to 04:32 UTC, users may have experienced errors, timeouts, or delays when creating or managing Droplets, Volumes, Load Balancers, Reserved IPs, Cloud Firewalls, Managed Databases, and Managed Kubernetes clusters through the API or Cloud Control Panel. Some users also experienced connectivity issues with Load Balancers and with newly created or restarted Droplets. The impact was caused by a network routing issue that disrupted connectivity to one of our core database systems after a load balancer node became unavailable. Services have been stable since mitigation. Operations that failed during the incident may need to be retried. We apologize for the inconvenience. If you continue to experience issues, please open a support ticket from your DigitalOcean account.

Monitoring

Our Engineering team has mitigated the issue, and services in the SFO3 region are currently operating normally. The team is continuing to monitor the situation to ensure stability given the scope of the impact. We will continue monitoring and share another update once the situation is fully resolved. In the meantime, if you encounter any further errors or timeouts, please retry the affected operations. If the issue persists, please submit a ticket through your Cloud Support page so our team can assist you further: https://cloudsupport.digitalocean.com/s/createticket

Monitoring

Our Engineering team has mitigated the increased error rates impacting multiple services in the SFO3 region and is monitoring the situation. The impact was caused by a network routing issue that disrupted connectivity to one of our core database systems after a load balancer node became unavailable. As of 04:32 UTC, users should no longer experience errors or timeouts when creating or managing resources in SFO3. Operations that failed during the incident may need to be retried. We will post an update as soon as the issue is fully resolved.

Investigating

As of 02:53 UTC, our Engineering team is investigating increased error rates impacting multiple services in the SFO3 region. At this point, users with resources in SFO3 may experience errors, timeouts, or delays when creating or managing Droplets, Volumes, Load Balancers, Reserved IPs, Cloud Firewalls, Managed Databases, and Managed Kubernetes clusters through the API or Cloud Control Panel. Some users may also experience connectivity issues with Load Balancers and with newly created or restarted Droplets. We apologize for the inconvenience and will share an update once we have more information.

Affected components

2
SFO3Networking - SFO3

DigitalOcean in the last 90 days

Full history
90 days agoToday

Incidents in 90 days

11

Days with an incident

13

Median duration

2 h 29 min

This incident

5.4× the median

How long each incident lasted, on the day it started. The outlined bar is this incident.

25 h 50 min
median 2 h 29 min
90 days agoToday

Other DigitalOcean incidents

All incidents

DigitalOcean·App Platform Accessibility in Cloud Panel

Between 16:40 UTC and 18:22 UTC, our Engineering team observed an increase in 4xx error rates with App Platform operations via the Cloud Control Panel. During this time, users may have experienced errors while attempting to view or update their apps on the Cloud Panel. This was a UI-only issue and apps continued to run throughout the duration of the incident. Our Engineering team has resolved this issue. As of 18:22 UTC, users should no longer be experiencing this issue and all services should now be functioning normally. We apologize for any inconvenience, and if you are still experiencing any problems or have additional questions, please open a Support ticket within your account.

Resolved

DigitalOcean·Kubernetes Cluster Management Operations

We have resolved the issue affecting Managed Kubernetes cluster operations across all regions. Cluster creation, deletion, scaling, and upgrade operations have returned to normal. We have monitored the environment and confirmed that services are operating normally. If you continue to experience any issues, please contact our Support team by opening a ticket here: https://cloudsupport.digitalocean.com/s/

Resolved

DigitalOcean·App Platform Deployments

Between 04:34 and 12:15 UTC, customers experienced service interruptions affecting App Platform deployment operations. During this window, customers may have been unable to deploy new applications or scale existing deployments. Applications already running prior to the incident were unaffected and continued to operate normally. The disruption was caused by a software update that prevented application containers from starting on newly provisioned infrastructure nodes. Our Engineering team rolled back the update and restored the affected nodes. App Platform deployment and scaling operations have recovered and are operating normally. We sincerely apologize for any inconvenience this caused. If you continue to experience any issues, please open a support ticket and our team will be happy to assist you.

Resolved

DigitalOcean·DNS Resolution for Managed Kubernetes

Our Engineering team has resolved the external DNS resolution failures affecting Managed Kubernetes across multiple regions. Applications running on Managed Kubernetes can now resolve external domain names. We sincerely apologize for any inconvenience this disruption may have caused to your operations. If you continue to experience problems, please open a ticket with our Support team: https://cloudsupport.digitalocean.com/s/

Resolved

DigitalOcean·VPC Networking Availability

Our Engineering team fully resolved the VPC Networking disruption as of 10:29 UTC. Customers can now create and delete VPC peering connections, subnets, and routes as expected. Following the recovery, we monitored the affected systems and confirmed that VPC Networking operations remain stable. We apologize for any inconvenience this disruption may have caused. If you continue to experience unexpected behavior, please open a ticket with our Support team for further assistance.

Resolved

Follow DigitalOcean

Your coding agent knows when DigitalOcean is down, and tells you when a failure is theirs, not yours. You can also get DigitalOcean incidents in a feed reader or in Slack.

Prompt
Install the outages.live MCP server, so that when something breaks you can tell if the cause is a provider we depend on and not my code. It reads the official status pages of cloud, AI, payment and developer providers and gives you their incidents as MCP tools.

Add it as a remote MCP server named "outages" at https://outages.live/mcp (Streamable HTTP, no authentication), with your CLI command or your MCP config file.

Then:
1. Make sure it connects and shows the tools get_provider, list_feed and send_feedback.
2. Call get_provider with "digitalocean" and tell me if DigitalOcean has a problem right now.
3. From now on, when a call to a third-party service fails or becomes slow, call get_provider for that service before you change my code.

Paste this into any coding agent. It adds the server with its own MCP setup.

RSS
https://outages.live/feed.xml?providers=digitalocean
Slack
/feed subscribe https://outages.live/feed.xml?providers=digitalocean

FAQ

Go from tracking outages to fixing production issues with Polylane.

Try Polylane