Summary

On July 16, 2026, between 07:54 and 08:56 UTC (approximately 62 minutes), Morpho's hosted API services were unavailable due to a global AWS CloudFront incident affecting VPC Origins connectivity. This impacted the Morpho API (public, partner, and app surfaces) and API documentation. Requests during this window returned 5xx errors or timed out after ~30 seconds.

We restored service by deploying an infrastructure workaround roughly 2.5 hours before AWS resolved the underlying issue on their side, and within 12 minutes of AWS first publicly acknowledging the issue.

Impact

07:54 UTC: API error rates began rising sharply; within minutes, most API endpoints were returning 5xx errors or 30-second timeouts.
All API surfaces fronted by our CDN were affected: REST and GraphQL APIs, and docs.
Integrator API traffic was effectively at zero for the duration of the incident.
08:56 UTC: Full recovery. Error rates returned to 0% and latency to normal levels.

Root Cause

The root cause was an incident within AWS CloudFront, our CDN provider, affecting customers using VPC Origins, the private connectivity path between CloudFront's edge network and origins inside a customer VPC (AWS Health event). Per AWS's resolution statement, an internal constraint was reached on the fleet managing connections to private VPC origins, and the system distributing routing configuration to their network processors failed to load updated configuration correctly. AWS reported a global impact window of 07:45–11:18 UTC.

Our backend services were healthy for the entire incident: workloads were up, internal load balancers were reachable from within our network, and no application or database changes were needed to recover. CloudFront could not reach our origins: it accepted requests at the edge and timed out waiting for origin responses.

We confirmed this diagnosis during the incident by observing identical failures across multiple independent CloudFront distributions and geographic edge locations simultaneously, while direct probes to our origins succeeded.

There was no security impact: no unauthorized access occurred and no data was exposed, including during the temporary workaround period.

Response Timeline (UTC)

July 16

Time	Event
07:45	Impact begins for CloudFront customers using VPC Origins (per AWS's resolution statement).
07:54	Morpho error rates spike; monitoring alerts fire. First customer-facing signal.
07:58	Incident response begins; on-call engineers engaged.
08:10	Investigation localizes the failure to the CDN-to-origin path; backends confirmed healthy.
08:12	Work begins on a workaround to bypass the failing connectivity path.
08:33–08:44	Proactive notifications sent to integrators; public status page updated (status.morpho.org).
08:32	Workaround change opened for review.
08:42	Workaround merged.
08:44	AWS posts its first public notice: "investigating increased 5xx errors for CloudFront customers utilizing VPC Origins connectivity."
08:50	Workaround applied: CloudFront origins switched from private VPC connectivity to temporary hardened public load balancers (access restricted to CloudFront's managed IP ranges, WAF and TLS posture preserved).
08:51	Origin latency returns to normal, from ~30 seconds to under one second.
08:56	Error rates reach 0%. Incident mitigated for Morpho traffic.
09:21	AWS confirms the issue is scoped to VPC Origins and recommends changing origin type as a workaround, which is the approach we had already deployed at 08:50.
09:57	AWS identifies the root cause (per their resolution statement).
11:18	AWS reports full recovery on their side.
12:21	AWS posts its final resolution summary and confirms origin-type changes can be safely reverted.

July 20

Time	Event
08:14–08:37	After several days of confirmed stability on AWS's side, we reverted to our standard private-origin architecture. CDN configuration propagation caused a brief (~1–2 minute) spike in 5xx errors.

What Went Well

Detection was fast: alerts fired within the first minute of impact.