Fri Oct 24 00:31:07 2025 (*097dfbf6*):: clown cloud https://x.com/copyconstruct/status/1981192300819862000 +public! *** Cindy Sridharan (@copyconstruct) on X *** The detailed AWS incident report is out, and it’s worth a read
– DNS records managed by 2 systems; a race condition led to regional record getting unset – EC2 lease establishment was borked as it depends on DynamoDB – fluctuating NLB health checks leading to EC2 DNS entry purges *** X (formerly Twitter)
(*4cfb807c*):: ^ the worst outage in their history, only finally getting an explanation on Thurs, my company surprisingly didn’t get effected that bad by it, but there were a lot of drunk ops this week (*53e37792*):: You have an abnormally busy week? (*4cfb807c*):: Nah the company I work at got off easy, they have a ton of us-east-1 resources but we dont lean heavily on ec2 or dynamodb. Dynamodb isn’t really used by anyone except aws at this point. (*4cfb807c*):: Theres probably a handful of other services they have that could cause cascading service failures like this though, but losing ec2 scale is a pretty big blast radius. (*097dfbf6*):: clown cloud https://x.com/copyconstruct/status/1981192300819862000 +public! *** Cindy Sridharan (@copyconstruct) on X *** The detailed AWS incident report is out, and it’s worth a read
– DNS records managed by 2 systems; a race condition led to regional record getting unset – EC2 lease establishment was borked as it depends on DynamoDB – fluctuating NLB health checks leading to EC2 DNS entry purges *** X (formerly Twitter) (*4cfb807c*):: https://aws.amazon.com/message/101925/
(*097dfbf6*):: shell uses AWS but only the clown groups were down. Our groups stuff is on Azure clown lol