My schools entire assignment system is out today.
I brought down all my department’s services to take a day off and blame it on Amazon. Next year when negotiating a raise/budget increase, I’ll point to this incident and take credit for migrating us off AWS after six months of in-person training classes (either in places I haven’t visited or would like to see again) and another six months of hard work in the office (napping in the server room).
2026 is looking pretty good already and I definitely won’t regret tempting fate by saying that.
Legendary.
great idea! show us your nap setup! sitting or lying down?
It’s sad that none of these people you deal with don’t know what service they use or vendors they deal with? That sounds fantastic for you, scary for everyone else.
You’d be hard pressed to find an online service that isn’t associated with AWS in some way.
Sadly, there are some who don’t even know it, because they’re buying services from someone else that buys them from someone else that buys them from Amazon. So they’re currently wondering what the fuck is even going on, since they thought they weren’t using AWS.
Well those people are fucking idiots.
That’s not really fair, I think. Smaller organizations are especially dispositioned here. Think small businesses, charities, local municipal services, etc. Small IT budgets, low staff (if any) and just enough to pad out a subscription cost to a service provider that fits their needs.
AWS is an incredibly low cost solution, and it’s probably where most of these low cost services point themselves at when building platforms at scale. Not everyone can build and maintain a datacentre or home server for their every need.
This isn’t to say that there are definitely idiots who pad their resume by chanting a prayer to SaaS and boasting about having moved their company to the “cloud” via a cheap and unreliable AWS rehoster, before failing upwards though.
Fine, most of them are fucking idiots. Know where your infrastructure is people! Whois your IP.
I’m pretty sure most of Azure (Microsoft), OCI (Oracle), and GCP (Google) have all been fine.
Bezos is a craven beast but I don’t see many companies above with CEOs that I’d feel comfortable babysitting my teenage daughter
The company I work for is an Azure shop. However, our provider for customer 2fa tokens uses AWS… So still in trouble.
Once Larry Ellison owns TikTok he’s going to be babysitting all the teenagers and a whole bunch of other people!
I really hope that super villain wannabe croaks out real soon
I don’t think his son is any better.
If daddy croaks now we will see how much is actually David’s doing vs Larry
Sure, but online services can certainly leverage multiple modules, from multiple companies, hosted in multiple places. So maybe your site mostly works fine, but a key aspect of it is broken.
from multiple companies
See the above post from the Azure shop … that uses AWS for 2FA tokens
You want to add multiple companies in parallel as alternates/failovers, not in serial where any one failure blocks the whole flow
Also allow things fail gracefully, independent of each other.
But that would cost more money, that’s anticapitalist.
Yes, it’s much more expensive to have two providers. Both in terms of outright costs but even more so in terms of ongoing engineering/technical overhead.
The calculus is how much the expectation downtime is, versus that cost. It’s a reasonable calculation and TBH if outages are a few hours once every few years for most cases it’s acceptable.
OFC if your hospitals or emergency services depend on a cloud service, you happily fork over the extra money same as you do for any other insurance.
If there’s anything I know, it’s that “businesspeople” are never proactive.
Lemmy seemed fine, Reddit did not.
Lemmy seemed fine
Federated, open source
Reddit did not
Centralized, corporate
Amen to that, good thing though. Got me to learn what Lemmy was. Apparently I’ve been under a rock.
Welcome to the fediverse!!
Walmart.com would likely work fine, as they are rabidly anti-Amazon, especially AWS. They don’t even want their SaaS vendors using AWS under the covers for them.
Can confirm, about 10 years ago, the company I worked for migrated to AWS, and I managed the transition. We planned everything meticulously so that there would be no downtime, and used it as excuse to fix a lot of tech debt. No one was supposed to even notice the cutover, and when we did it, I expected the only feedback to be that things seemed faster and were working as expected. A few hours later, we get a complaint from an Account Manager for Walmart that they can’t access the platform at all. There was a lot of confusion and back and forth, turns out their IT department had an allow list or something in the corporate DNS to not resolve to AWS owned IPs unless approved. We eventually got them to add our domain to their allowlist, but it seemed insane that they would spend the effort to implement and maintain that level of control.
EBSCO was down. Ask me how my dissertation writing progressed today…
How did your dissertation writing progress today?
Half the articles I needed to read couldn’t be accessed due to aws. I figured I would grade assignments in the mean time, up until I learned e-learning was down as well. Nothing came back up till about 3.
Thanks for asking. How was your day? Was it affected by the AWS outage?
My school uses canvas for assignments, and it was down. It was pretty hectic for the teachers. Its amusing to me to see so many services down. Reminds me of the xkcd cloud. https://xkcd.com/908/
Same, Canvas is perhaps the most used Learning Management System in the US and they apparently are entirely hosted on AWS East. The real kicker is I had my students midterm due date literally today for two classes. I’ve been swamped with panic emails (and I made clear my due dates aren’t even that important when there isn’t a national outage lol).
My head canon is someone wished for a miracle due date extension somewhere in the country and they monkey’s pawed AWS into non existence.
I thought Canvas was self hosted? Or do they offer service instances too?
I thought that too until today, it’s never clear until a quarter of the Internet is down randomly.
I’m sure there’s a self hosted option but no way in hell it’s as popular as it is without being a turnkey solution. Most smaller districts wouldn’t touch anything self hosted with a 10 foot pole. They don’t have the resources.
I just want AWS, Azure, Google, & Cloudflare to go down at the same time.
And maybe stay down.
If a nation wants to go to war with the US … this is how they do it, they just shut down one, two or all of these systems down and watch the country go crazy. It wouldn’t destroy the country, just disrupt it enough to make them go nuts and then do more things to them in other ways.
It’s amazing when you think about it, first the US invested in heavily defending and arming itself in the 60s, 70s and 80s … then it spent billions more in the 90s and 2000s to try to come up with ever more inventive ways to screw itself from the inside.
No. Everyone always focuses on the flashy stuff like datacenters but the truth is that the most vulnerable, overtaxed, and underfunded weak-spot for the United States is the electric grid.
Yup. Most of our core electrical infrastructure is over 100 years old now. And thanks to a combination of NIMBYism, profiteering, and the anti-nuclear brigades, we’re not likely to see that change any time soon.
I’m pretty convinced that if we did have a complete grid failure, we wouldn’t be able to complete a cold start.
Too many people would be pushing for their section to be started first so they could short the market first.
Edit: my proofreading sucks
Doesn’t matter who wants want, the power company has contingency plans and those plans are what is happening.
SOURCE: Worked for Cox after a major ice storm smashed north-central Oklahoma flat. Lived Hurricane Ivan and saw their rollout priorities.
The American dream
Note to self: Sabotage the US’ electric grid.
Which in turn would take down the datacenters too, so same effect, just more severe
Most data centers are backed up by generators.
… that run on fuel for a limited time … once the fuel runs out, the data centres go down
Need to transport more fuel there? Can’t because the entire system is down.
All*
And UPSs.
I feel like you got it backwards. Letting them run is doing more damage than turning them off would.
If the way someone goes to war with the US is by freeing us from the overly-centralized landlords of the Internet, maybe whoever they are isn’t so bad.
Cloudflare in particular can fuck off and stay fucked off.
The really stupid thing is that even if you weren’t in AWS east us 1 you were still boned because that is where AWS does it’s service authentication.
I love it when Cloud companies pretend there are “serverless” services that are “location-transparent”
You know, they sell this crap to governments and have to follow compliance regimes like FedRAMP but yet… this happens
But the only way to do this is to have a CSO willing to invest heavy in red-teaming – for attacks of every kind the team can brainstorm – and a CEO willing to spend the $$ and attention to get their recommendations implemented.
MS-365 next!
We all fund the devil pretending we don’t know.
Do I have any choice?
I wouldn’t look at it that way. Even companies not leveraging AWS directly will be impacted.
Especially Microsoft, Google, and Oracle, who are all at this very moment probably sending out sales droids in vast numbers
There’s so much vendor lock in with AWS, migrating to another provider over an outage even lasting 24h would be a tough sell. This isn’t unique to AWS either, each of the cloud vendors have their own lock in and their own problems. If you had the money you could run in multiple clouds, but for most businesses who were only running in a single region, I can’t imagine they’d choose this option.
Sounds like those companies are not considering the capabilities of their supply chain then.
My wife and I are at Disneyland today, and their site is down. xD
fr i cant believe clash royale is a part of that
In my country, a lot of the non-traditional bank apps are still down, with millions of people having lost access to their money. Can’t even buy groceries, pay the bills, or anything.
Yep, I just love how one service provider goes down and way to much of the internet goes down with it.
I haven’t been able to use Flickr for 10 hours. I’m so mad
scums
Not a plural/countable word.
*Amazon’s cums
One scum. Two scums. Three scums. Four scums.
Seems countable to me.
A summation of scums
Scummation or perhaps scumsock
