System downtime on 11/18/2020

neptronix

Administrator
Staff member
Joined
Jun 15, 2010
Messages
17,528
Location
Utah, USA
Well that was a white knuckle moment!
Amazon bricked the bootloader of our server after an update and i had to pull the disk and reconstruct the system.
Amazon's disk performance limits meant that this reconstruction took about 5 times longer than it should have while copying files.

I have to say i am not pleased with amazon right now and will not be updating the server or even rebooting it until we move away from this host.

Sorry for the interruption. I got us back up as fast as possible.
 
Thanks for the tip. I didn't even know that Oracle did cloud hosting. It does look more like they're oriented towards larger organizations, but i like the modularity of their offerings. And pricing. Will keep that in mind. :)

I've had clients on AWS since 2015. It's been a super smooth ride with AWS until this year. I think they are not extensively testing their custom patches to mysql and the linux kernel. This year, i've had to rescue two machines from an unbootable state, and had a remote database that had redundancy go down for over an hour with no explanation. I had to pay for Amazon's tech support in order to ask why the database went down. The answer i received came very late and they blamed me for the outage, despite all evidence showing the problem was at their datacenter.

I think AWS has gotten cocky because they have a majority marketshare over web hosting lately.

Anyway between moving to another host soonish, upgrading to the latest php3.2.x branch, and also implementing some pretty badass anti-DDOS techniques i've been trialing on 2 other clients' servers, ES should see unprecedented levels of stability.

There are some bigger tricks up my sleeve for 2021 :thumb:
 
Back
Top