Downtime Explaination

[Pimpin PSP]

Hello everyone. Sorry for the inconvenience, but our host decided to upgrade their servers yesterday, and in the process, they encountered some difficulties.

Here are our notes during this process taken from them

approx. 11:00am EST Sunday: servers go down for upgrade.

approx. 2:00pm EST Sunday: becomes known that something went wrong.

3:30pm EST Sunday - HOST RESPONSE - there are some issues with the storage array on that particular node and I am trying to fix it..

approx : 10:30am EST Monday: Servers seem to be functional again, but mysql still down, databases throwing errors

4:20pm EST Monday: Servers go back down, upgrade finally? in progress.

4:37 EST Monday: -HOST RESPONSE - [we are] trying to get everyones accounts right and the specs of the server right

8:07 Monday EST: after 33 hours of down time, we are back!!!!

So hopefully now it won't happen again. And if it does I set a redirect to our very old blog here:
http://hablog.wikidot.com/

 

[ImHumbled]

Thanks, but didnt really mean anything when the site got back up as I am still banned from the shoutbox.

 

[cr4zy_dude_h4x]

yay glad we are back, thanks for the update. i was a bit worried when it was down for two day :awe:

 

[Double_0_negative]

sorry guys, we had a small downtime again today. this time it wasn't caused by server upgrades, but rather by a rouge cron job script (more specifically the bots) going crazy and spamming the shoutbox. somehow the script got stuck in an infinite loop, and started spamming the shoutbox. the script started making 100,000 database queries a minute, and brought the server load averages up to around 16.00! we took the site down around then because of two things, one, with the rouge script running, nothing would load because the server was overloading, and trying to load pages would just cause more load, and two, theoretically, the way vbulletin crons jobs work, if absolutely no one is accessing the site, the cron cannot run.

eventually we got the script stopped, but the shoutbox and various other things where completely screwed up, so we had to load up a backup. lucky the database, other then the shoutboxes (which by the time the script stopped was over a 500mb with about 2,000,000 items) which we had to completely delete, so no post where lost, thank goodness

 

[cr4zy_dude_h4x]

thanks for the update. i wondered why the site souldn't load the other day. i love how you guys keep us updated

 

[Pimpin PSP]

No problem . We try to inform you guys if anything goes wrong. I don't like it when people worry.

 

[Double_0_negative]

we try to keep you all updated as soon as anything happens. like Pimpin, i dont like it when people are worried, but i also think that if people are like me, they are curios on what happend. i know a lot of other sites just say we where down and now we are back not really explaining anything, or they just dont say anything at all, but i like to explain thing to my members, because i would be curious and i want to satisfy that curiosity of any of my members.