[S] Site shutdowns

Post about what you like and dislike on AAO and suggest new features and improvements.

Moderator: EN - Forum Moderators

User avatar
Unas
Admin / Site programmer
Posts: 8850
Joined: Tue Jul 10, 2007 4:43 pm
Gender: Male
Spoken languages: Français, English, Español
Contact:

Re: [S] Site shutdowns

Post by Unas »

Sorry for the late answer.

Thanks for proposing your help.
Normally, money is not so much of an issue : I try to be conservative with it, but on paper the server where I moved on June 30th is more than powerful enough for running the site, and still cheap enough for me to afford without thinking twice. AAO shouldn't need a more expensive hosting - and really, all my monitoring shows that the site uses but a small fraction of the server's processing power and memory.

The issue here is that I have no idea left as to what causes these crashes.
- Ever since they started occurring, I never was able to find logs corresponding to the crash - everything seems perfectly normal until nothing runs anymore.
- They keep occurring on the new server, even though I changed system architecture and the version of most software involved, doubled the RAM and tripled the processing power, so it shouldn't be a hardware issue - or so I'd hope.
- They occur seemingly randomly : sometimes a long time after the previous reboot, sometimes a very short time (eg. I was forced to reboot the server on July 9th for a short time, but on July 10th we had yet another severe crash... but nothing since I rebooted it on July 11th), and monitoring does not show any issue regarding memory usage, so it cannot be a memory leak, or more generally a RAM issue.
- The new server is a VM, not a physical machine, so it cannot be a processor overheat (otherwise it would take down the whole hypervisor, affecting dozens of other clients, and the host would reboot it)
- The kernel is configured to reboot automatically in case of kernel panic, yet it doesn't... So the crash seems severe enough to bypass even the most basic kernel recovery systems.
- I also have other sites operating on other servers with the same setup (in fact, one is identical to the previous AAO server), and the others don't have this issue. They have much less traffic as well, but aside from that are pretty much identical... So this would seem to indicate it's specific to AAO, either because of the amount of traffic, or something in the site code deployed on AAO itself.

But I'm still stuck on what kind of application bug (if it's an issue within the AAO site or forum code) could crash the server so severely that it doesn't even go through kernel panic... And why it didn't occur on the previous server (the one before April 2017's move), where everything was pretty much the same - and we had more traffic back then as well.

So yeah, right now I'm keeping a close eye on it, but it's still a mystery to me :-/


And no, so far it doesn't look like there is a risk of data loss. In any case, I have weekly backups being exported so I should be able to recover most contents in case of issues.
ImageImageImage
If knowledge can create problems, it is not through ignorance that we can solve them.
Si le savoir peut créer des problèmes, ce n'est pas l'ignorance qui les résoudra. ( Isaac Asimov )
User avatar
Kroki
Admin
Posts: 7475
Joined: Fri Nov 23, 2007 10:05 pm
Spoken languages: Français, English, Español, 日本語
Contact:

Re: [S] Site shutdowns

Post by Kroki »

Seems AAO's been mining bitcoins in secret...
ImageImage Image Image Image
User avatar
Jofe
Posts: 14411
Joined: Fri Jun 10, 2011 1:44 am
Gender: Male
Spoken languages: Français [100%], English [70%]

Re: [S] Site shutdowns

Post by Jofe »

Unfortunelly, it looks like it's back. :?
EDIT: Ok, so AAO hates Saturday, now. xD
Image Image Image Image Image
Super legenda
Posts: 860
Joined: Mon Sep 11, 2017 8:10 pm
Gender: Male
Spoken languages: Español e Ingles

Re: [S] Site shutdowns

Post by Super legenda »

It was down again.
Super legenda
Posts: 860
Joined: Mon Sep 11, 2017 8:10 pm
Gender: Male
Spoken languages: Español e Ingles

Re: [S] Site shutdowns

Post by Super legenda »

Two days in a row?
User avatar
Enthalpy
Community Manager
Posts: 5169
Joined: Wed Jan 04, 2012 4:40 am
Gender: Male
Spoken languages: English, limited Spanish

Re: [S] Site shutdowns

Post by Enthalpy »

Yeah... It's been bad lately. Here's the last I heard from Unas, a few days back:
Unas wrote: FYI, as I told you in a previous mail, over the last two or three weeks I received serval mails informing me of issues on the physical machine that is hosting my AAO VM.
Apparently, this incident I reported was enough to tip them that it was too much, and they are going to migrate me on another physical host (should take place on 24/05, in two days). Let's hope it works better after that...
We are hopeful that things will be a lot better now. If not... I'll let Unas know.
[D]isordered speech is not so much injury to the lips that give it forth, as to the disproportion and incoherence of things in themselves, so negligently expressed. ~ Ben Jonson
Post Reply