[S] Site shutdowns

Post about what you like and dislike on AAO and suggest new features and improvements.

Moderator: EN - Forum Moderators

Forum rules

Re: [S] Site shutdowns 

Postby Unas » Thu Jul 19, 2018 12:34 am

Sorry for the late answer.

Thanks for proposing your help.
Normally, money is not so much of an issue : I try to be conservative with it, but on paper the server where I moved on June 30th is more than powerful enough for running the site, and still cheap enough for me to afford without thinking twice. AAO shouldn't need a more expensive hosting - and really, all my monitoring shows that the site uses but a small fraction of the server's processing power and memory.

The issue here is that I have no idea left as to what causes these crashes.
- Ever since they started occurring, I never was able to find logs corresponding to the crash - everything seems perfectly normal until nothing runs anymore.
- They keep occurring on the new server, even though I changed system architecture and the version of most software involved, doubled the RAM and tripled the processing power, so it shouldn't be a hardware issue - or so I'd hope.
- They occur seemingly randomly : sometimes a long time after the previous reboot, sometimes a very short time (eg. I was forced to reboot the server on July 9th for a short time, but on July 10th we had yet another severe crash... but nothing since I rebooted it on July 11th), and monitoring does not show any issue regarding memory usage, so it cannot be a memory leak, or more generally a RAM issue.
- The new server is a VM, not a physical machine, so it cannot be a processor overheat (otherwise it would take down the whole hypervisor, affecting dozens of other clients, and the host would reboot it)
- The kernel is configured to reboot automatically in case of kernel panic, yet it doesn't... So the crash seems severe enough to bypass even the most basic kernel recovery systems.
- I also have other sites operating on other servers with the same setup (in fact, one is identical to the previous AAO server), and the others don't have this issue. They have much less traffic as well, but aside from that are pretty much identical... So this would seem to indicate it's specific to AAO, either because of the amount of traffic, or something in the site code deployed on AAO itself.

But I'm still stuck on what kind of application bug (if it's an issue within the AAO site or forum code) could crash the server so severely that it doesn't even go through kernel panic... And why it didn't occur on the previous server (the one before April 2017's move), where everything was pretty much the same - and we had more traffic back then as well.

So yeah, right now I'm keeping a close eye on it, but it's still a mystery to me :-/


And no, so far it doesn't look like there is a risk of data loss. In any case, I have weekly backups being exported so I should be able to recover most contents in case of issues.
ImageImageImage
If knowledge can create problems, it is not through ignorance that we can solve them.
Si le savoir peut créer des problèmes, ce n'est pas l'ignorance qui les résoudra. ( Isaac Asimov )
User avatar
Unas
Admin / Site programmer
 
Posts: 8803
Joined: Tue Jul 10, 2007 4:43 pm
Gender: Male
Spoken languages: Français, English, Español

Re: [S] Site shutdowns 

Postby Kroki » Thu Jul 19, 2018 1:00 am

Seems AAO's been mining bitcoins in secret...
ImageImage Image Image Image
User avatar
Kroki
Admin
 
Posts: 7462
Joined: Fri Nov 23, 2007 10:05 pm
Spoken languages: Français, English, Español, 日本語

Re: [S] Site shutdowns 

Postby Jofe » Sun May 12, 2019 3:12 am

Unfortunelly, it looks like it's back. :?
EDIT: Ok, so AAO hates Saturday, now. xD
Image
Sprites, rearrangements, evidences and 3D stuff
Last Update: November 15th 2019 / Dernière mise à jour: 15 Novembre 2019
User avatar
Jofe
 
Posts: 14231
Joined: Fri Jun 10, 2011 1:44 am
Gender: Male
Spoken languages: Français [100%], English [60%]

Re: [S] Site shutdowns 

Postby Super legenda » Tue May 21, 2019 3:46 pm

It was down again.
Super legenda
 
Posts: 565
Joined: Mon Sep 11, 2017 8:10 pm
Gender: Male
Spoken languages: Español e Ingles

Re: [S] Site shutdowns 

Postby Super legenda » Wed May 22, 2019 5:05 pm

Two days in a row?
Super legenda
 
Posts: 565
Joined: Mon Sep 11, 2017 8:10 pm
Gender: Male
Spoken languages: Español e Ingles

Re: [S] Site shutdowns 

Postby Enthalpy » Sat May 25, 2019 3:51 pm

Yeah... It's been bad lately. Here's the last I heard from Unas, a few days back:

Unas wrote:FYI, as I told you in a previous mail, over the last two or three weeks I received serval mails informing me of issues on the physical machine that is hosting my AAO VM.
Apparently, this incident I reported was enough to tip them that it was too much, and they are going to migrate me on another physical host (should take place on 24/05, in two days). Let's hope it works better after that...


We are hopeful that things will be a lot better now. If not... I'll let Unas know.
[D]isordered speech is not so much injury to the lips that give it forth, as to the disproportion and incoherence of things in themselves, so negligently expressed. ~ Ben Jonson

Current AAO Development Priority: Issue #94: Grayscale Mode
User avatar
Enthalpy
Community Manager
 
Posts: 4584
Joined: Wed Jan 04, 2012 4:40 am
Gender: Male
Spoken languages: English, limited Spanish

Previous

Return to Comments and Ideas

Who is online

Users browsing this forum: No registered users and 2 guests