Subject: Re: [sloan-25mlog 2141] 2.5m obslog 20150228 (SJD 57082)
From: Fritz Stauffer
Date: Sun, 1 Mar 2015 12:44:15 -0700
On Mar 1, 2015, at 10:03 AM, Kaike Pan  wrote:
> Thank you for looking into this.
>=20
> I strong agree with you, you need to find a way to restart systems =
much faster.
I am hoping that SDSS will work with Jon and I to test restarting the =
computers.  It=92s just been a low priority because there have been so =
many other bigger issues.
>=20
> As JohnP pointed out, we had issues with computers two night in rows, =
we need to
> understand what caused these failures.  We appreciate developers=92 =
efforts in improving=20
> our systems. But I would like that to  say, please implement changes =
(include updating=20
> systems) during engineering time.=20
There was nothing changed.  Last night=92s failure was the computer =
freezing up for either OS or hardware reasons.  The night before, it was =
a bad bug in the script that tracks the apc ups run time.  I am hoping =
to have this fixed ASAP.
>=20
> I recall that we switch to VM system because it is more reliable and =
faster to restart. So
> far, we have not seen this, can we do more research on how to get =
where we plan for.=20
> On this, we suggest that we run a test on how to bring up systems if =
the VM system is
> down during one engineering time.=20
>=20
> Cheers,
>=20
> -kaike
>=20
>> On Mar 1, 2015, at 8:15 AM, Fritz Stauffer  =
wrote:
>>=20
>>=20
>> On Mar 1, 2015, at 7:21 AM, John K. Parejko  =
wrote:
>>=20
>>> Could we please get a bit more follow up on what happened here, and =
last night? Two server restarts in two nights is two too many.
>>=20
>> This comment is uncalled for.
>>=20
>> The previous night, Jon had a script trying to determine if the site =
power was failing, and it powered off 2 servers.  I asked Jon to put a =
human in the loop so the machines don=92t automatically shut down.
>>=20
>> There is another problem going on with the user network that there =
are periods when something goes awry and the users cannot access the =
network.  This causes confusion, but, it doesn=92t stop observations.  =
We are still analyzing the problem, but it is not an easy one.
>>=20
>> The failure last night looks like a hardware instability on the VM =
host.  I checked the monitoring logs, and there is no indication of =
anything amiss.  I have seen the servers completely freeze before, when =
I was first setting them up.  I found that interrupt remapping was not =
reliable, which leads to freeze ups.  I have also seen freeze ups due to =
OS instabilities.  I would call a freeze up after 6 months of running a =
pretty stable system.  I will look for any BIOS updates that improve =
stability.  The latest OS was booted last night, so maybe that will fix =
the freeze that we saw.
>>=20
>> The restart would have happened much faster if the startup scripts =
had been tested and kept up to date.  Also, we had an OS update on 2/22, =
which was the system that was booted up.  The problem we discovered, is =
that the DRBD module, which is an add-on, was not updated and the new =
kernel was missing the DRBD driver.  Now, when we see an OS update, we =
will update the DRBD driver.  The system came back up surprisingly =
quickly considering this was the first time a recovery was tried.
>>=20
>> -fs
>>=20
>>>=20
>>> Why didn't the hub, actors and other software start up =
automatically?
>>>=20
>>> Thanks,
>>> John
>>>=20
>>> On 1Mar 2015, at 10:04, gfreischlad@apo.nmsu.edu wrote:
>>>=20
>>>>                    APACHE POINT OBSERVATORY SDSS 2.5M
>>>>=20
>>>>                              OBSERVING LOG
>>>>                       Saturday February 28, 2015
>>>>=20
>>>>                               (SJD 57082)
>>>>=20
>>>> ERROR
>>>> 05:03:34 Started the Apogee Manga Sequence. STUI to disconnect =
entirely.
>>>> The sdss4 server locked up and had an update which gave it =
configurations that
>>>> made it unable to reconnect. Jon Brinkmann and Fritz gave this =
explanation and
>>>> got it running again. The aftermath of the situation took very long =
to fix. Most
>>>> of the actors had to be restarted manually. The /data and /home =
disks had to be
>>>> remounted. Many terminal server ports had to be manually =
disconnected. Petunia's web
>>>> server went down and couldn't be brought back up so Conor set us up =
a local
>>>> version to use for the rest of the night. SOS went down and had to =
be restarted.
>>>> DOS went down and was not restarted.=20
>>>>=20
>>>=20
>>> --
>>> *************************
>>> John Parejko
>>> john.parejko@yale.edu
>>> http://www.astro.yale.edu/~jp727/
>>> 203 432-9759
>>> JWG 465
>>> Department of Physics
>>> Yale University
>>> New Haven, CT
>>> **************************
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>>=20
>>=20
>> ____________________________________________________________
>> Logs of Nightly APO Activities
>> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html
>=20