Date: Fri, 20 May 2016 17:29:16 +0000
From: Jon Brinkmann
Subject: Re: [sloan-25mlog 2978] 2.5m obslog 20160519 (SJD 57528)
A technical correction:
The hub was not connected to the telemetry actor at all.  However, it was
unhappy that there was code for it in the tron product.  Hence, reverting
to v3.2, which has no telemetry actor code, took care of the problem.
Jon
On Fri, May 20, 2016 at 11:51:24AM +0000, gfreischlad@apo.nmsu.edu wrote:
>
>                        APACHE POINT OBSERVATORY SDSS 2.5M
>
>                                  OBSERVING LOG
>                           Thursday May 19, 2016
>
>                                   (SJD 57528)
>
> ----  Observing Team ----
>
> Dan Oravetz
> Gordon Freischlad
>
> Nathan (evening phone support)
> Kaike (evening phone support)
> Jon Brinkmann (evening software support)
>
> ---- Observing Summary ----
...
> Notes:
> -Apogee disk warnings triggered in the afternoon and continued into the evening.
> /data was losing space at around 1GB per minute. Nathan called and had JonB look
> into it. Jon found that hub had not connected to the telemetry actor properly
> and was spewing errors that had ballooned the log files for the last two days.
> He reverted hub back to v3_2 after the first cart change and compressed the
> large log files. There is now 1.4TB free in /data. Lost 10 minutes.
> -After restarting hub the apogee archiving behaved sporadically and then stopped
> altogether. Towards the end of the night we tried restarting apogeeql which
> seemed to fix things. Apogee images started archiving and processing again. The
> following Apogee images will need to be archived today:
> 19660019-19660021
> 19660028-19660029
> 19660034-19660072
...
>                                    NIGHT
> ============================================================================
> apogeeql.archDiskAlarm (Enum('Warning'), Int(671 GB))
>
>  Apogeeql disk warning triggered again bhile we were waiting for the fill to
> complete before opening. This time it read:
> apogeeql.archDiskAlarm (Enum('Warning'), Int(411 GB))
>  Checking the available /data space we found only 377GB down from the 643GB
> recorded during afternoon checkout.
>
>  During the first cart of the night Nathan called and asked JonB to look into
> the rapidly disappearing disk space.
>  Jon found that hub had not connected to the telemetry actor properly and was
> inflating the log files with error messages for the last two days.
>  Jon reverted hub back to v3_2 from v3_3 during cart change and compressed the
> log files from yesterday and today to free up space.
>  After the hub reboot the hartmann and gcamera actors had to be manually
> restarted by us. Took a bias and Apogee short dark to make sure.
> 03:59:17  apogee.expose=apRaw-19660022, 3, A, Dark
> 04:10:05  boss exposure    0.0, file=221375, C