APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Sunday October 05, 2026 (SJD 61319) ---- Observing Team ---- Dmitry Bizyaev ---- Observing Plan ---- Mon 05 Oct 2026 61319 0.21 01:35 01:49 11:53 19:45 05:50 --- dark (10.1 h) plus 0.2 h for 12 deg design ---- Observing Summary ---- The night was lost to high humidity and cloudy sky. Note that we have problems with sdss5-hub server that is being solved by Shane. The problem is a show stopped and it makes observations impossible, but we wouldn't observe anyway because of the weather. We can see that the statusof APOGEE is good, according to UVa monitoring. The status of BOSS LN2 system should be good, according to Jose's estimate. ---- Engineering Issues ---- Comments from Shane: The HUB VM went down this afternoon during/after the Veeam backup process, and because HUB provides the /data volume used by the observing systems, observing is currently blocked. I've isolated the problem to the large /data virtual disk and its VMware snapshot chain. The VM itself will boot without that disk, so the OS and the rest of the VM appear to be okay. The original /data disk chain is still intact, but VMware is hanging when it tries to attach it to the VM. I was able to use Veeam Instant Recovery to mount this afternoon's backup of the /data disk on a separate recovery VM. I verified the filesystem, ran a read-only filesystem check, and spot-checked the data; everything looks good there. I have now started migrating that recovered disk back onto the cluster onto a new recovery datastore I created on the SAN. The disk is 22 TB in size with roughly 18 TB of actual data, so this is a large migration and it is moving slowly. ... ---- Observing Log ---- Afternoon ========= Bacause the sdss5-hub is unavailable, most of our control software cannot ne run. ---- Telescope Status ---- Not opened. No interlocks bypassed. The slip detector is running on a spare power supply. Status at: 09:10Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 31 ? LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 125 lbs, 22 psi FPS Status ========== Unknown ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 61318 Fill night time in Hours below *=*=*= Total bright time : 0.2 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0.2 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 10.1 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 10.1 *=*=*= Dark time lost to technical issues : 0 _______________________________________________ Logs of Nightly APO Activities https://mailman.sdss.org/archives/list/sloan-25mlog@sdss.org/message/X2JT3TYTA2YOJWJUX4JXOEYCHSBZHSB7/