Subject: 3.5m Night Log 20081030 Date: Fri, 31 Oct 2008 13:39:01 +0000 (UTC) From: bketzeba@apo.nmsu.edu Apache Point Observatory 3.5m Telescope Night Log Thursday, October 30, 2008 ACTUAL ASTRONOMER OBSERVER(S) INSTRUMENT START FINISH -------------------------------------------------------------------- A.Becker A.Becker Agile 05:30 00:55 S.Moran S.Moran DIS 01:05 INST SCHED(TWILIGHT) WEATH EQUIP OBS DAY NOT USED I/ROW/DN START FINISH D B D B D B USE D B -------------------------------------------------------------------- UW02/R 19:08 01:00 0 0 0.4 0 5.4 0 0 0 0 JH02/R 01:00 06:26 0 0 0 0 5.4 0 0 0 0 note: scheduled times listed include instrument change time ------------- ACTIVITY LOG -------------- Observing Specialist: B.Ketzeback Clear skies with good to moderate seeing. The observer encountered a few problems with Agile aquisition tonight. At 19:05 the observer reported that the instrument acquisition was locked up and required a control-c to exit. Before the user quit aacq we both confirmed that nimble was alive and could be accessed through other sessions. After the control-c a new aacq could not be started. Uninstalling and reinstalling the kernel driver rspiusb solved the problem and aacq could be restarted. Again at 22:50 the observer exited a running aacq session using control-c. This led to a lock up of nimble. The machine could not be accessed and pinging was unresponsive. A power cycle was performed. An attempted restart of aacq reported the error message: "Anti_Blooming functions failed pl_get_param, with error code 116" Again unloading and reloading the device driver corrected the problem however the reboot caused newton to report a Stale NFS handle. Not wanting to waste much more time looking up the recovery proceedure in the apo wiki I rebooted newton without thinking that this would again crash the aacq when the observer lost his connection to nimble. A few more minutes and the observer was again taking data. About 20 minutes total were lost to both problems. bout an hour later the observer reported an unusual problem with hi data that only manifested itself after the reboot of newton, discussed above. Images in /export/nimble/2008oct30/o69888/ were written as identical pairs. The problem seems to manifest itself as 2 images being the same, then the next 2 are the same, then the next 2, etc. Meaning that sequence numbers 00001 and 00002 were identical, 00003 and 00004 were identical, etc. The exposure time between images was set to 45s. The cause is unknown however it seems likely somehow related to the reboot of newton although newton has no connection to the agile data aquisition. It just remotely mounts nimbles disk and acts as a gateway for the outside world to telnet/ssh into nimble. It was not clear from the user if the data for the remainder fo the night was not ususable (2 hours lost), only half usuable (1 hour lost) or fine. At this time we will assume the data was fine and no additional downtime is being counted. A PR was filed and the observer sent email to Anjum and Russell. I ran the autofocus script for much of first half until the gmech ran out of focus range and reported the following error: "Failed: gmech focus -10258... failed: Bad actuator status" Second half was quiet. No other problems seen the remainder of the night. ------- FAILURE LOG ------- PROG INST FAILURE MODE TIME (SEDFNVOG) TI/SHU START FINISH DESCRIPTION ---------------------------------------------------------------------- UW08 P I/SH 19:05 19:15 AACQ locked up UW08 P I/H 22:52 23:08 Nimble crash ---------------- FOCUS LOG --------------- Time Instrument Focus Az El Temp Strc Prim Sec Air filt FWHM ----------------------------------------------------------------- 18:53 AGILE -421( -449) 1 64 10.9 13.7 12.2 11.8 11.4 I 0.79 19:26 NA2gdr 3001(14001) -17 63 10.0 12.7 11.1 10.8 10.7 clear 1.14 22:50 AGILE -329( ) -4 57 9.6 9.8 9.7 9.0 9.6 I 0.83 23:15 AGILE -274( -495) -15 56 9.1 9.6 9.5 8.2 9.1 I 1.02 01:13 dis-slit -125(-75) -69 78 10.2 9.3 9.6 8.5 9.6 -- 0.97 ------------- WEATHER LOG ---------------- Wind Precip.or Time Temp F Dewp F MPH Direction Dust Sky Conditions ------------------------------------------------------------------- 16:00 56 38 6 207 (SSW) 1299 clear none 17:00 55 36 3 194 (SSW) 1120 clear none 18:00 52 36 1 212 (SSW) 1087 clear none 19:00 51 25 1 274 (W) 738 clear none 20:00 50 21 2 347 (NNW) 812 clear none 21:00 49 18 6 359 (N) 1148 clear none 22:00 49 18 11 28 (NNE) 721 clear none 23:00 49 16 5 358 (N) 1442 clear none 00:00 50 22 2 196 (SSW) 1152 clear none 01:00 50 18 1 335 (NNW) 1250 clear none 02:00 50 25 1 192 (SSW) 1354 clear none 03:00 50 20 1 247 (WSW) 1313 clear none 04:00 50 21 3 316 (NW) 1397 clear none 05:00 50 21 1 350 (N) 1327 clear none 06:00 51 20 5 79 (E) 1295 clear none Note: the wind was coming from the azimuth listed. The convention used is north=0 degrees, east=90 degrees. The dust count is particles > 1u per 0.1 cubic feet. TELESCOPE STATUS Telescope drives operational. Current TCC version: 2.9.1 Current TUI version: TUI 1.5.1