Date: Wed, 14 Mar 2001 07:22:49 -0600 From: "Eric H. Neilsen, Jr." Subject: 2.5-m Obslog 20010313 (MJD 51982) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday March 13, 2001 (MJD 51982) OBSERVING TEAM: =============== Atsuko Kleinman (APO) Eric Neilsen (FNAL) Mike Harvanek (APO) PT Alan Uomoto (JHU) PT Scot Kleinman (APO) VISITORS: ========= THE PLAN: ========= Science. SYNOPSIS: ========= The camera reboots itself shortly before 6:52pm. We believe this is the usual post-reboot reboot. We begin with spectroscopy, plate 548, cartridge 2. SoS reports some red monster and scattered light problems, and we are forced to close due to weather before finishing the plate. We took an all night bias with the imager. CCD33 begins reading zero values, jumps back and forth between high and low values a few times, eventaully landing near 750- counts and dropping erratically to ~4000 counts by the end of the night. Watcher continued to generate ignorable complaints, though not the usual communications errors. (Did anything change from last night?) See the "Problems in detail" section. GENERAL COMMENTS (see Problems below for details): ================================================== We open the doors and fiducialize, and turn on the cone fan around 6:45. The imager rebooted itself shortly before Wed Mar 14 01:52:21 UTC 2001. After the reboot, we did a readoutAbort, iack, and set the voltages on. Looking at the scrolling displays, imager nodes except id6 start scrolling! We issue an endDrift, and they all stop. The watcher then begins complaining of no connection to the imager. MakeRunlog completes successfully for the run in which the camera rebooted (run 2175). We then start another drift, and get the following: im> goDrift -bias camera linestarts differ from last run camera is now 26322 # MICROSECONDS was last at 26343 # MICROSECONDS Is this ok? [no/yes]: yes We remove the enclosure at Wed Mar 14 02:10:52 UTC 2001. (9:00PM MST) We begin doing spectroscopy with cartridge 2, plate 548. Except for the weather, this proceeds smoothly. Exposure 8986 and 8991, SoS complains of a "red monster" on b1. The hand paddle is definitely off. R2, which has the lowest signal to noise, has scattered light problem and 8992 and 8993. We are eventually close because the clouds are thick enough that we cannot get reasonable data, and there is rain to the south of the sight. When we add up the signal to noise with reasonable signal to noise, we only reach a signal to noise of 10, so we cannot declare this plate done. We the following error in an IOP window connected to the TCC: UserNum=9; MsgsLost=1787; Text="output resumes" UserNum=9; MsgsLost=1; Text="user mailbox full; no more errors reported for this user until output resumes" There does not seems to be any problems with the TCC; it has not slowed down, and is behaving normally. We notice this in the murmur log: Mar 13 20:01:05 sdsshost IOP 502969 TEXTONLY Couldn't open udp socket 30410 : Address already in use This repeats throughout the night, although nothing seems not to be functioning properly because of this. We notice in the low-light camera that there is a bright light from under the telescope. It is tracked down to a red LED on the guider CCD camera. We tape the LED. SoS complains of exposure 8992 r2 data having a scattered light problem. The clouds finally seem to break around 4:00, just too late to reopen. Bummer. Around 5:00, we note the the display for CCD35 is solid blue. The watcher shows that this CCD is happy, though, so we assume this is a astroline display quirk. DATA ACQUISITION (see IOP/SOP output below): =========================================== IMAGING: -------- Run Stripe Strip Flavor Lambda Last Frame # Begin End ------------------------------------------------------------------ 2175 100 O bias -98.91 -89.61 62 2176 100 0 bias all night, left running SPECTROSCOPY: ------------- UT Exp Time flavor comment =============================================== ------ sequence 8981, plate -9999 ------- 01:22 8981 0.0 bias {{}} ------ sequence 8982, plate 548 ------- 02:54 8982 10.0 flat pre-flat 02:56 8983 2.0 arc pre-arc 03:15 8984 900.0 target science 03:38 8985 900.0 target science 03:57 8986 901.0 target science 04:16 8987 901.0 target science 04:25 8988 240.0 smear {{Smear Pattern}} 04:28 8989 10.0 flat post-flat 04:30 8990 2.0 arc post-arc 05:15 8991 901.0 target science 05:40 8992 900.0 target science 06:00 8993 901.0 target science 06:18 8994 900.0 target science 06:21 8995 10.0 flat post-flat 06:23 8996 2.0 arc post-arc OFFSETS: Time Cart Plate getclock Az Alt Rot Scale 20:39 imager 984541150 0.005391 -0.001882 -0.005373 1.000000 TAPES: Goes to FNAL: JL3738 Stays at APO: JL3739 MIRROR NUMBERS ============== PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0289 0.0150 Axial B: -0.0670 -0.0510 Axial C: 0.0219 0.0300 Trans D: 3.7550 3.7560 Lateral E: 1.9084 1.9100 Lateral F: 0.4090 0.4095 GALILS Commanded: 4890. -2755. 465. -6642. 8737. 8322. Actual: 4900. -2750. 450. -6650. 8750. 8300. SETMIR VALUES primDesOrient: 0.00 -4.40 10.70 586.50 240.70 primOrient: 0.00 -4.56 10.53 586.58 240.03 SECONDARY: ---------- Focus: 150.0 Air temp: 1.7 C Altitude: 69.29 MIGS TONIGHT Axial A: -1.3240 Axial B: -1.3300 Axial C: -1.3420 Trans D: 0.3000 GALILS Commanded: 1616850. 1616850. 1616850. 9200. 10400. Actual: 1616827. 1616991. 1616997. 9200. 10400. SETMIR VALUES secDesOrient: 1282.47 0.00 0.00 0.00 -327.39 secOrient: 1282.84 -0.02 0.06 0.04 -326.55 FOCUS LOG: ========== setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm 20:40 1 548 1.00000 0 1121 0 163 76.8 1.3 14 235 WEATHER SYNOPSIS ================ Partly cloudy in the beginning of the night. The clouds increase until we are forced to close, and then clear around 4:00am. WEATHER DATA: ============= Wind Time Temp F Dewp F MPH Direction Dust 16:01 42 18 7 257 (WSW) 440 partly cloudy 17:01 38 17 11 267 (W) 545 partly cloudy 18:01 37 17 8 254 (WSW) 564 partly cloudy 19:01 35 16 6 196 (SSW) 864 clear 20:01 35 16 10 233 (SW) 915 mostly clear 21:01 34 17 12 224 (SW) 1147 cloudy 22:01 33 18 13 236 (SW) 890 partly cloudy 23:01 32 19 7 266 (W) 846 partly cloudy 00:01 33 19 11 293 (WNW) 568 cloudy 01:01 32 19 8 254 (WSW) 588 cloudy 02:01 31 20 9 241 (WSW) 541 cloudy 03:01 31 20 5 272 (W) 617 cloudy 04:01 30 20 12 303 (WNW) 563 clear 05:01 29 20 11 332 (NNW) 525 clear LN2 SCHEDULE: ============= Spectrographs filled at ~7:30. Spectro dewar weight 154 lb at 05:56. Imager dewar weight 177 lb at 05:56. TELESCOPE STATUS: ================= Stowed at 30 deg alt, cartridge 1 on telescope. Spectro auto-fill connected. INTERLOCKS: =========== none bypassed SOFTWARE USED: ============== IOP/SOP: v3_36_10 Watcher: v2_9_15 MCP: v2_5_0 TPM: v2_3_0 AstroDa: v14_36 TCC: 2.5.1 PROBLEMS IN DETAIL: =================== id3 and id6 complained of the not-ready to ready transition: archLastErr-1 (Drive 1 error: Unit Attention: Not Ready To Ready Transition: Mar 13 18:14:37 MST archLastErr-2 (Drive 2 error: Unit Attention: Not Ready To Ready Transition: Mar 13 18:14:37 MST archLastErr-1 (Drive 1 error: Unit Attention: Not Ready To Ready Transition: Mar 13 18:14:37 MST archLastErr-2 (Drive 2 error: Unit Attention: Not Ready To Ready Transition: Mar 13 18:14:37 MST Telescope/Alt-Aa-Rot watcher, when telescope in enclosure, stop button in, MCP has the semaphore, TCC does not: TCCStatus:alt (axis halted by a problem: Mar 13 17:41:26 MST) AxisStat:alt (semCmdPort_taken: Mar 13 17:34:51 MST) AxisDTime:alt (0.9: Mar 13 17:46:33 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 17:34:51 MST) TCCStatus:rot (axis halted by a problem: Mar 13 17:41:26 MST) TCCStatus:az (axis halted by a problem: Mar 13 17:41:26 MST) AxisDTime:az (0.9: Mar 13 17:46:33 MST) AxisStat:rot (out_closed_loop amp_bad semCmdPort_taken: Mar 13 17:34:51 MST) AxisDTime:rot (0.9: Mar 13 17:46:33 MST) AxisStat:alt (out_closed_loop amp_bad stop_in: Mar 13 18:38:17 MST) AxisDTime:alt (0.9: Mar 13 18:38:17 MST) AxisStat:az (out_closed_loop amp_bad stop_in: Mar 13 18:38:17 MST) AxisDTime:az (0.9: Mar 13 18:38:17 MST) AxisStat:rot (out_closed_loop amp_bad stop_in: Mar 13 18:38:17 MST) AxisDTime:rot (0.9: Mar 13 18:38:17 MST) The imager rebooted! Watcher complained of the following on id1: dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000004 rowIdx:1011 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000004 rowIdx:1011 : Mar 13 18:48:51 MST ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000003 rowIdx:1011 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000003 rowIdx:1011 : Mar 13 18:48:51 MST ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000002 rowIdx:1011 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000002 rowIdx:1011 : Mar 13 18:48:51 MST ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000001 rowIdx:1011 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000001 rowIdx:1011 : Mar 13 18:48:51 MST ) dsc E trcErr WRONG num of pix?? 1112 should be 2128 (dsc_E_trcErr WRONG num of pix?? 1112 should be 2128 : Mar 13 18:48:51 MST ) dsc W trcWarn msgQReceive err, TMO assumed, errno=0x3d0004 (349 interrupts (dsc_W_trcWarn msgQReceive err, TMO assumed, errno=0x3d0004 (349 interrupts : Mar 13 18:49:11 MST ) dsc E trcErr (throttle) WRONG num of pix?? 1112 should be 2128 (dsc_E_trcErr (throttle) WRONG num of pix?? 1112 should be 2128 : Mar 13 18:48:51 MST ) dsc E trcErr (throttle) WRONG num of pix?? for (sr<<16)|ccd:0x00000000 ro (dsc_E_trcErr (throttle) WRONG num of pix?? for (sr<<16)|ccd:0x00000000 ro : Mar 13 18:48:51 MST ) dsc E trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000005 rowIdx:1011 (dsc_E_trcErr WRONG num of pix?? for (sr<<16)|ccd:0x00000005 rowIdx:1011 : Mar 13 18:48:51 MST ) Similar watcher errors occurr for other nodes. Critical errors: phase_boot im exec_boot im Get more Alt-Az-Rotator Drive Motor compaints (telescope still in enclosure, stopped. etc.): AxisStat:alt (out_closed_loop amp_bad stop_in: Mar 13 18:55:43 MST) AxisDTime:alt (0.9: Mar 13 18:55:43 MST) AxisStat:az (out_closed_loop amp_bad stop_in: Mar 13 18:55:43 MST) AxisDTime:az (0.9: Mar 13 18:55:43 MST) AxisStat:rot (out_closed_loop amp_bad stop_in: Mar 13 18:55:43 MST) AxisDTime:rot (0.9: Mar 13 18:55:43 MST) Some more errors (due to camera reboot?) phase micro (UNACKNOWLEDGED: Mar 13 18:51:05 MST) exec micro (UNACKNOWLEDGED: Mar 13 18:51:05 MST) We also get a "rowsSkipped" astroline compaint. I assume this is due to the camera reboot. Again, we get Alt-Az-Rot errors: AxisStat:alt (out_closed_loop amp_bad stop_in: Mar 13 19:07:26 MST) AxisDTime:alt (0.9: Mar 13 19:07:26 MST) AxisStat:az (out_closed_loop amp_bad stop_in: Mar 13 19:07:26 MST) AxisDTime:az (0.9: Mar 13 19:07:26 MST) AxisStat:rot (out_closed_loop amp_bad stop_in: Mar 13 19:07:26 MST) AxisDTime:rot (0.9: Mar 13 19:07:26 MST) During opening (exact telescope state unknown, but there were not problems in using it to open): AxisStat:alt (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 19:13:19 MST) AxisDTime:alt (0.9: Mar 13 19:13:19 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 19:13:19 MST) AxisDTime:az (0.9: Mar 13 19:13:19 MST) AxisStat:rot (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 19:13:19 MST) AxisDTime:rot (0.9: Mar 13 19:13:19 MST) While setting up for spectroscopy: TCCStatus:alt (axis halted by a problem: Mar 13 19:26:38 MST) AxisInit:az (axis initialization failed: Mar 13 19:26:38 MST) AxisInit:alt (axis initialization failed: Mar 13 19:26:38 MST) AxisStat:alt (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 19:19:18 MST) AxisDTime:alt (0.9: Mar 13 19:19:18 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 19:19:18 MST) TCCStatus:rot (axis halted by a problem: Mar 13 19:26:38 MST) TCCStatus:az (axis halted by a problem: Mar 13 19:26:38 MST) AxisInit:rot (axis initialization failed: Mar 13 19:26:38 MST) AxisDTime:az (0.9: Mar 13 19:19:18 MST) AxisStat:rot (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 19:19:18 MST) AxisDTime:rot (0.9: Mar 13 19:19:18 MST) We get a TCC watcher error (unusual): Text="pointing error probe does not exist:check inst data": (Mar 13 19:38:16 MST) While taking spectra: AxisDTime:alt (-0.6: Mar 13 19:59:34 MST) AxisDTime:rot (-0.5: Mar 13 19:56:29 MST) In telescope/TCC Facil="SYSMSG":VMSMsg:"NOMSG" (Mar 13 19:51:30 MST) Alt-Az-Rotator drive motors: AxisDTime:alt (-0.4: Mar 13 20:20:47 MST) While taking spectra: AxisDTime:alt (-0.4: Mar 13 20:41:09 MST) AxisDTime:az (-0.3: Mar 13 20:41:09 MST) AxisDTime:rot (-0.5: Mar 13 20:58:22 MST) After the telescope is put away (due to weather): TCCStatus:alt (axis halted by a problem: Mar 13 23:37:22 MST) AxisStop:alt (axis stopped: Mar 13 23:37:22 MST) AxisStat:alt (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 23:39:48 MST) AxisDTime:alt (-0.4: Mar 13 21:18:47 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 23:39:48 MST) TCCStatus:rot (axis halted by a problem: Mar 13 23:37:22 MST) AxisStop:rot (axis stopped: Mar 13 23:37:22 MST) TCCStatus:az (axis halted by a problem: Mar 13 23:37:22 MST) AxisDTime:az (-0.4: Mar 13 23:20:44 MST) AxisStat:rot (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 13 23:39:48 MST) AxisStop:az (axis stopped: Mar 13 23:37:22 MST) AxisDTime:rot (-0.4: Mar 13 21:26:39 MST) AxisStat:alt (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 02:13:57 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 02:13:57 MST) AxisStat:rot (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 02:13:57 MST) AxisStat:alt (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:00:44 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:00:44 MST) AxisStat:rot (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:00:44 MST) AxisStat:alt (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:06:33 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:06:33 MST) AxisStat:rot (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:06:33 MST) AxisStat:alt (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:12:33 MST) AxisStat:az (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:12:33 MST) AxisStat:rot (out_closed_loop amp_bad stop_in semCmdPort_taken: Mar 14 04:12:33 MST) etc. DATA LOGS FROM IOP/SOP: =======================