Date: Tue, 25 Nov 2003 12:12:00 GMT From: snedden@apo.nmsu.edu (Stephanie Snedden) Subject: 2.5m obslog 20031124 (MJD 52968) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday November 24, 2003 (MJD 52968) ---=== OBSERVING TEAM ===--- swing: Atsuko Kleinman night: Howard Brewington Steph Snedden ---=== OBSERVING PLAN ===--- Science! ---=== OBSERVING SUMMARY ===--- Lots of clouds and radar echoes. Plate 1107 completed. ---=== OBSERVING LOG ===--- Afternoon ------------ Initial try of "initNight -tapeLoad" failed. The second try, with new tape in id1-1, worked. I am not convinced that the tape is the problem since usually when one hs a bad tape, all other drives but the drive with the bad tape load successfully. But today, all drives didn't load. Error messages, tb messsages, watcher messages related to this are in the peoblem section. While doing the doghouse bias, both of the dr1 drives shutdown. Steph put in cleaning tape cartridge in just in case. (I did clean all drives at the beginning of the dark run, but it never hurts to clean it again.) The watcher errors associated with this is in the problem section. I followed the "Re-enabling two failed drives" instruction in Don's DA trouble shooting page. But the drive id1-1 shut down again after archResume (again the watcher messages related to this is in the problem section) I then did the "Re-enable single failed drives" and id1-1 was happy. I ran another bias to make sure all is well and it was. Steph completed the telescope checkout while I was busy with collimation and id1 drives. Night: ----- Partly cloudy at sunset with promises of more to come. Spectroscopy. Around 00:25Z Atsuko saw id5 and id6 complain about no gangs in watcher. Although we were just doing a bias, this usually means that TMS died. She did not see any TMS activities for few minutes so she restarted a teamster. After the new teamster was started, Atsuko saw gangs getting transferred in murmurlog. 00:55Z Cartridge 2, plate 1107 Stars on slew. It took two attempts to get the guider loop to work. The first time, it took one image and stopped. The weather deteriorated while we observed this plate, but we did manage to finish it in a total exposure of 2400s (it already had some signal). 02:13Z We closed. Although we could see stars overhead, there was a radar return to the west moving straight toward us, and more clouds were forming to the east in the radar shadow blind spot. We decided to play it safe and close. 02:49Z While closed, we decided to adjust the sp1 collimator mirror tilt. XMAX was out of spec. This is the third time we have had to adjust the tilt this dark run that I (Steph) know of. The new collimator actuator values are at the end of the spectro focus section. We never reopened. Clouds and radar echoes stayed with us all night Spectro focus ------------- 5C -- collimate told us to refocus. Before -------- sp1 Coll_motor_A -6524 Coll_motor_B -6525 Coll_motor_C -13243 sp2 Coll_motor_A -5991 Coll_motor_B -5993 Coll_motor_C -5991 After ------ sp1 (after mechPiston -4233) Coll_motor_A -10757 Coll_motor_B -10757 Coll_motor_C -17476 sp2 (after mechPiston -1906) Coll_motor_A -7897 Coll_motor_B -7897 Coll_motor_C -7896 After adjusting tilt for XMAX: ----------------------------- sp1: A=-9757 B=-9757 C=-18477 sp2: A=-7897 B=-7897 C=-7896 ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- 4281 23:15Z 23:36Z 100 O 121.49 126.69 49 ignore 4282 00:00Z 00:22Z 100 O 132.90 138.37 50 ignore ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ------------------------------------------------------------------------ ---=== SPECTROSCOPY DATA SUMMARY ===--- Summary Checked (y/n):y QA Procedures Done (y/n ):y UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 23447, plate -9999 ------- 22:47 23447 0.0 bias ----- sequence 23448, plate 1107 ------- 23:03 23448 10.0 flat 23:05 23449 2.0 arc 23:11 23450 2.1 arc 23:14 23451 2.1 arc 23:33 23452 2.1 arc 23:35 23453 2.1 arc 23:48 23454 0.0 bias ----- sequence 23454, plate 1107 ------- 01:03 23455 10.0 flat 01:05 23456 2.0 arc 01:28 23457 900.1 target 01:31 23458 10.0 flat 01:33 23459 2.0 arc 02:08 23460 1500.1 target 02:40 23461 10.0 flat Tweak of sp1 collimator tilt 02:57 23462 10.0 flat XMAX was out of spec 03:11 23463 10.0 flat 03:14 23464 2.0 arc Check of focus after tilt adjust 03:18 23465 2.1 arc 03:20 23466 2.1 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ 01:15Z 2 0 -15.79 0.0018 56.20 0.0009 166.71 0.0199 0.999960 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL7032 Stays: JL7033 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 22:51Z 2 0 1.00000 0 1257 0 121 30.2 3.6 12 247 01:15Z 2 0 0.99996 395 1639 150 -16 56.2 4.1 16 265 - 1.9 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 22:44Z 38 2 12 242 (WSW) 284 - mcl 23:16Z 38 -9 8 264 (W) 159 - " 23:47Z 36 -11 12 261 (W) 177 - " 00:21Z 38 -9 17 268 (W) 111 - " 00:53Z 39 -8 16 245 (WSW) 139 - " 01:26Z 39 -8 15 258 (WSW) 134 - " 01:58Z 38 -9 18 259 (W) 131 - " 02:30Z 37 -10 21 262 (W) 159 - " 03:02Z 37 -10 19 245 (WSW) 132 - " 03:32Z 36 -11 20 258 (WSW) 177 - " 04:04Z 35 -12 24 272 (W) 197 - " 04:35Z 34 -13 20 270 (W) 197 - " 05:08Z 35 -12 21 266 (W) 235 - " 05:40Z 36 -11 24 270 (W) 173 - " 06:13Z 36 -11 19 282 (WNW) 187 - " 06:44Z 35 -11 19 281 (W) 222 - " 07:17Z 36 -11 24 271 (W) 197 - " 07:49Z 35 -11 24 265 (W) 231 - " 08:22Z 36 -4 25 289 (WNW) 302 - " 08:53Z 36 -4 21 285 (WNW) 407 - " 09:26Z 37 -3 21 286 (WNW) 309 - " 09:58Z 37 -2 16 269 (W) 383 - " 10:30Z 35 1 15 268 (W) 371 - " 11:02Z 35 8 12 268 (W) 367 - " 11:34Z 34 11 20 272 (W) 333 - " ---=== TELESCOPE STATUS ===--- 00:00Z Fans & blowers on. 00:35Z Opened enclosure 02:12Z Closed enclosure - threatening weather. 11:38Z Blowers off. Doors closed. Telescope stowed at 30 deg alt with cartridge 6 mounted. Counterweights at 270. Spectro auto-fill connected. 11:40Z Spectro dewar weight: 180 11:40Z Imager dewar weight: 206 ---=== SOFTWARE USED ===--- IOP/SOP: v3_121_0 Watcher: v2_24_0 MCP: v5_23_0 TPM: tpm_v2_36_0 AstroDa: v14_47 TCC: TCC 2.7.1 July 22 2003 sdssProcedures: v1_79 SoS: v4_10_6 hoggPT: v1_6_9 plate-mapper: v4_3_1 ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.5500 0.5450 Axial B 0.6390 0.6370 Axial C 1.0860 1.0850 Trans D -8.4640 -8.4550 Lateral E 10.7950 10.7442 Lateral F 10.8077 10.7823 GALILS Commanded: 4600. -2850. 850. -9250. 18400. 17850. Actual: 4600. -2866. 866. -9275. 18411. 17825. SETMIR VALUES PriDesOrient: 0.00 0.00 15.20 829.90 1.60 PriOrient: 0.00 0.00 14.91 830.49 3.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 3.2 Alt.: 30.235956 MIGS TONIGHT NOMINAL Axial A 1.5160 1.5150 Axial B 1.1080 1.1100 Axial C 1.1180 1.1150 Trans D -0.8270 -0.8260 GALILS Commanded: 1628663. 1548184. 1574345. -3900. -8750. Actual: 1628641. 1548187. 1574504. -3899. -8726. SETMIR VALUES SecDesOrient: 1257.00 0.00 -25.00 0.00 146.27 SecOrient: 1256.96 0.06 -25.02 0.25 146.52 ---=== PROBLEMS IN DETAIL ===--- initNight failed ------------------- The first "initNight -tapeLoad" failed with the following message. It looked like id1-1 was the problem, but watcher showed that all drives had no tapes loaded, but id1-1. But after a few min, id1-1 showed "no tapes". This was confirmed by the archDrvSts. I decided to swap the id1-1 tape and try again. 2003-11-24 22:47:01Z: tapeWaitFor reports success Waiting for command response from id1 (crate 1, ICC 0), archiver server (client 2) Waiting for command response from id1 (crate 1, ICC 0), archiver server (client 2) 2003-11-24 22:47:38Z: IOP Not ready (blew up on pad): tapeMount: Request sense failure Failed to receive exabyte command completion status invoked from within "iccExec $tapeList archReadSense 1" invoked from within "set sense1 [iccExec $tapeList archReadSense 1]..." (procedure "tapeRewind" line 5) invoked from within "tapeRewind $tapeList " invoked from within "if {$doRewind} { tapeRewind $tapeList }" (procedure "tapeLoad" line 5) invoked from within "tapeLoad $tapeList $doRewind $doUnload" im> im> tb Request sense failure Failed to receive exabyte command completion status invoked from within "iccExec $tapeList archReadSense 1" invoked from within "set sense1 [iccExec $tapeList archReadSense 1]..." (procedure "tapeRewind" line 5) invoked from within "tapeRewind $tapeList " invoked from within "if {$doRewind} { tapeRewind $tapeList }" (procedure "tapeLoad" line 5) invoked from within "tapeLoad $tapeList $doRewind $doUnload" while executing "tapeMount" im> Watcher complained the following: --------------------------------------------- dsc I archFailMsg Rimfire failure ID = 1 Status = 0x0 0x0 (dsc_I_archFailMsg Rimfire failure ID = 1 Status = 0x0 0x0 : 2003-11-24 22:49:52Z ) dsc E archWrFail Tape write error while archiving frame 4281 6.21 (dsc_E_archWr Fail Tape write error while archiving The second attempt to load the tape got the following message in watcher, but the loading succeed. archLastErr-2 (Drive 2 error: Unit Attention: Reset Occured: 2003-11-24 23:03:17 Z) Watcher message about the id1 failure during bias ------------------------------------------------------------ dsc E trcErr dscArch CheckPrevOp: rstatus = 0x8001421c, drive 1 (dsc_E_trcErr d scArch_CheckPrevOp: rstatus = 0x8001421c, drive 1 : 2003-11-24 23:18:48Z ) dsc E archShutDown1 Archiver shutting down first drive (dsc_E_archShutDown1 Arc hiver shutting down first drive : 2003-11-24 23:18:48Z) dsc E archWrFail Tape write error while archiving frame 4281 6.21 (dsc_E_archWr Fail Tape write error while archiving dsc E trcErr RMF unit options sense failure (dsc_E_trcErr RMF_unit_options_sens e failure : 2003-11-24 23:25:13Z) deadDisk archUseDrv (drive 2: archUseDrv is 0: 2003-11-24 23:19:41Z) dsc E trcErr dscArch CheckPrevOp: rstatus = 0x8001421c, drive 1 (dsc_E_trcErr d scArch_CheckPrevOp: rstatus = 0x8001421c, drive 1 : 2003-11-24 23:18:48Z ) dsc E trcErr dscArch CheckPrevOp: rstatus = 0x8001421c, drive 2 (dsc_E_trcErr d scArch_CheckPrevOp: rstatus = 0x8001421c, drive 2 : 2003-11-24 23:18:59Z ) dsc E archStsFail Failed to receive exabyte command completion status (dsc_E_ar chStsFail Failed to receive exabyte command completion status : 2003-11-24 23:21:38Z ) archEnable (archEnable is 0: 2003-11-24 23:19:41Z) dsc E archShutDown1 Archiver shutting down first drive (dsc_E_archShutDown1 Arc hiver shutting down first drive : 2003-11-24 23:18:48Z) startDa -scd dsc E archNotInited Archiver is not initialized (dsc_E_archNotInited Archiver i s not initialized : 2003-11-24 23:25:13Z) dsc E archShutDown2 Archiver shutting down second drive (dsc_E_archShutDown2 Ar chiver shutting down second drive : 2003-11-24 23:18:59Z) After recovering both drives, id1-1 failed again after archResume was issued. ------------------------------------------------------------------------------ dsc E archWrFail Tape write error while archiving frame 4281 6.21 (dsc_E_arc Fail Tape write error while archiving dsc E trcErr RMF unit options sense failure (dsc_E_trcErr RMF_unit_options_s e failure : 2003-11-24 23:46:25Z) archUseDrv (drive 1: archUseDrv is 0: 2003-11-24 23:44:15Z) dsc E trcErr dscArch CheckPrevOp: rstatus = 0x8001421c, drive 1 (dsc_E_trcEr scArch_CheckPrevOp: rstatus = 0x8001421c, drive 1 : 2003-11-24 23:46:25Z ) dsc E trcErr dscArch CheckPrevOp: rstatus = 0x8001421c, drive 2 (dsc_E_trcEr scArch_CheckPrevOp: rstatus = 0x8001421c, drive 2 : 2003-11-24 23:46:25Z ) dsc E archStsFail Failed to receive exabyte command completion status (dsc_E chStsFail Failed to receive exabyte command completion status : 2003-11-24 23:46:25Z ) dsc E archShutDown1 Archiver shutting down first drive (dsc_E_archShutDown1 hiver shutting down first drive : 2003-11-24 23:46:25Z) dsc E archNotInited Archiver is not initialized (dsc_E_archNotInited Archive s not initialized : 2003-11-24 23:46:25Z) dsc E archShutDown2 Archiver shutting down second drive (dsc_E_archShutDown2 chiver shutting down second drive : 2003-11-24 23:46:25Z) archDrvSts (drive 2: Error: 2003-11-24 23:40:49Z)