Date: Fri, 2 Feb 2001 07:27:12 -0700 From: Dan Long Subject: 2.5m obslog 20010201 (MJD 51942) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday February 1, 2001 (MJD 51942) OBSERVING TEAM: =============== Dan Long (APO) Jurek Krzensinski (APO) Bing Chen (JHU) PT Craig Loomis (APO) by phone VISITORS: ========= THE PLAN: ========= Science. SYNOPSIS: ========= Despite frost scare and guider troubles 3 plates were done ( 301, 483, 491 ). GENERAL COMMENTS (see Problems below for details): ================================================== Cart 7 412 was observed less than 15 deg from the half full moon. I think we've shown tonight through SoS results, that this is too close. We closed around 23:00 when we found frost on the floor, doghouse, counterweight covers, and heavy frost on flat field screen. During this time the dewpoint sensor on the 3.5m registered between 3 and 4 degrees C depression. Fortunately no frost was found on the optics. The 3.5m had only light frost in front of the shutters at this time. In calm winds such as we had during this period, relying on a dewpoint sensor 30 feet above ground ( and a couple hundred feet away ) when our telescope is at ground level ( at least to the east ) could be asking for trouble. We opened about half an hour later when the dewpoint depression increased a couple more degrees. There were some odd occurrences involving the TCC either directly or indirectly tonight. Sporadically throughout the night the TCC would send out messages about messages being lost ( usually from T_STATUS ). It didn't seem to effect the ability to send commands. The problems section has more details, and sample output. Once tonight the TRACK process died. It was quickly restarted, but did impact a science exposure on plate 483 ( 8514 ). See problems section. Shortly after midnight we started having guider communication problems. This ended up costing the only science exposure we got on plate 487, as it was almost entirely taken without guiding ( we got (S/N)^2 of 3 though ... better than 412 ). Basically after several power cycles of the guider and/or its ICC, with some short periods where it looked like things were working again, the problem was narrowed to the Paralans. We ended up finding that the computer room Paralan had failed, and swapped it with the spare. It cost us about 1:45 total time. Still, we got two plates done after it was fixed. If you want the gory details, see the problem section. Finally when closing, I noticed odd messages in the murmur log during the slew. The azimuth was repeatedly complaining of an invalid bar code message following a message from "barcodecan" saying port 3 was canceled. Murmur log output is in the problem section. DATA ACQUISITION (see IOP/SOP output below): =========================================== IMAGING: -------- Run Stripe Strip Flavor Lambda Last Begin End Frame # Comments ------------------------------------------------------------------ 2092 100 O ignore -167.00 -145.56 157 ( bias in dog house ) SPECTROSCOPY: ------------- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ------ sequence 8497, plate 412 ------- 2.1 2.9 2.8 4.2 23:23 8497 0.0 bias {{}} 01:22 8498 0.0 bias {{}} 01:50 8499 2.0 arc pre-arc 01:54 8500 10.0 flat pre-flat 01:56 8501 2.0 arc pre-arc 02:15 8502 900.0 target science 02:33 8503 900.0 target science 02:55 8504 900.0 target science 03:12 8505 900.0 target science 03:19 8506 240.0 smear {{Smear Pattern}} 03:26 8507 10.0 flat post-flat 03:28 8508 2.0 arc post-arc ------ sequence 8509, plate 483 ------- 19.3 24.5 21.2 20.4 03:57 8509 10.0 flat pre-flat 03:59 8510 2.0 arc pre-arc 04:18 8511 900.0 target science 04:35 8512 900.0 target science 04:53 8513 900.0 target science 05:11 8514 900.0 target science {{}} {{telescope stopped tracking during exposure}} 05:29 8515 900.0 target science 05:39 8516 240.0 smear {{Smear Pattern}} 05:44 8517 10.0 flat post-flat 05:46 8518 2.0 arc post-arc ------ sequence 8519, plate 487 ------- 3.3 2.9 2.6 2.8 07:12 8519 10.0 flat pre-flat 07:14 8520 2.0 arc pre-arc 07:32 8521 900.0 target science telescope stopped guiding during exp. ------ sequence 8522, plate 491 ------- 28.2 20.4 25.3 18.5 09:14 8522 0.0 bias {{}} 09:24 8523 10.0 flat pre-flat 09:26 8524 2.0 arc pre-arc 09:47 8525 900.0 target science 10:06 8526 900.0 target science 10:23 8527 900.0 target science 10:41 8528 900.0 target science 10:59 8529 900.0 target science 11:06 8530 240.0 smear {{Smear Pattern}} 11:09 8531 10.0 flat post-flat 11:11 8532 2.0 arc post-arc ------ sequence 8533, plate 301 ------- 28.4 16.3 19.7 20.5 11:35 8533 10.0 flat pre-flat 11:37 8534 2.0 arc pre-arc 11:56 8535 900.0 target science 12:14 8536 900.0 target science 12:32 8537 900.0 target science 12:50 8538 900.0 target science 12:57 8539 240.0 smear {{Smear Pattern}} 13:01 8540 10.0 flat post-flat 13:03 8541 2.0 arc post-arc Plate 412 was observed in sequence(s) 8497 for a total of 3600 seconds Plate 483 was observed in sequence(s) 8509 for a total of 4500 seconds Plate 487 was observed in sequence(s) 8519 for a total of 900 seconds Plate 491 was observed in sequence(s) 8522 for a total of 4500 seconds Plate 301 was observed in sequence(s) 8533 for a total of 3600 seconds OFFSETS: getclock Calib offsets Scale Time Ct Plate time Az El Rot Factor 18:45 7 412 981078355 0.000096 0.001715 0.007347 0.999500 21:00 1 483 981086418 0.001142 -0.001255 -0.083659 0.999940 00:10 3 487 981097828 0.003690 -0.000617 -0.051406 0.999900 02:31 8 491 981106319 0.001057 -0.000430 -0.061615 0.999810 04:33 9 301 981113631 0.000320 0.001628 0.007893 0.999880 TAPES: Goes: JL3595 Stays:JL3596 MIRROR NUMBERS ============== PRIMARY: -------- Scale: 1.000000 -------- MIGS TONIGHT SHOULD BE ( from 12/19 ) Axial A: 0.0299 0.0150 Axial B: -0.0359 -0.0510 Axial C: 0.0450 0.0300 Trans D: 3.7560 3.7560 Lateral E: 1.9075 1.9100 Lateral F: 0.4090 0.4095 GALILS Commanded: 4900. -2750. 450. -6650. 8750. 8300. Actual: 4900. -2750. 450. -6650. 8750. 8300. SETMIR VALUES primDesOrient: 0.00 -4.40 10.70 586.50 240.70 primOrient: 0.00 -4.56 10.53 586.58 240.03 SECONDARY: ---------- Focus: 0.0 Air temp: -5.9 C Altitude: 29.99 MIGS TONIGHT Axial A: -1.2910 Axial B: -1.3270 Axial C: -1.3029 Trans D: 0.1930 GALILS Commanded: 1583850. 1583850. 1583850. 2600. 3800. Actual: 1583731. 1583794. 1583725. 2600. 3800. SETMIR VALUES secDesOrient: 1257.00 0.00 0.00 0.00 -175.00 secOrient: 1256.93 0.02 0.02 0.01 -175.13 FOCUS LOG: ========== setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt C MPH Dir filt fwhm " 16:20 Doors open, fan on 17:30 Enclosure open 19:10 7 412 0.99950 4935 5501 250 -20 55.5 -8.4 12 73 - 4.1 19:46 7 412 0.99955 4443 5178 325 -34 52.0 -8.9 9 84 - 3.2 21:00 1 483 0.99994 593 1873 200 141 49.7 -6.5 0 185 - 2.1 22:17 1 483 0.99987 1284 2418 200 150 59.0 -8.2 7 1 - 2.3 22:55 Enclosure closed 23:40 Enclosure opened 00:11 3 487 0.99990 988 2154 180 167 58.4 -6.3 9 87 - 3.0 00:15 Guider Trouble 02:00 Guider Fixed 02:32 8 491 0.99981 1875 2736 20 180 56.0 -5.4 4 105 - 2.5 03:28 8 491 0.99981 1875 2739 20 189 55.0 -5.8 8 107 - 2.2 04:40 9 301 0.99988 1185 2163 20 14 56.4 -5.4 0 219 - 2.2 05:19 9 301 0.99984 1579 2458 -10 -4 57.1 -5.6 1 247 - 2.2 05:47 9 301 0.99980 1974 2767 -30 -16 56.0 -5.6 1 348 - 2.1 06:25 Enclosure closed, fan off WEATHER SYNOPSIS ================ It was clear all night with light and variable winds, and very variable dewpoint! WEATHER DATA: ============= Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 16:01 26 16 0 251 (WSW) 492 - Clear 17:01 24 15 4 256 (WSW) 588 - Clear 18:01 20 9 11 80 (E) 336 - Clear 19:01 17 9 11 82 (E) 380 - Clear 20:01 18 10 2 96 (E) 459 - Clear 21:01 20 12 2 342 (NNW) 656 - Clear 22:01 19 12 6 358 (N) 612 - Clear 23:01 20 12 0 331 (NNW) 576 - Clear 00:01 20 4 8 84 (E) 481 - Clear 01:01 23 -23 6 91 (E) 549 - Clear 02:01 21 -18 4 108 (ESE) 697 - Clear 03:01 22 -26 1 119 (ESE) 1016 - Clear 04:01 22 -24 4 89 (E) 887 - Clear 05:01 22 -20 1 353 (N) 914 - Clear 06:01 23 -26 4 0 (N) 821 - Clear 07:01 23 -30 4 301 (WNW) 943 - Clear LN2 SCHEDULE: ============= Spectrograph dewar 06:00 Dewar wt 81 lb Imager dewar 06:00 Dewar wt 88 lb TELESCOPE STATUS: ================= Stowed at 30 deg alt, Cartridge 7 installed. Spectro auto-fill connected. INTERLOCKS: =========== none bypassed SOFTWARE USED: ============== IOP/SOP: v3_25_4 Watcher: v2_8_14 MCP: v4_4_2 TPM: v2_2_1 AstroDa: 14_34 TCC: 2.4.1 PROBLEMS IN DETAIL: =================== TCC/SOP communication troubles. sp2> fk5 LST: 43.3749 Slewing to an fk5 at 48.193496 -1.196078 (J2000) Tracking to 48.11964121113882, -0.55123610155333336 I think LST is: 43.3749 Text="process T_STATUS lost 1 messages that could not be sent to the message hub" I think that we haven't started slewing To kill this message, type 'schedule watchTCCSlew {} 0' 0 sp2> RHL Modu axe_i_SchMove:{axe_i_SchMove} RHL Modu axe_i_SchMove:{axe_i_SchMove} I think that the slew has started timed out while writing to hub message mailbox" Text="process T_STATUS lost 1 messages that could not be sent to the message hub" Text="process _TNA874: lost 1 messages that could not be sent to the message hub" Text="process T_STATUS lost 2 messages that could not be sent to the message hub" I think that the slew has started I think that the slew has started A telstop/telrun seemed to stop the complaints upon slewing to an fk5 star, but the messages about lost messages continued to appear sporadically throughout the night. At no time did these messages correspond to any inability of SOP to send a command to the TCC. ------------------------------------------------------------------- T_TRACK dying: Shortly before 22:00 the telescope stopped tracking and the following messages began popping up in SOP and the TCC. DeadProc="T_TRACK" DeadProc="T_TRACK" DeadProc="T_TRACK" RHL Modu exe_Track:{exe_Track} /data/spectro/guider/gimg0776.fits Getting out of TELRUN, issuing RESTART TRACK, going back into telrun, and issuing TRACK, got us going again quickly. The problem did not recur. Unfortunately this happened during a science exposure, and while it had useful signal to noise, it should be considered suspect when it comes to the reduction. ------------------------------------------------------------------- Guider Troubles. The guider stopped working. It started out with SOP sending gcam doread commands and the guider never returning with images. After a couple of these, the TCC sent back messages saying it couldn't abort the exposure, but that the connection timed out with an error code of ppk 86. At first the Paralan in the computer room would show a red error LED. Other times it would flash the busy light for 30 seconds before a response would come. We power cycled the guider and everything worked for about 15 minutes, then the same error started occurring. This time after power cycling the guider we couldn't even send an init without it failing. However the Paralan in the computer room was no longer showing its error light. We called Craig and he gave us some hints on how to track down the problem. We checked the Paralan in the guider box, and the SCSI converter as well. The SCSI converter only had its green power LED ( its orange error LED was not lit ). The Paralan however had a red error LED. It stayed red even when we swapped fibers at both ends ( ruling out the fibers ). Next we got the spare Paralan from the computer room and swapped out the one in the guider box. It too displayed a red error LED. This led us to suspect that it was the transmit end of the one in the computer room which was bad. So we took the one we had just removed from the guider and swapped it for the Paralan in the computer room. Success!! Green LEDs only, at both ends. This solved the guider problem, although SOP apparently got confused with all these communication errors, because it stopped being able to grab the latest image even though they were being written to host. StopMon and startMon got it to display the latest image again once, but then stopped again. Exiting SOP and restarting it cured that problem. Back on the sky after 1:45 of fiddling. ------------------------------------------------------------------- Fiducial reading problem: b 02 06:05:11 sdssmcp tLatch dsc_I_trcInfo err = -99 poserr = -11 ticks Feb 02 06:05:11 sdssmcp tLatch dsc_I_trcInfo Applying correction 11 to rotator Feb 02 06:05:12 sdssmcp tLatch dsc_I_trcInfo Intermediate rot fiducial 60.39 deg Feb 02 06:05:13 sdssmcp barcodcan dsc_I_trcInfo canceled port 3 (status=0x0) Feb 02 06:05:14 sdssmcp tLatch dsc_E_trcErr Invalid barcode in azimuth: fididx = 12288, pos = 1098026337 Feb 02 06:05:14 sdssmcp barcodcan dsc_I_trcInfo canceled port 3 (status=0x0) Feb 02 06:05:14 sdssmcp tLatch dsc_S_trcSucc failed to read latch position: status = 0 axis 0x8 Feb 02 06:05:15 sdssmcp tLatch dsc_E_trcErr Failed to identify rotator fiducial at 53.15 Feb 02 06:05:16 sdssmcp tLatch dsc_I_trcInfo Intermediate rot fiducial 50.91 deg Feb 02 06:05:18 sdssmcp tLatch dsc_I_trcInfo rot fiducial 68 (48.41) deg Feb 02 06:05:18 sdssmcp tLatch dsc_I_trcInfo err = -280 poserr = -30 ticks Feb 02 06:05:18 sdssmcp tLatch dsc_I_trcInfo Applying correction 30 to rotator Feb 02 06:05:19 sdssmcp tLatch dsc_I_trcInfo Intermediate rot fiducial 46.18 deg Feb 02 06:05:20 sdssmcp barcodcan dsc_I_trcInfo canceled port 3 (status=0x0) Feb 02 06:05:21 sdssmcp tLatch dsc_E_trcErr Invalid barcode in azimuth: fididx = 12288, pos = 1098266843 Feb 02 06:05:21 sdssmcp barcodcan dsc_I_trcInfo canceled port 3 (status=0x0) Feb 02 06:05:22 sdssmcp tLatch dsc_S_trcSucc failed to read latch position: status = 0 axis 0x8 Feb 02 06:05:22 sdssmcp tLatch dsc_E_trcErr Failed to identify rotator fiducial at 38.93 Interesting SOP message after slew: I think we got there before I knew we started ???