Date: Wed, 24 Oct 2001 06:54:01 -0600 (MDT) From: prn@apo.nmsu.edu (Pete Newman) Subject: 2.5m obslog 20011023 (MJD 52206) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday October 23, 2001 (MJD 52206) ---=== OBSERVING TEAM ===--- Swing, night: Jurek Krzesinski, Pete Newman (APO) Visitors: Eric Neilsen, Rob Sparks (FNAL) ---=== OBSERVING PLAN ===--- Spectroscopy until moonset (00:18), then imaging. ---=== OBSERVING SUMMARY ===--- Clear all night but poor seeing. Finished two new standard plates (718, 720) and got good data on one more (722). Probably finished a photo-z plate (672). Incomplete re-observations of plate 547. Several old software problems revisited, only 1 troublesome. New software problems: maybe 2. FF screens aborted alt axis on closing. All axis offsets within expected ranges. ---=== OBSERVING LOG ===--- Excellent clear skies (no clouds between here and the Pacific!) and west winds, but highly variable seeing (1.5 -- 4.5 arcsec), so spectroscopy all night. SoS reported many scattered light warning tonight for no apparent reason. Holloman Airforce Base has actually turned off many of its maintenance pad lights. None of the CCD temperatures were out of spec according to watcher. The warnings continued after moonset. We may have seen another instance of PR2817 when the SP2 LN2 fill after opening did not stop for a long time. However, after physicalyl handling the fill connectors, the fill DID stop automatically some pounds later. The sky camera stalled at 19:33 and was restarted ok about 20:30. Messages in the murmur log and watcher from sdssmcp reported corrections too large, but did NOT mention for which axis. At the same time as the last set seen before writing this, there were good correction applied to all three axes (around 22:50 MDT). We wonder if this is a bug in the MCP debugging code. See problems section #1 for examples. We had one instance of PR2437 (TCC/MCP read time-out) during plate 671. See problem section #2 for details. SoS failed on the postCals for plate 671. See problems section #3 for details of the recovery. The IOP watcherVersion command (and logTool) reported the wrong version of watcher. PR raised on IOP. The flat-field screens (we think - as PR 2709) caused the alt axis to abort at about 16 deg when descending to close. Pulling on each in turn to force to motors to run solved it. Observing sequence: 19:19 Plate 718, cartridge 1. Done in 3 exposures tonight when combined with exposured from MJD 52204. 21:24 Plate 720, cartridge 2. Done in 3 exposures. 22:51 Plate 722, cartridge 3. Not done in 2 exposures, as beyond design time, but good data. Smear taken under excellent conditions. 00:30 Plate 671 (photo-z), cartridge 4. Done in 6 exposures (assuming weaker exposures are summable). Target (S/N)^2 = 40 (the general target for photo-z plates is 30, with one plate at 40). Minimum measured by SoS = 42.1. Axes halted a minute into the first exposure, so exposure aborted. Lost about 3 minutes recovering. Remaining exposures seem good, but the B camera S/N values are poorly determined. See http://sos.apo.nmsu.edu/snplot-52206-0671.ps which shows too few objects with g' around 20.2 mag for SoS to fit the S/N. The i' S /N values from the same plot look reliable. We reported this in more detail in an email to sdss-obs, sdss-galaxies and sdss-photoz (sdss-obs message 2594). 03:53 Plate 547, cartridge 8. (S/N)^2 >= 7.7 in 3 exposures. Reobservation to fill in for lack of unobserved end-of-night plates. Seeing highly variable 2-4 arcsec. ---=== IMAGING RUN SUMMARY ===--- Run Stripe Flavor Lambda Last Begin End Frame Comments ------------------------------------------------------------------ 2674 100 O ignore 85.05 101.49 86 doghouse bias ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot --------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields nGood rowMean rowSig colMean colSig rot --------------------------------------------------------------- ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 11460, plate -9999 ------- 22:54 11460 0.0 bias ----- sequence 11461, plate -9999 ------- 01:08 11461 0.0 bias ----- sequence 11462, plate 718 ------- 20.3 14.6 24.4 21.1 DONE (Totals) 01:21 11462 10.0 flat 8.2 4.6 12.4 5.7 From MJD 52204 01:23 11463 2.0 arc 11.1 10.0 14.0 15.4 From tonight 02:00 11464 1201.0 target 02:26 11465 1203.0 target 02:47 11466 1103.0 target 02:54 11467 240.0 smear 02:57 11468 10.0 flat 02:59 11469 2.0 arc ----- sequence 11470, plate 720 ------- 13.6 13.4 17.9 17.4 DONE 03:09 11470 10.0 flat 03:11 11471 2.0 arc 03:41 11472 901.0 target 04:09 11473 1502.0 target 04:36 11474 1502.0 target 04:43 11475 240.0 smear ----- sequence 11476, plate 722 ------- 10.5 9.9 10.8 8.3 NOT DONE 04:54 11476 10.0 flat 04:56 11477 2.0 arc 05:42 11478 1800.0 target 06:14 11479 1800.0 target 06:21 11480 240.0 smear ----- sequence 11481, plate 671 ------- ??.? 42.1 ??.? 59.1 DONE 06:33 11481 10.0 flat 06:35 11482 2.0 arc 07:13 11483 1801.0 target 07:40 11484 1502.0 target 08:08 11485 1501.0 target 08:36 11486 1500.0 target 09:09 11487 1801.0 target 09:36 11488 1500.0 target 09:43 11489 240.0 smear 09:46 11490 10.0 flat 09:48 11491 2.0 arc ----- sequence 11492, plate 547 ------- 18.4 11.8 18.4 7.7 NOT DONE REOBS 09:57 11492 10.0 flat 09:59 11493 2.0 arc 10:39 11494 1501.0 target 11:07 11495 1500.0 target 11:40 11496 1801.0 target 11:46 11497 240.0 smear 11:49 11498 10.0 flat 11:51 11499 2.0 arc ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument getclock Az Alt Rot Scale ------------------------------------------------------------------------------ 19:52 1 718 1003888331 0.000938 -0.001692 0.051782 0.999890 20:29 1 718 1003890564 0.000938 -0.001692 0.051782 0.999830 21:28 2 720 1003894096 0.001824 -0.001527 -0.006785 0.999720 00:00 3 722 1003903220 0.001519 -0.002650 0.014453 0.999800 00:14 3 722 1003904086 0.001519 -0.002650 0.014453 0.999890 01:08 4 671 1003907280 0.001083 -0.000888 0.009762 0.999950 03:05 4 671 1003914322 0.001083 -0.000888 0.009762 0.999970 04:18 8 547 1003918737 0.000736 -0.001321 -0.035912 1.000200 ---=== TELESCOPE OFFSETS AND SCALE II ===--- ---=== DATA TAPE SUMMARY ===--- Goes: JL4572 Stays: JL4573 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ 19:52 1 718 0.99989 1085 2095 0 25 45.5 8.2 18 267 -- 1.9 20:29 1 718 0.99983 1678 2577 0 12 48.0 8.7 21 263 -- 1.6 21:28 2 720 0.99972 2764 3475 0 -4 48.4 8.1 18 286 -- 1.9 00:00 3 722 0.99980 1974 2850 -20 -47 35.2 7.0 4 270 -- 2.6 01:07 4 671 0.99995 493 1488 -80 4 57.3 7.2 11 278 -- 1.8 03:05 4 671 0.99997 296 1361 -70 -44 48.3 6.9 9 327 -- 2.5 04:18 8 547 1.00020 -1974 -568 -100 121 54.9 6.9 12 342 -- 2.6 ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 16:30 56 27 22 262 (W) 337 - Clear all night 17:01 52 26 10 254 (WSW) 325 - 17:33 52 26 23 251 (WSW) 410 - 18:05 50 25 15 271 (W) 557 - 18:38 48 23 17 263 (W) 676 - 19:09 47 24 10 250 (WSW) 660 - 19:39 47 23 16 249 (WSW) 572 - 20:10 46 23 20 266 (W) 495 - 20:40 47 23 24 269 (W) 437 - 21:12 46 23 16 267 (W) 404 - 21:42 46 24 27 283 (WNW) 750 - 22:12 47 24 17 265 (W) 696 - 22:44 46 24 18 273 (W) 527 - 23:17 45 25 11 280 (W) 470 - 23:50 44 25 10 269 (W) 549 - 00:22 43 25 12 277 (W) 439 - 00:52 43 24 8 277 (W) 313 - 01:26 45 24 8 292 (WNW) 294 - 01:58 44 23 8 278 (W) 271 - 02:30 44 23 14 289 (WNW) 371 - 03:01 44 22 14 313 (NW) 271 - 03:34 43 23 13 305 (NW) 233 - 04:07 44 23 19 308 (NW) 247 - 04:39 44 23 7 314 (NW) 384 - 05:09 44 23 14 353 (NNW) 385 - 05:40 44 23 16 358 (NNW) 358 - 06:12 44 24 7 347 (NNW) 318 - NB. Between 04:00 and 06:00, the wind gusts were strong. 06:48 44 23 12 17 (NNE) 354 - ---=== TELESCOPE STATUS ===--- Rotator slip light on at end of night, but box not reset at start of night. 17:00 Fans on, east door open, west door cracked. 18:30 Enclosure off 06:14 Enclosure on, fans off Telescope stowed at: 30 deg Instrument mounted: Cartridge 8 Counterweights at: 290 Interlocks disabled: - Building - Axis slip - West door safety strip switch (has been so for some days, but was not logged) . Instruments on autofill 180L LN2 dewar weights at 06:45 Spectro: 239 lb Imager: 85 lb ---=== SOFTWARE USED ===--- IOP/SOP: v3_58_2 Watcher: v2_12_2 MCP: v5_6_5 TPM: v2_4_2 AstroDa: v14_39 TCC: 2.5.5 sdssProcedures: v1_31, but using revised spectro procedures. ---=== MIRROR NUMBERS ===--- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A 0.0760 0.0710 Axial B 0.8030 0.8030 Axial C 0.0600 0.0640 Trans D -1.4950 -1.4590 Lateral E 1.8500 1.8600 Lateral F 1.4190 1.4280 GALILS Commanded: 5400. -3700. 900. -200. 31550. 30650. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES PriDesOrient: 0.00 -11.80 23.00 1256.90 642.10 PriOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.00 Air Temp.: 8.7 Alt.: 29.993373 MIGS TONIGHT NOMINAL Axial A 1.5700 1.5670 Axial B 1.2930 1.2900 Axial C 1.3220 1.3200 Trans D 0.2260 0.2310 GALILS Commanded: 1599643. 1566156. 1586213. -10400. -9750. Actual: 1599600. 1565250. 1586350. -10400. -9750. SETMIR VALUES SecDesOrient: 1257.00 -3.40 -10.00 0.00 133.82 SecOrient: 1257.21 -3.14 -9.78 -0.09 133.53 ---=== PROBLEMS IN DETAIL ===--- Problem #1 ''Correction too large'' errors without an axis specified During the night, we were seeing good fiducial corrections applied on all three axes, for example: 22:49:09 dsc_I_trcInfo az fiducial 15, delta = 172 22:49:09 dsc_I_trcInfo az fiducial 60.70 deg 22:49:09 dsc_I_trcInfo err = 10 poserr = -172 ticks 22:49:09 dsc_I_trcInfo Applying correction 172 to azimuth 22:49:09 dsc_S_trcSucc failed to read latch position: status = 0 axis 0x8 22:49:10 dsc_E_trcErr Failed to identify rotator fiducial at 65.12 22:49:12 dsc_I_trcInfo rot fiducial 64 (67.38) deg 22:49:12 dsc_I_trcInfo err = 285 poserr = 2 ticks 22:49:12 dsc_I_trcInfo Applying correction -2 to rotator 22:49:13 dsc_I_trcInfo Intermediate rot fiducial 69.86 deg 22:49:14 dsc_I_trcInfo alt fiducial 45.78 deg 22:49:14 dsc_I_trcInfo err = 19 poserr = 63 ticks 22:49:14 dsc_I_trcInfo Applying correction -63 to altitude 22:49:14 dsc_I_trcInfo rot fiducial 63 (72.12) deg 22:49:14 dsc_I_trcInfo err = 291 poserr = -4 ticks This has been the case for the last several nights, except for an azimuth slip on MJD 52204 (2001/10/21). However, we got some strange errors from the MCP section of the watcher indicating corrections too large, but not mentioning an axis. On checking back through the murmur logs, we found many. The examples below were generated on sdsshost by: cd /skycat/oplogs/murmurlog/ cat 5220?/murmur.1* | grep 'is too large' | grep -v 'correction for' | tail -50 We plan on rebooting the MCP tomorrow (52207) before operations to see if this clears the error message. Oct 23 03:10:35 sdssmcp TCC dsc_E_trcErr correction 37010 is too large (max 600) Oct 23 03:10:36 sdssmcp TCC dsc_E_trcErr correction 37010 is too large (max 600) Oct 23 05:21:09 sdssmcp TCC dsc_E_trcErr correction 37027 is too large (max 600) Oct 23 05:21:11 sdssmcp TCC dsc_E_trcErr correction 37027 is too large (max 600) Oct 23 05:21:12 sdssmcp TCC dsc_E_trcErr correction 37027 is too large (max 600) Oct 23 05:26:20 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 05:26:22 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 05:26:23 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:11:16 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:11:17 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:11:19 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:14:03 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:14:05 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:14:06 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:19:24 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:19:25 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:19:27 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:19:42 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:19:44 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 06:19:45 sdssmcp TCC dsc_E_trcErr correction 37264 is too large (max 600) Oct 23 17:29:46 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 17:29:47 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 17:29:49 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:08:43 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:08:45 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:08:46 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:21:08 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:21:09 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:21:11 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:54:27 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:54:28 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:54:30 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:56:44 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:56:46 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 18:56:47 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 19:02:02 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 19:02:04 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 19:02:05 sdssmcp TCC dsc_E_trcErr correction 37209 is too large (max 600) Oct 23 21:00:10 sdssmcp TCC dsc_E_trcErr correction 37127 is too large (max 600) Oct 23 21:00:11 sdssmcp TCC dsc_E_trcErr correction 37127 is too large (max 600) Oct 23 21:00:12 sdssmcp TCC dsc_E_trcErr correction 37127 is too large (max 600) Oct 23 21:03:54 sdssmcp TCC dsc_E_trcErr correction 37160 is too large (max 600) Oct 23 21:03:55 sdssmcp TCC dsc_E_trcErr correction 37160 is too large (max 600) Oct 23 21:03:57 sdssmcp TCC dsc_E_trcErr correction 37160 is too large (max 600) Oct 23 22:44:26 sdssmcp TCC dsc_E_trcErr correction 37026 is too large (max 600) Oct 23 22:44:27 sdssmcp TCC dsc_E_trcErr correction 37026 is too large (max 600) Oct 23 22:44:28 sdssmcp TCC dsc_E_trcErr correction 37026 is too large (max 600) Oct 23 22:48:08 sdssmcp TCC dsc_E_trcErr correction 37163 is too large (max 600) Oct 23 22:48:10 sdssmcp TCC dsc_E_trcErr correction 37163 is too large (max 600) Oct 23 22:48:12 sdssmcp TCC dsc_E_trcErr correction 37163 is too large (max 600) ----------------------------------------- Problem #2 - Axes halted: TCC/MCP time-out (PR2437 again) Murmur message as the guider crashes: Oct 24 00:46:46 sdsshost IOP 382391 TEXTONLY expected floating-point number but got ''NaN'' Relevant TCC messages aroudn that time 0 8 > Started; Cmd='offset rotator -0.0000185' 0 0 F Modu=''prt_Read''; Text=''read timed out on port LTA19:'' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''ROT +MOVE -0. 0000185 -0 .0000003''; Received=''ALT +MOVE 0.0000000 0.0000000'' 0 0 F Modu=''axe_o_Move''; Text=''command failed for ALT, ROT'' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''AZ MOVE 13. 8175950 -0. 0074302 24370.49000''; Received='' '' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''ALT MOVE 56. 5555692 0. 0008423 24370.49000''; Received=''AZ MOVE 1 3.8175950 -0.0074302 24370.49000'' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''ROT MOVE 191. 5325072 -0 .0061909 24370.49000''; Received='' '' 0 0 F Modu=''axe_o_Move''; Text=''command failed for AZ , ALT, ROT'' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''AZ MOVE''; Received='' ALT MOVE 56.5555692 0.0008423 24370.49000'' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''ALT MOVE''; Received='' '' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''ROT MOVE''; Received='' ROT MOVE 191.5325072 -0.0061909 24370.49000 '' 0 0 F Modu=''axe_o_Move''; Text=''command failed for AZ , ALT, ROT'' 0 0 F Modu=''prt_ReadMatch''; Text=''bad reply''; Expected=''AZ MOVE''; Received='' '' ------------------------------------------------- Problem #3 - SoS failed on plate 547 It became apparent that SoS was not reporting frames from the post-cals from plate 671. The sos.log file showed aporsync running for a long time. I logged in to observer@sos, and did the following to stop and restart SoS, whereupon it worked. top showed no processes going beforehand: [observer@sos ~]$ sos_status Son-of-Spectro cron IS running for USER=observer MACHINE=sos.apo.nmsu.edu. Loaded version=v4_7_6. The following aporsync jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 13595 13591 0 60 0 - 412 wait4 03:56 ? 00:00:00 sh /local/ups/Linux+2.2/idlspec2d/v4_7_6/bin/aporsync.sh You may need to kill old jobs if there is a problem. The following lock files exist: -rw-r--r-- 1 observer dss 1 Oct 24 03:57 /data/spectro/spectrologs/52206/logfil -52206.html.lock You may need to delete old lock files if there is a problem. [observer@sos ~]$ sos_stop Son-of-Spectro stopped. [observer@sos ~]$ sos_status The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.apo.nm u.edu. The following aporsync jobs are currently running: F S UID PID PPID C PRI NI ADDR SZ WCHAN STIME TTY TIME CMD 000 S observer 13595 13591 0 60 0 - 412 wait4 03:56 ? 00:00:00 sh /local/ups/Linux+2.2/idlspec2d/v4_7_6/bin/aporsync.sh You may need to kill old jobs if there is a problem. The following lock files exist: -rw-r--r-- 1 observer dss 1 Oct 24 03:57 /data/spectro/spectrologs/52206/logfil -52206.html.lock You may need to delete old lock files if there is a problem. [observer@sos ~]$ kill 13595 [observer@sos ~]$ kill 13595 13595: No such process [observer@sos ~]$ sos_status The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.apo.nm u.edu. No aporsync jobs are currently running. The following lock files exist: -rw-r--r-- 1 observer dss 1 Oct 24 03:57 /data/spectro/spectrologs/52206/logfil -52206.html.lock You may need to delete old lock files if there is a problem. [observer@sos ~]$ rm /data/spectro/spectrologs/52206/logfile-52206.html.lock rm: remove `/data/spectro/spectrologs/52206/logfile-52206.html.lock'? yes [observer@sos ~]$ sos_status The Son-of-Spectro cron job is NOT running for USER=observer MACHINE=sos.apo.nm u.edu. No aporsync jobs are currently running. No lock files exist. [observer@sos ~]$ sos_start Now running Son-of-Spectro version v4_7_6. [observer@sos ~]$