Date: Fri, 7 Mar 2003 13:14:40 GMT From: prn@apo.nmsu.edu (Pete Newman) Subject: 2.5m obslog 20030306 (MJD 52705) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday March 6, 2003 (MJD 52705) ---=== OBSERVING TEAM ===--- Swing: Jurek Krzesinski Night: Howard Brewington, Pete Newman Support: Eric Neilsen (FNAL, by phone) ---=== OBSERVING PLAN ===--- Science, with observer training, this being Howard's first night driving. ---=== OBSERVING SUMMARY ===--- Two plates done (936, 1265) while the Moon was up, then imaging the rest of the night with over 6 hours of data all in good seeing. Strip 27 S, -41 $startTime + $timeout} { set errorMessage "timed out waiting for the TCC to complete command $tccCommand" murmur "tccPutAn ..." ("while" body line 3) invoked from within "while {!$TCCCommandComplete($commandIndex)} { listenerToTCC if {[getclock] > $startTime + $timeout} { set errorMessage "timed ou ..." (procedure "tccPutAndWait" line 17) invoked from within "tccPutAndWait "track 121,90 mount /rotangle=$rot_dest /rottype=mount"" (procedure "instChange" line 108) invoked from within "instChange" (procedure "GUIinstChange" line 40) invoked from within "GUIinstChange" invoked from within ".sgui.instchange.go invoke" ("uplevel" body line 1) invoked from within "uplevel #0 [list $w invoke]" invoked from within "if {($w == $tkPriv(window)) && ([$w cget -state] != "disabled")} { uplevel #0 [list $w invoke] }" invoked from within "if {$w == $tkPriv(buttonWindow)} { set tkPriv(buttonWindow) "" $w config -relief $tkPriv(relief) if {($w == $tkPriv(window)) && ([$w cget -state] ..." (procedure "tkButtonUp" line 3) invoked from within "tkButtonUp .sgui.instchange.go" (command bound to event) Eric was called and after discussion, we decided to continue and he will investigate further during the day. In fact, the next instrument change (to the imager) was the last of the night, and we used TCC commands instead of the instChange command. +++++++++++++++++++++++++++++++++++++++++++ Problem #2 - imager teamster died --------------------------------- Shortly after 10Z, watcher reported "no gang transfers" erros while we were imaging. A 'teamster 60' command in IOP confirmed that the old teamster had died and a new one was started. The murmur log contained the following death cry from the old teamster: 2003-03-07 09:59:27Z sdsshost TMS 225027 TEXTONLY Gangs transfer: id1:1/1 id2:1/1 id3:1/1 id4:1/1 id5:1/1 id6:1/1 ad1:1/1 ad2:1/1 ad3:1/1 ad4:1/1 -> unix 2003-03-07 09:59:29Z sdsshost TMS 225027 dsc_E_shareLost Access to shared data area at 0x18000000 lost 2003-03-07 09:59:29Z sdsshost TMS 225027 dsc_E_writeFail Write failed to PT icc22 CSB=0x80430010 2003-03-07 09:59:29Z sdsshost TMS 225027 dsc_E_writeFail Write failed to rmt=0x1848CE7C cnt=0x00002288 2003-03-07 10:00:23Z sdsshost IDA 18049 TEXTONLY Reaping 225027 2003-03-07 10:08:55Z sdsshost IDA 66002 TEXTONLY The old teamster, PID 225027, appears to have died; restarting We then saw many messages like this in the murmur log: 2003-03-07 10:15:07Z sdsshost TMS 418118 TEXTONLY Waiting for command response from id6 (crate 2, ICC 2), data server (client 1) Following folklore, we ran the ptvmeTest commands to see what was what im> ptvmeTest -n 1 -a 0x18000000 PID: 420775 IOPB size: 64 bytes ping: no operation: read loopCnt: 1 (defaulted) 100 loops per dot LWordCnt: 0x00000001 (defaulted) controller: 0 (defaulted) remNode: 1 --> /dev/ptvme0n1 remAddr: 0x18000000 (defaulted) IxferMode: 0x00 (defaulted) TxferMode: 0x00 (defaulted) but that then seemed to hang up the IOP session for at least a few minutes. As predicted in folklore, gans transfers restarted after a few minutes, although that teaamster died when the hung IOP was killed, and another had to be started. This took some time, as a new IOP took many minutes to get to its prompt (I guess because the DA was so busy anyway). We did not complete the other ptvmeTest's from folklore as it was taking a very long time to start IOP sessions and we did not want to hang any we already had open! +++++++++++++++++++++++++++++++++++++++++++ Problem #3 - endNight error message During a full endNight, we saw the following error: Fri Mar 7 12:44:15 GMT 2003 Starting /data/photo/52705/3704 archiving. 917576 blocks to archive. Estimate finish at Fri Mar 7 12:49:13 2003 pgxwin_server: Another server is already active. pgxwin_server: Another server is already active. filling out runlog /astrolog/52705/runlog-3706.txt for run 3706. invalid command name "summarizeManyNights" while executing "summarizeManyNights -help" ("eval" body line 1) invoked from within "eval summarizeManyNights -help" invoked from within "ftclHelpDefine iop summarizeManyNights " [eval summarizeManyNights -help]..." Fri Mar 7 12:51:45 GMT 2003 /data/photo/52705/3704 finished. but endNight appeared to continue. PR 4916 opened. ---=== SPECTROGRAPH FOCUS AND COLLIMATION ===--- Spectrograph focus check - everything in focus, no pistoning today: ------------------------------------------------------------------- sp1> Coll_motor_A -2434 Coll_motor_B -2435 Coll_motor_C -3639 sp2> Coll_motor_A 17 Coll_motor_B 13 Coll_motor_C 15 During afternoon checkout, we've got the sp2> mechanical problem: Flushing charge for 15 sec mcpPut: SYSTEM.STATUS 2003-03-07 00:20:06Z: Original command was !e10! 2003-03-07 00:20:06Z: expected reply was !e10 OK ! Error: Mechanicals did not respond correctly; response was e10 ---------------- but mechSend s didn;t show any problem here. We took another flat exposure and it went fine. IDL> collimate, 19672 % Compiled module: COLLIMATE. Reading /data/spectro/52705/sdR-b1-00019672.fit Reading /data/spectro/52705/sdR-b1-00019673.fit COLLIMATE: Min offset = -0.83 pix COLLIMATE: Max offset = 0.55 pix COLLIMATE: RMS across CCD = 0.35 pix COLLIMATE: Mean offset = -0.14 pix COLLIMATE: Camera b1 appears to be IN-FOCUS (|mean|