Date: Sun, 30 Jun 2002 10:34:22 GMT From: prn@apo.nmsu.edu (Pete Newman) Subject: 2.5m obslog 20020629 (MJD 52455) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday June 29, 2002 (MJD 52455) ---=== OBSERVING TEAM ===--- Night: Steph Snedden, Pete Newman Support: Craig Loomis (by phone), Eric Neilsen (FNAL, by phone) Visitors: Paul Hickson (UBC), Ken Lanzatta (SUNYSB) and family ---=== OBSERVING PLAN ===--- Shakedown tasks - night 1. ---=== OBSERVING SUMMARY ===--- Lots of problems, few tasks completed. ---=== OBSERVING LOG ===--- We met an interesting set of problems during this first night of shakedown for the 2002 July dark run. See the noted problems section for details, but in summary, we had problems with: - The sdsshost web server failing to start after the system was rebooted (problem #1). - logTool failing to start because it could not log in to hoggpt, sos and plate-mapper (problem #2). Observers and any other logTool users please note: this probably requires action by each of you to be solved. See the problem section and a seaparate email for details. - The new MCP azimuth fiducial tables being grossly wrong (problem #3). We reverted to the preceeding MCP version and all was well. - Communications with the engineering camera failing (problem #4). - Losing TPM readings many parameters (problem #5). We thought we had solved the engcam communications problem and started work on observations for a pointing model, but it soon recurred. Around 08:45Z (1h30m before the start of morning twilight, we tried again, and quickly saw a repeat failure. Although not normalyl part of this log, we note that we were unable to get MOP to talk to the PT camera tonight, getting either "no communications" or "unable to telnet" errors. We found the monts1 terminal server in various configurations after these failures, with ports 2 and 5 being connected or disconnected. Although the guide camera system (and hence the engineering camera) does not seem to share the same network, we find it suspicious that we had three major communications-related failures tonight - the first night since the new telephone system was installed. We ran endNight and shipped toapes so the TPM failure was recorded for later analysis. On changing the spectrograph dewar at the end of night, we found tha tthe new dewar's vent vale was slightly open, so there was no pressure in the dewar. We had just forced a fill from the old dewar, so we trust sufficient pressure will build before the next demand fill. We will check this before going to sleep this morning (but after this log is sent). Successes: - Spectrographs monthly checkout completed yesterday (but no dark exposures) - sdssProcedures v1_50 changes look ok. ---=== IMAGING RUN SUMMARY ===--- Run Time Stripe Lambda Last Flavor Comments Start End Begin End Frame ------------------------------------------------------------------------- ---=== IMAGING RUN DETAILS ===--- ---=== SKIPPY RESULTS ===--- Run Frame nFrames stars muErr muRms nuErr nuRms rot az el --------------------------------------------------------------------------- ---=== LTMATCH RESULTS ===--- Run Field nFields alt az nGood rowMean rowSig colMean colSig rot ---------------------------------------------------------------------- ---=== SPECTROSCOPY DATA SUMMARY ===--- UT Exp Time flavor comment (S/N)^2 totals ========================================== b1 r1 b2 r2 ----- sequence 15793, plate -9999 ------- 03:35 15793 0.0 bias ---=== TELESCOPE OFFSETS AND SCALE I ===--- Time Instrument Az Alt Rot Scale pos offset pos offset pos offset ------------------------------------------------------------------------------ ---=== DATA TAPE SUMMARY ===--- Although no data was taken tonight, we are writing and shipping tapes because of the TPM failure that occured. We do not know the cause or severity of the underlying problem, so feel it useful to record the data. Goes: JL5434 Stays: JL5435 ---=== FOCUS LOG ===--- setmir piston Temp Wind Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt fwhm ------------------------------------------------------------------------------ ---=== WEATHER LOG ===--- Wind Time Temp F Dewp F MPH Direction Dust DIMM Sky 03:15Z 61 41 11 109 (ESE) 906 - Patchy cloud 03:51Z 61 42 12 125 (SE) 1223 - " 04:23Z 61 42 12 142 (SE) 1345 - " 04:56Z 60 42 12 143 (SE) 1142 - " 05:28Z 61 42 12 150 (SSE) 1078 - " 06:00Z 58 42 12 134 (SE) 1092 - " 06:36Z 57 42 6 125 (SE) 1102 - " 07:08Z 57 42 2 134 (SE) 1174 - " 07:41Z 60 41 6 197 (SSW) 1481 - " 08:14Z 62 40 5 262 (W) 1301 - " 08:47Z 61 41 8 249 (WSW) 1317 - " 09:20Z 60 41 6 232 (SW) 1019 - " 09:53Z 60 41 4 213 (SSW) 1181 - " 10:30Z 59 42 3 263 (W) 1326 - ---=== TELESCOPE STATUS ===--- Status at 10:00Z: Telescope stowed at: 30 deg Instrument mounted: Cartridge 2 Counterweights at: 130 Autofill systems: On 180L LN2 dewar weights: Spectro 240 lb Imager 125 lb Interlocks bypassed: Not noted. ---=== SOFTWARE USED ===--- IOP/SOP: v3_81_0 Test Watcher: v2_18_0 Current MCP: v5_13_0 Current (not test) TPM: tpm_v2_14_0 Test AstroDa: v14_47 Current TCC: 2.6.5 Current sdssProcedures: v1_50 Test SoS: v4_9_10 N.B. Not the test version hoggPT: v1_5_8 Current plate-mapper: v4_1_0 Current ---=== MIRROR NUMBERS ===--- Not recorded - see note in main log section on TPM failure. ---=== PROBLEMS IN DETAIL ===--- Problem #1 - HTTP server on sdsshost not running after reboot ---------------------- We rebooted sdsshost, then discovered the WWW server had not started. Craig was called. He determined that the way the new version of the server script is registered with UPS is wrong, so that it is not auto-started. We started it by hand using (on 'host, as root): ps -ef | grep httpd to check there really were no HTTP daemons running (there were not). Then: setup www_server start_wwwgate PR 3969 filed for action by Scot K. Problem #2 - logTool would not start ---------------------- Pete found his logTool would not start - it just hung. Ctrl-C produced an error message saying it could not log on to hoggpt. With Eric's help, we traced this to a need to add the ssh public keys (identity.pub files) to the authorised key lists on sos, hoggpt and plate-mapper, each of which logTool now logs in to to get the software version numbers for the log. Each user who will run log tool needs to append the contents of their .ssh/identity.pub file to the .ssh/authorized_keys file on each of those machines, and also log in once as observ on each of those machines, to clear the "never heard of this machine" warning that ssh produces on first login. Process to get this done for all observers started. Problem # 3 - MCP v5_13_1 fiducial tables wrong ---------------------- Slews during pre-flight checks showed large errors in the azimuth encoder positions on crossing fiducial marks. We forced acceptance on a fiducial to zero the offset, but then saw errors on sequential fiducials of ~1000 counts, then ~3000 counts. We reverted tothe previous version of MCP, v5_13_0, and the problem went away, so we decided to proceed with that version. It is not clear to us if this is related to a scale error in the calculated fiducial table, or the absence of measurements for fiducial 19. Problem # 4 - Engineering camera commincations ---------------------- The engineering camera failed to initialize following a TCC 'set inst=engcam' command, with the TCC reporting: 0 0 I Modu="prt_Init_Forth"; Text="sending initialization command" 0 0 F Modu="cam_o_SetCamera"; Text="could not understand reply from guide image controller:" 0 0 I Modu="cam_o_SetCamera"; Text="reply: ; ppk error: Error code = 86" 0 0 W Modu="prt_ChangeDev"; Text="could not select camera: 2" 0 0 W Modu="prt_ChangeDev"; Text="the guide camera TCC_GCAM1 will not be usable until you fix the problem" We found an error LED lit intermittently on the Paralan SCSI-fiber converter inside the engineering camera cartridge, and found we could make the error go away by slightly bending the right-hand fiber of the pair towards the other fiber, just where it connects to the Paralan. Swaping the fibers did not swap the error, so we suspected the Paralan box, and we removed it. We discussed swapping it for a spare with Craig, and learned there is no spare on site. At the same time, we made a detailed examination of the connectors using the lab microscope. We found a sub-millimeter scrap of polychromatic debris of unknown material almost exactly half covering the fiber end inside the right-hand connector on the Paralan box. A can of air removed that, and after reassembly and rebooting the guider Macintosh, everything worked fine - for about 10 minutes. The same failure then returned. To further test if the failure was in the Paralan box from the engineering camera end, we then swapped the Paralan boxes between the engineering camera and the computer room. On rebooting the Mac, we saw the error light on the same Paralan box (now in the computer room) within a couple of gcam commands, and were able to cure it the same way as before (slight pressure on the right-hand fiber towards the other fiber). This indicates the failure is indeed in the Paralan unit. At this point, we gave up. The failure Paralan unit in the computer room should be fixed, replaced or swapped back for the one in the engineering camera, so that we can do spectroscopy. Problem #5 - TPM lost many data streams ---------------------- About 09:15Z, we noticed the TPM display was missing readings for MIGs, dewar weights, temperature sensors, mirror Galils (and probably others), while continuing to get nreadings for other values, e.g. axis position, error and current. This failure may have occured much earlier, but the TPrl M was seen to be working at the start of the night at least. This failure also meant IOP could not get the mirror numbers for the log.