Date: Sun, 24 Jun 2001 05:15:03 -0600 From: "s. a. snedden" Subject: 2.5-m ObsLog 20010623 (MJD 52084) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday, Jun 23, 2001 (MJD 52084) OBSERVING TEAM: =============== Steph Snedden (APO) Pete Newman (APO) Bryan Laubscher (LANL) - PT SUPPORT: ======== Jon Brinkmann (APO) - by phone VISITORS: ========= Jason Laubscher THE PLAN: ========= Spectroscopy until moonset, unless photometric, in which case we do Apache Wheel. Then imaging, spectroscopy, or Apache Wheel, depending on conditions. SYNOPSIS FOR NIGHT: =================== We were never able to open, thanks to clouds, threatening weather and dust. We had a warm-up on spectro 2. GENERAL COMMENTS (see Problems below for details): ================================================== Safety issue: During afternoon checkout, Pete stuck his hands into the cryo gloves for the all-sky camera and encountered both a moth and a spider. Eeeew. I guess the spider went in to get the moth. Like last night, we had threatening weather and clouds. Unlike last night, we were never able to open. While checking out a SOP PR during the cloudy weather, Pete took a spectro bias. Spectro 2 showed a huge dark count on both cameras. The cameras had warmed up WITHOUT THE WATCHER OR LOGVIEWER INDICATING ANY PROBLEM. We called Jon B. and were able to safely re-cool the CCDs. See the problems section for all the gory details. PRs were filed. DATA ACQUISITION (see IOP/SOP output below): ============================================ IMAGING: -------- Run Stripe Strip Flavor Lambda Last Frame # Begin End ------------------------------------------------------------------ 2413 100 O bias -17.55 -13.69 39 doghouse bias SPECTROSCOPY: ------------- UT Exp Time flavor comment =============================================== ------ sequence 10330, plate -9999 ------- 23:38 10330 0.0 bias {{}} ------ sequence 10331, plate 9999 ------- 00:52 10331 0.0 bias {{}} ------ sequence 10332, plate 628 ------- 07:16 10332 0.0 bias {{}} high dark count 07:20 10333 0.0 bias {{}} " " 07:22 10334 0.0 bias {{}} " " 07:31 10335 0.0 bias {{}} " " 07:34 10336 0.0 bias {{}} " " ------ sequence 10337, plate -9999 ------- 07:36 10337 0.0 bias {{}} " " ------ sequence 10338, plate 628 ------- 08:40 10338 0.0 bias {{}} " " 08:42 10339 0.0 bias {{}} " " 08:45 10340 0.0 bias {{}} " " 09:11 10341 0.0 bias {{}} levels going down 09:14 10342 0.0 bias {{}} Normal bias levels again 09:23 10343 2.0 flat {{Test after recooling sp2}} 09:26 10344 10.0 flat {{Another test after recooling sp2}} TAPES: ------ None shipped. Stay at APO: JL4187, 4188 MIRROR NUMBERS: =============== Time dpTemp dewPt Press TmpOut HumOut Windd Winds 18:24 71.7 32.9 21.60 23 242.6 6.8 22 PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0730 0.0710 Axial B: -0.1010 -0.0980 Axial C: 0.0650 0.0610 Trans D: 4.1810 4.1800 Lateral E: 1.9360 1.9365 Lateral F: 0.4370 0.4370 GALILS Commanded: 5377. -3699. 923. -208. 31535. 30647. Actual: 5400. -3700. 900. -200. 31550. 30650. SETMIR VALUES primDesOrient: 0.00 -11.80 23.00 1256.90 642.10 primOrient: 0.00 -12.16 22.81 1257.41 642.19 SECONDARY: ---------- Focus: 0.0 Air temp: 22.1 C Altitude: 30.01 MIGS TONIGHT NOMINAL Axial A: -1.2900 -1.2900 Axial B: -1.3140 -1.3150 Axial C: -1.3080 -1.3060 Trans D: 0.1820 0.1890 GALILS Commanded: 1583837. 1583837. 1583837. 2595. 3795. Actual: 1583857. 1583838. 1583914. 2600. 3800. SETMIR VALUES secDesOrient: 1257.00 0.00 0.00 0.00 -175.00 secOrient: 1257.02 -0.03 -0.02 -0.01 -175.10 WEATHER SYNOPSIS: ================= Cloudy. Storms nearby. A dust event (pollen)? WEATHER DATA: ============= Wind Time Temp F Dewp F MPH Direction Enchantment DIMM Sky Precip. 16:01 73 34 6 267 (W) 713 - mcl none 17:01 75 35 8 258 (WSW) 721 - mcl none 18:01 74 33 5 260 (W) 707 - mcl none 19:01 71 37 6 216 (SW) 843 - mcl none 20:01 63 38 10 135 (SE) 958 - mcl none 21:01 62 38 8 22 (NNE) 893 - mcl none 22:01 63 36 12 305 (NW) 806 - mcl none 23:01 58 40 14 71 (ENE) 872 - mcl none 00:01 57 40 8 89 (E) 878 - mcl none 01:01 58 40 12 130 (SE) 1034 - mcl none 02:01 60 39 14 137 (SE) 1099 - mcl none 03:01 61 38 4 177 (S) 2962 - mcl none 04:01 60 39 2 259 (W) 4743 - mcl none 05:01 58 40 5 228 (SW) 4176 - mcl none [pmt]cl = partly, mostly, totally cloudy LN2 SCHEDULE: ============= Imager on autofill Spectrographs on autofill Spectro dewar weight at 04:45: 137 lb. Imager dewar weight at 04:45: 180 lb. TELESCOPE STATUS: ================= Opened doors, blower and cone fans on: 20:15 Closed enclosure, blower & cone fans off: 21:30 Opened enclosure: never Stowed at 30 deg alt with cartridge 9 mounted. Counterweights at 280. INTERLOCKS: =========== None bypassed SOFTWARE USED: ============== IOP/SOP: v3_50_4 Watcher: v2_9_0 MCP: v5_2_1 AstroDa: 14_36 TCC: 2.5.3 TPM: v2_3_7 tccmon: v1_6 sdssProcedures: v1_20 PROBLEMS IN DETAIL: =================== Spectro 2 Warmup: ----------------- At about 01:45, while waiting on clouds, Pete took a bias to check on a SOP PR. The readout showed large dark counts on spectro2, saturated on b2, in fact. The 180-liter dewars had been left disconnected while we waited on clouds, and that's why we had a warm-up. What is really bad, is that we received NO warnings from the watcher that anything bad was happening. Some cc tchecks showed that the LN2 temps had reached 267K, and the CCD temps had reached 12C. The watcher displays showed the temperatures as all in spec! The watcher PGPLOTS showed the warmup. The cameras warmed up without the watcher throwing any voltage errors either for the CCD or lN2 temps. There were also no fill fault critical errors. The watcher indicated the server was connected, however a look at the spectrograph server window showed ftelnet errors to the spectrographs. Jon B. thinks the problem lies in the IOP grabInstruments -handlers process. It appears the ports were unavailable, so the grabInst process failed. If so, there should be a *noticable* warning about not being able to get the ports. The dangerous thing here is that the watcher behaved normally, throwing appropriate warnings, for other systems. Granted, there were ftelnet error messages in the murmur log, but there were also camcheck and ln2check messages for sp1 and sp2 that appeared normal. There were also no warnings in the logViewer, and we are now using that instead of watching the murmur log scroll by. We note that we were also able to communicate with the cameras normally. Of the following murmur log entries, only the 'Cleared ln2 fill fault' appeared in the 'unexpected' section of logViewer. prn:sdsshost:~/ > grep 438835 /astrolog/iop.murmur.log | grep '20:0[678]' Jun 23 18:50:06 sdsshost IDA 438835 TEXTONLY Next packeteer event loop pending in 9000 ms at Sat Jun 23 21:20:06 2001 Jun 23 20:08:30 sdsshost IOP 438835 TEXTONLY CAMERA: sp1 camera: sent "CAMCHECK\r". Read: "CAMCHECK ok\r\n" Jun 23 20:08:32 sdsshost IOP 438835 TEXTONLY CAMERA: sp1 camera: sent "SHOWREADVOLTS\r". Jun 23 20:08:33 sdsshost IOP 438835 TEXTONLY CAMERA: sp1 camera: sent ".CAM\r". Read: ".CAM \r\nE11 CAM_01=1\r\nE12 CAM_04=1 ok\r\n" Jun 23 20:08:33 sdsshost IOP 438835 TEXTONLY sp1: appending to idCCDLog parameter files Jun 23 20:08:35 sdsshost IOP 438835 TEXTONLY CAMERA: sp2 camera: sent "CAMCHECK\r". Read: "CAMCHECK \r\nLN2_FILL FAULT!!! ok\r\n" Jun 23 20:08:36 sdsshost IOP 438835 TEXTONLY CAMERA: sp2 camera: sent "CLEARFFAULT\r". Read: "CLEARFFAULT ok\r\n" Jun 23 20:08:36 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault Jun 23 20:08:36 sdsshost IOP 438835 TEXTONLY CAMERA: camcheck sp2 camera LN2_FILL FAULT!!! Jun 23 20:08:38 sdsshost IOP 438835 TEXTONLY CAMERA: sp2 camera: sent "SHOWREADVOLTS\r". Jun 23 20:08:39 sdsshost IOP 438835 TEXTONLY CAMERA: sp2 camera: sent ".CAM\r". Read: ".CAM \r\nE11 CAM_03=1\r\nE12 CAM_02=1 ok\r\n" Jun 23 20:08:39 sdsshost IOP 438835 TEXTONLY sp2: appending to idCCDLog parameter files Jun 23 20:08:41 sdsshost IOP 438835 TEXTONLY starting to fill out getWeather parameter file Jun 23 20:08:41 sdsshost IOP 438835 TEXTONLY finished with getWeather parameter file The following messages were found in the murmur log, but did NOT appear in the logViewer "unexpected" section: prn:sdsshost:~/ > grep 438835 /astrolog/iop.murmur.log | grep '01:2[123]:00' Jun 24 01:21:00 sdsshost IOP 438835 TEXTONLY CAMERA: Changing sp2 camera as requested: 1 SBINSET Jun 24 01:21:00 sdsshost IOP 438835 TEXTONLY CAMERA: sp2 camera: sent "1 SBINSET\r". Read: "1 SBINSET ok\r\n" Jun 24 01:23:00 sdsshost IOP 438835 TEXTONLY CAMERA: sp2 camera: sent "CAMCHECK\r". Read: "CAMCHECK \r\nCCD03_LN2TEMP READVOLTS=969 # ERROR: OUT OF SPEC\r\nCC... Jun 24 01:23:00 sdsshost IOP 438835 TEXTONLY ...D03_CCDTEMP READVOLTS=533 # ERROR: OUT OF SPEC\r\nCCD02_LN2TEMP READVOLTS=950 # ERROR: OUT OF SPEC\r\nCCD02_CCDTEMP... Jun 24 01:23:00 sdsshost IOP 438835 TEXTONLY ... READVOLTS=519 # ERROR: OUT OF SPEC ok\r\n" Jun 24 01:23:00 sdsshost IOP 438835 TEXTONLY CAMERA: camcheck sp2 camera CCD03_LN2TEMP READVOLTS=969 # ERROR: OUT OF SPEC Jun 24 01:23:00 sdsshost IOP 438835 TEXTONLY CAMERA: camcheck sp2 camera CCD03_CCDTEMP READVOLTS=533 # ERROR: OUT OF SPEC Jun 24 01:23:00 sdsshost IOP 438835 TEXTONLY CAMERA: camcheck sp2 camera CCD02_LN2TEMP READVOLTS=950 # ERROR: OUT OF SPEC Jun 24 01:23:00 sdsshost IOP 438835 TEXTONLY CAMERA: camcheck sp2 camera CCD02_CCDTEMP READVOLTS=519 # ERROR: OUT OF SPEC Also, why did IOP clear these faults? The PID corresponds to Pete's IOP process. Pete did the grabInst -handlers. Jun 23 20:08:36 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault Jun 23 21:43:59 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault Jun 23 23:23:15 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault Jun 23 23:27:23 sdsshost IOP 438835 TEXTONLY Cleared ln2 empty fault Jun 23 23:31:30 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault Jun 24 00:29:26 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault Jun 24 01:06:52 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault Jun 24 01:22:54 sdsshost IOP 438835 TEXTONLY Cleared ln2 empty fault Jun 24 01:32:22 sdsshost IOP 438835 TEXTONLY Cleared ln2 fill fault After talking to Jon B., we were assured that as long as the getter temp. was below freezing we could safely start to cool the CCDs. We proceeded to do so with a cc stopfillseq and a cc startfillseq for the cameras, after hooking up the 180-liter dewars. The cameras & CCDs appeared to recover well, with normal bias levels. We looked for signs of condensation by taking a flat. The images looked fine. An IOP PR was filed regarding grabInst -handlers. Another was filed on the logViewer not showing the above critical messages.