Date: Sun, 11 May 2008 20:54:41 -0600 (MDT) Subject: Re: 2.5m obslog 20080510 (MJD 54597) From: "Daniel Oravetz" To provide an answer to the problems in last night's log, it seems the problems we were having last night trying to get things working after the MSP program hung during the attempt at `z 2' QE is related to teamster permissions. Lately, if the MSP program hangs, the remedy has been to restart the teamster and then pick up where it left off. We got through linearity on 3 chips and had just begun a go at QE when it hung. When we tried to force-restart the teamster, we got the reply "Ignoring -force, as I can't kill the old teamster." There was another teamster that had been open from the afternoon and even with the -force flag, we couldn't kill it and therefore we were stuck. This afternoon Howard and Kaike tracked down the other teamster, killed the process, started another teamster, and things worked fine. Thanks Howard and Kaike for remedying this problem this afternoon. Right now, the person who owns the teamster is the only person that can kill it since the -force is being ignored. Kaike has said he will talk to Jon about this tomorrow afternoon. DanO > > APACHE POINT OBSERVATORY SDSS 2.5M > > OBSERVING LOG > Saturday May 10, 2008 > > (MJD 54597) > > ---=== OBSERVING TEAM ===--- > > Night: Howard Brewington, Dan Oravetz > Swing: Kaike Pan > > ---=== OBSERVING PLAN ===--- > > Science! > > ---=== OBSERVING SUMMARY ===--- > > We had high dust counts (~8000 counts) most of the night, so we did MSP > imager > calibrations. At around 08:40Z, the dust count dropped and the thin > clouds > disappeard, so we decided to try spectroscopy. > > Spectroscopy > ------------ > Finished cartridge 3, 2561. > > MSP Imager Calibrations > ----------------------- > Linearity: completed `z 5,' `z 4,' `z 3' > > We tried getting `z 2' QE finished, but after only 2 frames it hung (see > log), > so we didn't bother counting these two frames and whenever `z 2' QE is > attempted in the future, just start from the beginning. Jon, we think > that > maybe rebooting 'host2 could help? Could you try rebooting 'host2 and > seeing > if a `goDrift -bias' would work? Thank you. > > We accumulated ~15,000 dust-hours (in addition to last night's ~20,000), > so > please check the mirror. > > The altitude drive surface near the laptop has a heavy grease appearing > periodically on the surface, please check this surface as well. Thanks. > > Imager dewar is just less than 100 lbs. > > ---=== OBSERVING LOG ===--- > > Afternoon: > ---------- > All systems were checked and working properly. The spectro focus was > set for 9C. > > Dust count was about 6.5 - 7k, no evening ventiltion was started. > > Cartridge 1 was mounted for the begining of the night. > > Night: > ------ > 01:55Z - Dust count was still 7000 counts. > > 01:58Z - Sunset. > > 02:10Z - In lieu of the dust, we set up for MSP imager calibrations. > > MSP Imager Calibrations > *********************** > 02:18Z - MSP Bias. Run 7489, frame 1. > > 03:21Z - Started `z 5' linearity: `getLinearCCD z 5' > Run 7489, frames 2 - 22 > > 04:40Z - Started `z 4' linearity: `getLinearCCD z 4' > Run 7489, frames 23 - 41 > > 05:53Z - Started `z 3' linearity: `getLinearCCD z 3' > Run 7489, frames 42 - 60 > > 07:09Z - Started `z 2' QE: `getQeCCD z 2' > Run 7489, frames 61 and 62 > > During the `z 2' QE, the MSP program hung because after frame 62, nothing > was > happening for about 15 minutes. We tried force-killing the teamster and > restarting, but it replied "...I can't kill the old teamster." We killed > the > MSP/IOP process, started up a new IOP session, and tried taking an imager > bias, but it hung at the "camera not set up for drifting, performing an > initData" part. We tried rebooting the crates and still the same problem. > We > manually logged out the port (port 18) and tried grabbing the "msp im" > instruments, tried a `ping' and it worked. We then issued an `mspSend > -echo > quit' and it let us quit. We dropped the instruments and exited and > everything behaved normally, but when we tried taking another bias, it > hung at > the same place as before. > > The dust count dropped to ~ 5000 counts and the thin clouds cleared out, > so we > opened in attempt to finish off a seguefaint plate that had previous > signal on > it. > > 08:48Z - Enclosure off, doors and louvers opened, fans on. > > Cartridge 3, 2561, seguefaint149 > ******************************** > 08:58Z - gotoField. Guide stars found on gotoField, but the seeing was > very > poor at ~ 3 - 4 arcseconds fwhm and the efficiency was only 11%. > > While we waited for the seeing and efficiency to improve, we tried to > diagnose > the imager problems we were having before during the MSP imager > calibrations. > During the setup of a new IOP session, this time it said that the IOP > server > had died. We restarted the servers and tried another bias and it still > hung > on the "camera not set up for drifting, performing an initData" part. > > 09:36Z - Pre-Calibs. > > 09:43Z - Science. 1800s. > > 10:17Z - Science. 1500s. > > 10:45Z - Post-Calibs. > > This plate is DONE. > > 11:06Z - Enclosure on, doors and lovuers closed, fans off. > > 11:11Z - endNight rejoiced. > > ---=== IMAGING RUN SUMMARY ===--- > > Run Time Stripe Lambda Last Flavor Comments > Start End Begin End Frame > ------------------------------------------------------------------------- > > 7488 23:41Z 00:00Z 100 O -66.37 -61.61 41 ignore > 7489 02:18Z 100 O MSP `z' linearity tests > > ---=== IMAGING RUN DETAILS ===--- > > ---=== SKIPPY RESULTS ===--- > > Run Frame nFrames stars muErr muRms nuErr nuRms rot az el > --------------------------------------------------------------------------- > > ---=== LTMATCH RESULTS ===--- > > Run Field nFields alt az nGood rowMean rowSig colMean colSig rot > ------------------------------------------------------------------------ > > ---=== SPECTROSCOPY DATA SUMMARY ===--- > > Summary Checked (y/n): y > QA Procedures Done (y/n): y > > UT Exp Time flavor comment (S/N)^2 totals > ========================================== b1 r1 b2 r2 > > ----- sequence 52714, plate -9999 ------- > 23:37 52714 0.0 bias > ----- sequence 52715, plate 2561 ------- > 23:46 52715 10.0 flat > 23:49 52716 2.0 arc > 23:53 52717 3.0 arc > 23:56 52718 3.0 arc > ----- sequence 52719, plate 2922 ------- > 00:41 52719 10.0 flat > 00:44 52720 2.0 arc > 00:49 52721 3.0 arc > 00:52 52722 3.0 arc > ----- sequence 52723, plate -9999 ------- > 01:25 52723 0.0 bias > ----- sequence 52724, plate 2922 ------- > 04:24 52724 0.0 bias > ----- sequence 52725, plate 2561 ------- > 09:39 52725 10.0 flat > 09:41 52726 2.0 arc > 10:16 52727 1800.0 target > 10:44 52728 1502.0 target > 10:47 52729 10.0 flat > 10:50 52730 2.0 arc > > ---=== TELESCOPE OFFSETS AND SCALE I ===--- > > Time Instrument Az Alt Rot Scale > pos offset pos offset pos offset > ------------------------------------------------------------------------------ > 09:49Z 3 2561 186.82 0.0008 58.02 0.0010 13.27 0.0262 1.000200 > > ---=== TELESCOPE OFFSETS AND SCALE II ===--- > > ---=== FOCUS LOG ===--- > > setmir piston Temp Wind > Time Inst scale M1 M2 Foc Az Alt (C) MPH Dir filt > fwhm > ------------------------------------------------------------------------------ > 09:49Z 3 2561 1.00020 -1975 -402 75 187 58.0 9.4 2 149 ---- > 2.0" > > ---=== WEATHER LOG ===--- > > Wind > Time Temp F Dewp F MPH Direction Dust DIMM Sky > > 22:47Z 59 12 17 306 (NW) 6860 - partly cloudy > 23:19Z 58 11 11 285 (WNW) 6699 - partly cloudy > 23:50Z 59 13 10 259 (W) 6444 - partly cloudy > 00:21Z 59 13 6 258 (WSW) 6403 - partly cloudy > 00:51Z 58 12 7 278 (W) 6620 - partly cloudy > 01:21Z 56 10 8 269 (W) 6870 - partly cloudy > 01:52Z 54 8 5 260 (W) 7038 - mostly clear > 02:22Z 51 6 6 246 (WSW) 7364 - mostly clear > 02:52Z 53 7 8 260 (W) 7290 - mostly clear > 03:22Z 51 6 8 236 (SW) 7036 - mostly clear > 03:52Z 52 6 11 206 (SSW) 7821 - mostly clear > 04:23Z 52 6 6 201 (SSW) 7649 - mostly clear > 04:53Z 51 6 6 178 (S) 8161 - mostly clear > 05:25Z 51 5 7 177 (S) 8112 - mostly clear > 05:55Z 49 4 8 184 (S) 8206 - mostly clear > 06:26Z 49 4 8 202 (SSW) 8291 - mostly clear > 06:57Z 50 6 10 214 (SW) 8393 - mostly clear > 07:28Z 46 3 12 78 (ENE) 8301 - mostly clear > 07:59Z 43 14 18 72 (ENE) 6268 - mostly clear > 08:31Z 44 13 6 103 (ESE) 5933 - mostly clear > 09:02Z 44 14 5 80 (E) 5322 - clear > 09:33Z 47 8 7 105 (ESE) 5574 - clear > 10:03Z 49 5 6 172 (S) 7564 - clear > 10:33Z 49 3 5 183 (S) 7957 - mostly clear > 11:03Z 49 5 7 178 (S) 7891 - mostly clear > > ---=== TELESCOPE STATUS ===--- > > 08:48Z - Enclosure off, doors and louvers opened, fans on. > 11:06Z - Enclosure on, doors and louvers closed, fans off. > > No interlocks bypassed. > > Status at: 11:13Z > > Telescope stowed at: 30,121,00 mount > Instrument mounted: Cartridge 1 > Counterweights at: 252 lbs > LN2 autofill systems: Connected and turned on. > 180L LN2 dewar scales: Spectro 212.10 lbs, 13.10 psi > Imager 99.50 lbs, 15.18 psi > > ---=== SOFTWARE USED ===--- > > IOP/SOP: v4_38_0 > Watcher: v2_36_0 > MCP: v6_4_0 > TPM: tpm_v3_5_0 > AstroDa: v15_13_1 > TCC: TCC 2.7.2.1 August 6 2004 > sdssProcedures: v2_40 > SoS: v5_1_8 > plate-mapper: v4_3_1 > > ---=== MIRROR NUMBERS ===--- > > PRIMARY: > -------- > Scale: 1.000000 > > MIGS TONIGHT NOMINAL > Axial A 0.0190 0.0200 > Axial B -0.0160 0.0000 > Axial C -0.0240 -0.0230 > Trans D -0.0020 0.0120 > Lateral E 10.0838 10.0584 > Lateral F 11.6459 11.6205 > > GALILS > Commanded: 4600. -2000. 2650. -12450. -6500. -6550. > Actual: 4600. -2000. 2650. -12450. -6500. -6550. > > SETMIR VALUES > PriDesOrient: 0.00 0.00 0.00 100.00 -200.00 > PriOrient: 0.00 0.00 0.00 99.59 -200.69 > > SECONDARY: > ---------- > Focus: 0.00 > Air Temp.: 14.3 > Alt.: 29.999997 > > MIGS TONIGHT NOMINAL > Axial A -0.0410 -0.0440 > Axial B -0.0490 -0.0570 > Axial C -0.0540 -0.0670 > Trans D -0.0200 -0.0110 > > GALILS > Commanded: 1629047. 1652990. 1470893. -5900. -7800. > Actual: 1627500. 1653150. 1470900. -5900. -7800. > > SETMIR VALUES > SecDesOrient: 1257.00 60.00 -4.00 -20.00 118.90 > SecOrient: 1257.36 60.13 -4.56 -20.29 119.20 > > ---=== PROBLEMS IN DETAIL ===--- > > Spectro Focus Info: > ------------------- > Focus checked by used cartridge 3. No adjustment made. It is currently > set > for 9 C. > sp1> caget tpm_TM_PMSSAvg tpm_TM_TelairAvg.VALA > tpm_TM_PMSSAvg 10.37 > tpm_TM_TelairAvg.VALA 11.26 > sp1> mechSend s > Coll_motor_A 834 01P0000000834 > Coll_motor_B 1007 02P0000001007 > Coll_motor_C 813 01P0000000813 > Coll_motor_A_status 133 01S0133 > Coll_motor_B_status 133 02S0133 > Coll_motor_C_status 133 01S0133 > Requested_exp.time 0 > Exp_time_left 0 > Last_exp.time 3 > Last_open_transit.time 700 > Last_close_transit.time 530 > Humid_Hartmann 9.7 0xFE3B > Humid_Cen_Optics 10.1 0xFE35 > Temp_Median 12.4 > sp2> mechSend s > Coll_motor_A 2829 01P0000002829 > Coll_motor_B 2286 02P0000002286 > Coll_motor_C 2633 01P0000002633 > Coll_motor_A_status 132 01S0132 > Coll_motor_B_status 133 02S0133 > Coll_motor_C_status 133 01S0133 > Requested_exp.time 0 > Exp_time_left 0 > Last_exp.time 3 > Last_open_transit.time 715 > Last_close_transit.time 635 > Humid_Hartmann -25.0 0x0 > Humid_Cen_Optics 10.2 0xFE34 > Temp_Median 10.3 > IDL> collimate, 52717 > Reading /data/spectro/54597/sdR-r1-00052717.fit > Reading /data/spectro/54597/sdR-r1-00052718.fit > COLLIMATE: Min offset = -0.68 pix > COLLIMATE: Max offset = 0.32 pix > COLLIMATE: RMS across CCD = 0.28 pix > COLLIMATE: Mean offset = -0.22 pix > COLLIMATE: Camera r1 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) > COLLIMATE: Predict (red) piston movement of 1976 steps for spectro-1 > COLLIMATE: Issue SOP command: "sp1; mechPiston 1976" > Log file = Collimate-54597-r1-00052717.log > PostScript file = Collimate-54597-r1-00052717.ps > Reading /data/spectro/54597/sdR-r2-00052717.fit > Reading /data/spectro/54597/sdR-r2-00052718.fit > COLLIMATE: Min offset = -0.90 pix > COLLIMATE: Max offset = 0.10 pix > COLLIMATE: RMS across CCD = 0.28 pix > COLLIMATE: Mean offset = -0.21 pix > COLLIMATE: Camera r2 appears to be OUT-OF-FOCUS (|mean| > 0.15 pix) > COLLIMATE: Predict (red) piston movement of 1936 steps for spectro-2 > COLLIMATE: Issue SOP command: "sp2; mechPiston 1936" > Log file = Collimate-54597-r2-00052717.log > PostScript file = Collimate-54597-r2-00052717.ps > > DSS DSS DSS DSS DSS Digital Sky Survey DSS DSS DSS DSS DSS DSS > DSS *** The information posted to SDSS mailing lists is proprietary *** > DSS This is message 4043 in the sdss-25mlog archive, URL > DSS http://www.astro.princeton.edu:81/sdss-25mlog/msg.4043.html > DSS The index is at > http://www.astro.princeton.edu:81/sdss-25mlog/INDEX.html > DSS To join/leave the list, go to > DSS > http://www.astro.princeton.edu:81/cgi-bin/mailinglists.pl/show_subscription?list=sdss-25mlog > DSS To post a message, mail it to sdss-25mlog@astro.princeton.edu > DSS > DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS DSS >