Date: Wed, 9 Oct 2019 11:08:57 -0000 Subject: Re: [sloan-25mlog 4696] 2.5m obslog 20191008 (SJD 58765) From: kgrabowski@apo.nmsu.edu > > APACHE POINT OBSERVATORY SDSS 2.5M > > OBSERVING LOG > Tuesday October 08, 2019 > > (SJD 58765) > > ---- Observing Team ---- > > Dan Oravetz > Katie Grabowski > > ---- Observing Plan ---- > > Tue 8 OCT 2019 58765.0 APOGEE-2 until 09:13 (7.7 h), then MaNGA (2.7 > h) > > ---- Observing Summary ---- > > Notes > ===== Lost 7.7h of APOGEE time and 2.7h of APOGEE > (1) We experienced an Az run-away slew during afternoon checkout and one > in Alt > during the night. Please see Problems section for details. Note that of > several > different slews tested this afternoon, including gotoGangChange, we only > had the > lone Az run-away slew, and only one in Altitude while testing in during > the > night. > > > ---- Observing Log ---- > > ============================================================================ > AFTERNOON CHECKOUT > ============================================================================ > > All systems alive and responsive. > > The first attempt at 'sop collimateBoss' produced a Ne lamp error, but we > don't > believe this to be a bug in the MCP. We successfully commanded the Ne, > HgCd, and > FF lamps on using the 'mcp ne.on', 'mcp hgcd.on', and 'mcp ff.on' commands > through STUI (ported through the hub). We also successfully commanded the > lamps > using the MCP Menu. The next 'sop collimateBoss' sequence turned the lamps > on > just fine and produced no errors, so this is not a reproducible error for > this > version of the MCP. This seems to have been a one-off event. > > 21:30:28 OBSERVER.doravetz.sop 592 mcp ! controlLamps > 21:30:28 OBSERVER.doravetz.sop 592 mcp ! command="NE_ON" > 21:30:28 OBSERVER.doravetz 32 sop e text="Failed to turn Ne lamps on > (bypass(lamp_ne) = False)" > 21:30:28 OBSERVER.doravetz 32 sop i > stageState="collimateBoss.lamps","failed",0.0,0.0 > 21:30:28 OBSERVER.doravetz 32 sop i > collimateBossState="running","OK","failed","idle" > 21:30:28 OBSERVER.doravetz 32 sop i > collimateBossStages="collimate","cleanup" > 21:30:28 OBSERVER.doravetz 32 sop i collimateBossState="failed","Failed to > warmup lamps/close FFS in preparation for Hartamnns.","failed","idle" > 21:30:28 OBSERVER.doravetz 32 sop f text="Failed to warmup lamps/close ffs > in > preparation for hartamnns." > 21:30:28 CmdDone: OBSERVER.doravetz 32 sop collimateBoss > > Adjusted b2 residual +2 deg based on last night's Collimate results. > > Ventilation not started because of humidity and nearby precip. > > ============================================================================ > NIGHT > ============================================================================ > > Didn't open, due to high humidity, clouds, and radar. > > During the night (while closed for weather), we performed many dozens of > slews in > an effort to tickle MCP bugs. We commanded slews in several different > ways: > directly from the MCP Menu, using 'tcc track' commands, and using > sop-commanded > slews (e.g.,'sop gotoGangChange', 'sop gotoInstrumentChange', 'sop > gotoStow', etc.). > > More details in about runaway slews in "Problems in Detail" section, > as well as an email sent to the mcp list. > > 10:53Z-Started APOGEE and BOSS cals. > > ---- BOSS Spectrographs Collimation Summary ---- > > --- Afternoon focus check --- > > Adjusted the b2 residual +2 deg based on last night's Collimate output. b2 > was > reporting -1.0 deg residual before the blue-ring adjustment this afternoon > and > now after the ring move (-5 deg total moverequired) b2 is reporting ~1-1.5 > deg > residual. > > Time Inst r1 b1 move1 b1pred Tsp1 r2 b2 move2 b2pred > Tsp2 > -------------- ---------------------------- > ---------------------------- > Pre-ring adjustment b2: > 22:22 4-10224A -659 -2.2 24 -2.1 8.4 -315 -1.0 0 > -1.0 11.5 > 22:55 4-10224A -659 -2.2 24 -2.1 8.5 -315 -1.0 0 > -1.0 11.7 > 22:58 4-10224A -659 -2.2 24 -2.1 8.5 -315 -1.0 0 > -1.0 11.8 > > Post-ring adjustment b2: > 23:04 4-10224A -659 -2.2 24 -2.1 8.5 0 2.0 -315 > 1.0 11.6 > 23:08 4-10224A -659 -2.2 24 -2.1 8.5 315 2.0 -157 > 1.5 11.6 > > Still in-focus as of that last set of hartmanns: > > 23:08:28 OBSERVER.telescope.sop 743 hartmann i r1MeanOffset=-0.10,"In > Focus" > 23:08:28 OBSERVER.telescope.sop 743 hartmann i r2MeanOffset=0.05,"In > Focus" > 23:08:28 OBSERVER.telescope.sop 743 hartmann i b1MeanOffset=0.11,"In > Focus" > 23:08:29 OBSERVER.telescope.sop 743 hartmann i b2MeanOffset=-0.10,"In > Focus" > > ---- Mirror Numbers ---- > > PRIMARY: > -------- > Scale: 1.000000 > > MIGS TONIGHT NOMINAL > Axial A: -5.331 -5.329 > Axial B: -5.053 -5.05 > Axial C: -5.272 -5.268 > Trans D: -0.58 -0.577 > Lateral E: 9.3472 9.3472 > Lateral F: 14.4526 14.4526 > > GALILS > Commanded: 11200 -1850 6850 29650 21250 21050 > Actual: 11200 -1850 6850 29650 21250 21050 > > SETMIR VALUES > PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 > PrimOrient: 0.0 -11.4 6.58 375.38 620.77 > > SECONDARY: > ---------- > Focus: 0.0000 > Air temp: 12.3 > Altitude: 30.0 > > MIGS TONIGHT NOMINAL > Axial A: -6.665 -6.664 > Axial B: -0.029 -0.028 > Axial C: -6.667 -6.669 > Trans D: -0.069 -0.076 > > GALILS > Commanded: 1096750 1137100 1118050 -7200 -6550 > Actual: 1058630 1101282 1079317 -7200 -6550 > > SETMIR VALUES > SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 > SecOrient: 856.91 2.7 12.74 -37.86 156.14 > > ---- Telescope Status ---- > > > No interlocks bypassed > > Status at: 11:06Z > Telescope stowed at: 121.0, 30.0, 0.0 mount > Instrument mounted: cartridge = 4 > Counterweights at: 366 361 324 319 > LN2 autofill systems: Connected and turned on > 180L LN2 dewar scales: SP1 195.6 lbs, 23.2 psi > SP2 196.6 lbs, 20.9 psi > > > > > ---- Software Used ---- > > --- Afternoon disk space --- > > /dev/vdd 688G 643G 46G 94% /home > /dev/vdc1 11T 6.6T 4.2T 62% /data > > --- STUI software --- > STUI : 1.7.8 > alerts : trunk > apo : trunk > boss : v4_1_1 > -boss.specMec : 3.0.4 > gcamera : v2_2 > guider : 3.8.1 > hartmann : 1.6.0 > hub : Tag: v4.2 > mcp : 1.2-1-gab77|2019-10-07|18:44 > -mcp.plc : 34 > -mcp.azFiduci : v1_140 > -mcp.altFiduc : v1_140 > -mcp.rotFiduc : v1_140 > platedb : 2.8.0 > sop : 3.12.3 > tcc : 1.3.1 > apogee : not availble > apogeecal : not availble > apogeeql : v1_6_1 > APO-local : not availble > > --- Other versions --- > idlspec2d : v5_7_12 > plate-mapper3 : NOCVS:v6_0_12 > petunia : git > autoscheduler : 1.3.0 > sdss_python_module: trunk > mangadrp : trunk exported > > ---- APOGEE Data Summary ---- > > [observer@sdss4-hub ~]$ list_ap -m 58764 -morning > APOGEE data list, mjd=58764, morning > raw_data: /data/apogee/utr_cdr/58764/apRaw-*.fits > archive: /data/apogee/archive/58764/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/58764/ap2D-a-*.fits.fz > -------------------------------------------------------------------------------- > mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset > %Flux Comm > -------------------------------------------------------------------------------- > 58764 1 11:08 32020071 Dark 60 A 13.0 4-10224 a-b-c - - > 58764 2 11:19 32020072 Dark 60 A 13.0 4-10224 a-b-c - - > 58764 3 11:29 32020073 Dark 60 A 13.0 4-10224 a-b-c - - > 58764 4 11:40 32020074 QuarFlat 10 A 13.0 4-10224 a-b-c - 46 > 58764 5 11:42 32020075 QuarFlat 10 A 13.0 4-10224 a-b-c - 46 > 58764 6 11:44 32020076 QuarFlat 10 A 13.0 4-10224 a-b-c - 46 > 58764 7 11:46 32020077 Arc-Thar 12 A 13.0 4-10224 a-b-c -2.79 - > 58764 8 11:48 32020078 Arc-Une 40 A 13.0 4-10224 a-b-c - - > 58764 9 11:55 32020079 Arc-Thar 12 B 13.5 4-10224 a-b-c -2.27 - > 58764 10 11:57 32020080 Arc-Une 40 B 13.5 4-10224 a-b-c - - > 58764 11 12:05 32020081 Dark 30 A 13.0 4-10224 a-b-c - - > 58764 12 12:10 32020082 InteFlat 30 A 13.0 4-10224 a-b-c - 0 > 58764 13 12:15 32020083 InteFlat 30 A 13.0 4-10224 a-b-c - 0 > 58764 14 12:21 32020084 InteFlat 30 A 13.0 4-10224 a-b-c - 0 > 58764 15 12:26 32020085 Dark 30 A 13.0 4-10224 a-b-c - - > -------------------------------------------------------------------------------- > > ---- Problems in Detail ---- > > 22:30Z -- runaway Az slew > > Slewed from 121,30,00 to 121,90,00 using the 'sop gotoInstrumentChange'. > The > telescope slewed to 121,90,00 normally and then the axes went into the > 'Halted' > state, which is expected because the sop-commanded slews end the slews > with a > 'tcc axis stop'. So at this point everything behaved as expected. > > From 121,90,00, and with the axes still 'Halted', we issued a slew to > 75,90,35. > The slew started normally, but then Az kicked into gear and abruptly sped > up at > Az=106 deg after it had slowed to a stop. We hit a stop button to stop the > slew. > > We slewed back to 121,90,00 using 'sop gotoInstrumentChange' and the slew > executed normally. We tried to reproduce the runaway slew (slewed to > 75,90,35 > from 121,90,00 with the axes in the 'Halted' state), but this time the > slew > behaved normally. > > We tried several slews this afternoon including the 'sop gotoGangChange' > slew > (from 121,90,00 to 121,45,00). We did not have any problems with any slews > this > afternoon apart from the above-described Az run-away. We alternated > between using > MCP Menu, 'tcc track' commands, and sop-commanded slews. We did not > experience > any problems with the semaphore transfer between any slews. > > Runaway Alt Slew during nighttime testing > ========================================= > The telescope was at 121,30,00 with the axes in a "Halted" state. A slew > from 121,30,00 to 121,90,00 was commanded using 'tcc track 121,90 mount > /rota=0 /rotty=mount' and it started at 02:17:11Z. The slew started > normally, but Alt slowed down prematurely and eventually ceased motion at > Alt=43 deg. After several seconds Alt took off quickly, moving upward > toward zenith, and a 'stop' button was pressed when Alt was at 62 deg. The > slew misbehavior is captured in the attached 'runawayAlt_cpos_enc1' TPM > plot. > > This slowing-to-a-stop-before-reaching-target behavior is very similar to > what we used to experience when we had axes go out of closed loop. When we > had those out-of-closed-loop events, the axis would slow down prematurely > and eventually stop short of the commanded position, then after several > seconds it would go out of closed loop. In the cases described above, > after slowing down prematurely and stopping before the target position, > there was *NOT* an out-of-closed-loop, but instead a take off -- the > behavior was as though the stopped axis had fallen behind and needed to > race to catch up. > > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html >