Subject: 2.5m obslog 20220310 (SJD 59649) From: viktorm@apo.nmsu.edu Date: Fri, 11 Mar 2022 11:09:56 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday March 10, 2022 (SJD 59649) ---- Observing Team ---- Katie Grabowski Viktor Malanushenko John Donor - in STUI ---- Observing Plan ---- Thursday-Saturday March 10-12 observe design 12798 : please do only one configuration and take 6 sets of APOGEE+BOSS exposures, i.e. six BOSSx900 and 6 APOGEE AB pairs (so 1.5 hours total). You can load the design from the command line or, if you prefer, try to put the design at the top of the queue and observe from there. If you load this design into the queue and run it from the queue, do not use the Kronos exposure completion criteria; just get the 1.5 hours no matter what. if conditions are good, do a joint BOSS-APOGEE dither test on design 106921. Following instructions on the observing instructions page, take both BOSS (300s) and APOGEE (single 30 read) at each dither position. If conditions are not stable/good, do later (or skip altogether); if too late for 106921, use design 106935 (want to keep whole dither sequence <~1.4 airmass) refresh the Kronos queue, and observe from the queue ---- Observing Summary ---- Night completely lost due to the weather conditions: cloudy, high wind and radar return. During afternoon and in the evening we had problems with - apogeeql.heartbeat : disabled alert - jaeger f text="[GET_STATUS, [199], 5657]: received a reply but the command has already timed out - fps.heartbeat,critical Solution: Restarted jaeger twice, reloaded jaeger to reconnect it to STUI. All ok now - Some messages is in the 'Problems in Detail" section. - Also in the e-mail Katie at 2:12 AM 'critical alert on fps.heartbeat' Notes: ====== -- sdR-b1-00338673.fit.gz: WARNING: SOS disk /data/boss/sos/59649 is 96% full 10:56:57 APOGEE morning cals started. ---- Observing Log ---- Afternoon: ========== All systems alive and responsive. John Donor said the large job transferring files is currently running, but almost done. It doesn't seem to be causing issues with APOGEE at the moment like it has previously. sdR-b1-00338673.fit.gz: WARNING: SOS disk /data/boss/sos/59649 is 96% full 00:10Z- started APOGEE calibrations 00:12:06 boss exposure 4.0, file=338675 00:12:48 boss exposure 4.0, file=338676 00:13:34 sp1.motor.move= 1159, 1159, 1160 00:13:34 Hartmann collimate output sp1: offset: r= 329 (In Focus); b= -6.2 (Out of focus) sp1: pred. move: spAverageMove= 1160 sp1: pred. resid: r= -831, b= -2.6, txt= OK, spTemp = 4.0 Delayed ventilation and opening due to high winds and clouds. Night: ====== ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0030 0.0040 Axial B: 0.0000 0.0030 Axial C: -0.1110 -0.1150 Trans D: -0.0342 -0.0338 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 4.94 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -9.6560 -9.6570 Axial B: -9.6680 -9.6710 Axial C: -9.6960 -9.6970 Trans D: 0.1050 0.1140 GALILS Commanded: 4892200 4889250 4913600 0 -3400 Actual: 4859337 4858361 4858783 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.68 -0.04 -0.31 -0.2 41.67 ---- Telescope Status ---- Started ventilation encl25m : open encl25m : closed No interlocks bypassed Status at:11:03Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: Instrument = FPS Counterweights at: 685 685 704 704 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 127 lbs, 23.5 psi ---- Software Used ---- Get Versions, 03/11/22, 00:11:51 -- STUI software: STUI : 1.7.16 alerts : 2.2.0 boss : 6.0.1 -boss.specMec : 3.0.4 guider : FAILED ecamera : FAILED apo : 3.0.1 fliswarm : 0.2.1-alpha.0 jaeger : 0.14.1-alpha.0 kaiju : 1.2.2-alpha.1 coordio : 1.2.2-alpha.1 fps_calibrati : APO.2021.12.25+960cb05 cherno : 0.1.1-alpha.0 hal : 0.1.1-alpha.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 34 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : 3.1.12 --------------Other Versions-------------- sdss-obstools : 3.7.10 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V34 Fiducials : v1_148 idlspec2d : v6_0_5 Kronos : 1.1.2a roboscheduler : 1.1.0 /dev/sdb1 2.0T 166G 1.9T 9% /home 10.25.1.230:/data 21T 20T 931G 96% /data ---- Problems in Detail ---- 0:39:22 OBSERVER.viktorm 1062 keys f keysTxt="actor guider is not connected" 00:39:22 CmdDone: OBSERVER.viktorm 1062 keys getFor=guider version guiderVersion cartridgeLoaded guideEnable expTime stack pid scales survey gprobeBits guideState movieFile movieStatus moviePID mangaDither seeing refractionBalance refractionWavelengths refractionOffset ecam_star 00:39:22 OBSERVER.viktorm 1063 keys f keysTxt="actor platedb is not connected" 00:39:22 CmdDone: OBSERVER.viktorm 1063 keys getFor=platedb version activePlugging plPlugMapM gprobe gprobesInUse gcamFiberInfoVersion guideInfo pointingInfo guideWavelength apogeeDesign mangaExposureTime pluggedInstruments text 00:41:08 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 600 sec ago",enabled,noack,-1 00:41:27 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 619 sec ago",enabled,noack,-1 00:41:48 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 639 sec ago",enabled,noack,-1 00:41:53 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 644 sec ago",enabled,noack,-1 00:42:06 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 657 sec ago",enabled,noack,-1 00:42:07 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 658 sec ago",enabled,noack,-1 00:42:14 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 665 sec ago",enabled,ack,observer.kgrabow 00:42:19 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 670 sec ago",disabled,ack,observer.kgrabow 00:42:28 .alerts 0 alerts e alert=apogeeql.heartbeat,critical,"at 2022-03-11 00:41:08; last seen 679 sec ago",disabled,ack,observer.kgrabow .... 1:56:06 CmdDone: flicamera.gfa5 42 keys getFor=jaeger fvc_deltarms fvc_perc_90 fvc_percent_reached fps_status snapshot trajectory_dump_file configuration_snapshot chiller_temperature_value chiller_temperature_setpoint chiller_flow_value 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [199], 5657]: received a reply but the command has already timed out." 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [40], 5658]: received a reply but the command has already timed out." 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [228], 5659]: received a reply but the command has already timed out." 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [322], 5660]: received a reply but the command has already timed out." 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [422], 5661]: received a reply but the command has already timed out." 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [296], 5663]: received a reply but the command has already timed out." 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [361], 5664]: received a reply but the command has already timed out." 01:56:35 .jaeger 0 jaeger f text="[GET_STATUS, [409], 5662]: received a reply but the command has already timed out." 0 ..... 01:56:39 .jaeger 0 jaeger f text="[GET_STATUS, [1378], 6155]: received a reply but the command has already timed out." 01:56:39 .jaeger 0 jaeger f text="[GET_STATUS, [1396], 6156]: received a reply but the command has already timed out." 01:56:39 OBSERVER.kgrabow 140 jaeger f error="Command 'power on --gfas' failed because of an uncaught error 'JaegerError: Some positioners failed to initialise: Positioner 199: cannot get status.'. See traceback in the log for more information." 01:56:39 CmdDone: OBSERVER.kgrabow 140 jaeger power on --gfas 01:57:33 .jaeger 0 jaeger f text="[SWITCH_OFF_PRECISE_MOVE_ALPHA, [170], 8310]: received a reply but the command has already timed out." 01:57:33 .jaeger 0 jaeger f text="[SWITCH_OFF_PRECISE_MOVE_ALPHA, [126], 8312]: received a reply but the command has already timed out." 01:57:33 .jaeger 0 jaeger f text="[SWITCH_OFF_PRECISE_MOVE_ALPHA, [245], 8314]: received a reply but the command has already timed out." .... --------------------- 01:57:34 OBSERVER.kgrabow 143 jaeger f error="Command 'power on' failed because of an uncaught error 'JaegerError: Some positioners failed to initialise: Positioner 1142: failed switching precise moves.'. See traceback in the log for more information." 01:57:34 CmdDone: OBSERVER.kgrabow 143 jaeger power on 01:58:35 .jaeger 0 jaeger f text="stopping positioners and cancelling due to SIGTERM" 01:59:03 OBSERVER.kgrabow 144 hub f NoTarget="the target named jaeger is not connected" 01:59:10 OBSERVER.kgrabow 145 hub f NoTarget="the target named jaeger is not connected" 01:59:38 .alerts 0 alerts e alert=fps.heartbeat,critical,"at 2022-03-11 01:59:38; last seen 120 sec ago",enabled,noack,-1 01:59:53 .alerts 0 alerts e alert=fps.heartbeat,critical,"at 2022-03-11 01:59:38; last seen 134 sec ago",enabled,noack,-1 02:00:02 .alerts 0 alerts e alert=fps.heartbeat,critical,"at 2022-03-11 01:59:38; last seen 143 sec ago",enabled,noack,-1 02:00:02 .alerts 0 alerts e alert=fps.heartbeat,critical,"at 2022-03-11 01:59:38; last seen 143 sec ago",enabled,noack,-1 02:00:10 .alerts 0 alerts e alert=fps.heartbeat,critical,"at 2022-03-11 01:59:38; last seen 151 sec ago",enabled,noack,-1 02:00:10 .alerts 0 alerts e alert=fps.heartbeat,critical,"at 2022-03-11 01:59:38; last seen 151 sec ago",enabled,noack,-1 02:00:23 OBSERVER.kgrabow 147 hub f NoTarget="the target named jaeger is not connected" --------------------- ..... 02:04:14 OBSERVER.kgrabow 154 jaeger f error="Command 'reconnect' failed." 02:04:14 CmdDone: OBSERVER.kgrabow 154 jaeger reconnect 0 ..... 02:07:37 CmdDone: OBSERVER.kgrabow 165 jaeger power off ..... Output from sdss5@sdss5-fps: systemctl --user restart jaeger (3.9.7) [sdss5@sdss5-fps ~]$ systemctl --user status jaeger ● jaeger.service - jaeger Loaded: loaded (/home/sdss5/config/services/services/jaeger.service; linked; vendor preset: enabled) Active: failed (Result: exit-code) since Fri 2022-03-11 01:59:05 UTC; 53s ago Process: 11117 ExecStart=/home/sdss5/config/services/execScripts/jaeger.sh (code=exited, status=1/FAILURE) Main PID: 11117 (code=exited, status=1/FAILURE) Mar 11 01:59:05 sdss5-fps jaeger.sh[11117]: File "/home/sdss5/opt/pyenv/vers...e Mar 11 01:59:05 sdss5-fps jaeger.sh[11117]: return future.result() Mar 11 01:59:05 sdss5-fps jaeger.sh[11117]: File "/home/sdss5/software/jaege...r Mar 11 01:59:05 sdss5-fps jaeger.sh[11117]: await fps.initialise() Mar 11 01:59:05 sdss5-fps jaeger.sh[11117]: File "/home/sdss5/software/jaege...e Mar 11 01:59:05 sdss5-fps jaeger.sh[11117]: raise JaegerError(f"Some positio...) Mar 11 01:59:05 sdss5-fps jaeger.sh[11117]: jaeger.exceptions.JaegerError: S.... Mar 11 01:59:05 sdss5-fps systemd[1927]: jaeger.service: main process exited...E Mar 11 01:59:05 sdss5-fps systemd[1927]: Unit jaeger.service entered failed .... Mar 11 01:59:05 sdss5-fps systemd[1927]: jaeger.service failed. Hint: Some lines were ellipsized, use -l to show in full. (3.9.7) [sdss5@sdss5-fps ~]$ systemctl --user restart jaeger