Subject: 2.5m obslog 20221203 (SJD 59917) From: dmbiz@apo.nmsu.edu Date: Sun, 4 Dec 2022 12:46:30 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Saturday December 03, 2022 (SJD 59917) ---- Observing Team ---- Dmitry Bizyaev Kaike Pan (R) ---- Observing Plan ---- Sat 03 Dec 2022 59917 0.86 00:54 01:09 12:37 22:57 10:27 --- bright until 10:27 ( 9.6 h, w/ 0.3 h for 12 deg design), then dark ( 2.2 h) ---- Observing Summary ---- Cloudy sky, fog, often rain. The entire night was lost. Absorbent pads were placed in the upper enclosure. ---- Engineering Issues ---- none ---- Software Issues ---- none ---- Observing Log ---- Afternoon --------- Many systems found responsive, some not. jaeger failed to get status of some robots, see Problem in Detail section. At the same time all GFAs and FVC failed to initialize and suggester the contained reboot, see Problem in Detail section. All fixed as: - jaeger power off, - restarted jaeger actor, - restarted fliswarm actor, - fliswarm reboot -n sdss-gfa{1,2,3,4,5,6,fvc} - jaeger power on --gfas - fliswarm reconnect After that the cameras, robots and actors started working normally. Collimation is reported as bad, but we are not going to open soon, so it is left as is to avoid changing settings again when it will be colder later. sp1: offset: r= 0 (In Focus); b= -8.3 (Out of focus) sp1: pred. move: spAverageMove= 1317 sp1: pred. resid: r= -1317, b= -4.1, txt= Bad angle: move blue ring -8.3 degrees then rerun gotoField with Hartmanns checked., spTemp = 5.8 Ventilation postponed due to rain and fog on site. Night ----- The entire night was lost. 12:45Z calibrations started LN2 force filled ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0110 0.0080 Axial B: 0.0070 0.0040 Axial C: -0.0230 -0.0290 Trans D: 0.0390 0.0320 Lateral E: 0.3315 0.3320 Lateral F: 0.5355 0.5350 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 92.83 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.0310 -0.0330 Axial B: -0.0240 -0.0290 Axial C: -0.0490 -0.0450 Trans D: -0.2770 -0.3170 GALILS Commanded: 4892000 4863000 4959150 -50 -3450 Actual: 4859211 4859167 4859199 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.99 -0.0 -0.01 -0.01 42.87 ---- Telescope Status ---- Not opened. No interlocks bypassed Status at: 12:38Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 696/714 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 215 lbs, 19 psi FPS Status ========== CP10 Loop = 7.16 L/min GFA Loop = 1.65 L/min Supply pressure = 41.4 psi Return pressure = 29.2 psi Chiller Status ============== Set temperarature = 7.5 C Flow = 5.2 Gpm ---- Software Used ---- TUI : 1.7.18b1 alerts : 2.3.0 boss : 6.0.2-alpha.0 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.3.1 jaeger : 1.2.0b1 kaiju : 1.3.0 coordio : 1.5.0b1 fps_calibrati : 2022.09.11+7ffa0ab cherno : 0.4.0b3 astrometrynet : 0.91 hal : 0.3.1a0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : 2.0.0a0 STUIAdditions : FAILED Boson : 0.1.2-beta.1 --------------Other Versions-------------- sdss-obstools : 3.9.2 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V35 Fiducials : v1_148 idlspec2d : v6_0_9 Kronos : 1.5.0 roboscheduler : 1.2.2 /dev/sdb1 2.0T 224G 1.8T 11% /home 10.25.1.230:/data 21T 8.8T 12T 44% /data ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 19:00 10.9 1.2 13.4 216.1 238 19:15 10.8 1.4 21.0 201.6 220 19:30 10.5 1.4 15.3 190.3 310 19:45 9.4 1.3 18.5 200.8 751 20:00 8.9 1.3 14.0 204.4 1988 20:15 8.1 1.3 12.6 189.2 2056 20:30 8.0 1.3 11.7 185.1 852 20:45 8.9 1.3 12.5 196.8 317 21:00 8.4 1.3 14.6 183.0 208 21:15 7.7 1.3 8.8 185.6 242 21:30 7.3 1.3 14.0 238.0 167 21:45 5.9 1.2 7.0 190.6 194 22:00 6.5 1.3 2.4 189.6 dpDep 22:15 5.6 1.3 6.8 258.6 dpDep 22:30 5.3 1.3 6.6 264.3 dpDep 22:45 4.4 1.3 5.3 261.3 dpDep 23:00 4.9 1.2 6.1 253.5 dpDep 23:15 5.0 1.2 1.9 288.4 dpDep 23:30 5.4 1.2 5.8 176.4 dpDep 23:45 5.4 1.2 5.1 192.3 dpDep 00:00 5.5 1.2 18.1 178.5 dpDep 00:15 7.3 1.2 17.8 182.4 dpDep 00:30 7.5 0.7 16.4 180.7 404 00:45 7.4 0.7 20.3 179.4 572 01:00 7.1 0.7 15.7 195.3 404 01:15 5.9 1.0 14.6 192.6 382 01:30 7.9 0.5 19.2 179.6 405 01:45 7.9 1.2 16.6 181.3 643 02:00 7.9 0.7 19.8 190.0 702 02:15 7.8 1.0 18.2 184.9 876 02:30 8.5 1.2 20.8 197.8 781 02:45 8.2 0.6 15.3 198.6 1004 03:00 8.1 1.0 23.0 196.6 1042 03:15 8.2 0.7 24.7 208.1 990 03:30 7.5 0.9 12.2 195.0 1020 03:45 8.1 1.0 17.6 207.6 1249 04:00 8.0 1.2 16.9 213.5 1335 04:15 8.5 1.2 19.2 202.0 1443 04:30 7.8 1.2 18.3 195.6 1674 04:45 7.9 1.2 16.5 223.0 1705 05:00 7.0 1.2 12.2 202.4 2094 05:15 6.7 -2.2 12.8 203.8 1996 05:30 6.6 1.0 17.6 205.0 2024 05:45 6.8 0.9 21.4 222.2 1719 06:00 7.1 1.2 20.5 217.1 1457 06:15 7.1 1.1 16.2 230.6 1446 06:30 6.6 1.2 12.2 211.6 1830 06:45 6.5 1.1 13.3 205.5 1795 07:00 7.0 1.1 11.0 249.8 1496 07:15 6.9 0.4 12.9 244.9 1500 07:30 7.4 0.2 12.0 263.6 1335 07:45 7.0 1.0 17.1 244.6 1435 08:00 7.6 1.1 17.9 231.9 1257 08:15 7.3 1.0 17.4 234.1 1201 08:30 7.1 1.0 13.7 250.9 1159 08:45 7.2 0.9 11.6 257.5 1244 09:00 7.2 0.9 13.0 234.9 1561 09:15 7.4 0.9 15.0 250.1 1336 09:30 7.4 0.9 17.9 231.8 1198 09:45 7.4 0.9 15.3 257.6 1303 10:00 6.9 0.9 11.3 254.8 1492 10:15 7.3 1.0 14.3 262.5 1557 10:30 7.4 0.9 15.1 253.6 1498 10:45 7.5 1.0 13.0 265.0 1520 11:00 7.3 1.0 14.3 263.2 1472 11:15 7.5 0.8 14.4 264.9 1532 11:30 7.0 0.7 13.0 264.8 1838 11:45 6.9 0.8 11.2 271.9 1813 12:00 6.7 -0.1 10.5 271.8 1624 12:15 6.7 1.0 14.5 274.8 1525 12:30 6.9 0.9 12.8 280.5 1215 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 59917 Fill night time in Hours below *=*=*= Total bright time : 2.2 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 2.2 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 9.6 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 9.6 *=*=*= Dark time lost to technical issues : 0 ---- Problems in Detail ---- 23:15:00 OBSERVER.dmbiz 38 fliswarm w text="Node fvc is not pinging back or the Docker daemon is not running. Try rebooting the computer." 23:15:00 OBSERVER.dmbiz 38 fliswarm w text="Node gfa5 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 23:15:00 OBSERVER.dmbiz 38 fliswarm w text="Node gfa6 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 23:15:00 OBSERVER.dmbiz 38 fliswarm w text="Node gfa2 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 23:15:00 OBSERVER.dmbiz 38 fliswarm w text="Node gfa4 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 23:15:00 OBSERVER.dmbiz 38 fliswarm w text="Node gfa3 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 23:15:00 OBSERVER.dmbiz 38 fliswarm w text="Node gfa1 is not pinging back or the Docker daemon is not running. Try rebooting the computer." 23:17:38 OBSERVER.dmbiz 40 jaeger f error="Command 'power on --gfas' failed because of an uncaught error 'JaegerError: Some positioners failed to initialise: Positioner 199: cannot get status.'. See traceback in the log for more information." 23:18:29 OBSERVER.dmbiz 41 jaeger f error="Command 'power on --gfas' failed because of an uncaught error 'JaegerError: Some positioners failed to initialise: Positioner 1363: cannot get status.'. See traceback in the log for more information."