Subject: 2.5m obslog 20221014 (SJD 59867) From: doravetz@apo.nmsu.edu Date: Sat, 15 Oct 2022 10:37:17 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday October 14, 2022 (SJD 59867) ---- Observing Team ---- Dan Oravetz Dylan Gatlin ---- Observing Plan ---- Fri 14 Oct 2022 59867 dark until 03:45 (2.1 h), then bright (8.5 h, w/ 0.2 h for 12 deg design) ---- Observing Summary ---- The night was lost to weather: clouds and associated radar returns. We completed a home/relax of M1 and a home of M2. Saved new nominal MIGS. Had a few problems getting 'fliswarm reconnect' to complete during afternoon checkout. Eventually, a manual power-cycle followed by a "soft" reboot restored things to normal. There are details under Afternoon Checkout of the Log body. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ All systems alive and responsive upon arrival, except there were some issues with getting 'fliswarm reconnect' to complete after powering on FPS (see below). Afternoon disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 196G 1.9T 10% /home 10.25.1.230:/data 21T 15T 5.4T 74% /data ********************************** FLISWARM RECONNECT ISSUES -- start ********************************** After powering-on FPS, the 'fliswarm reconnect' produced an error: 23:52:10 APO.doravetz 406 fliswarm f error="Command 'reconnect' failed because of an uncaught error 'ConnectionRefusedError: [Errno 111] Connect call failed ('10.25.1.125', 19995)'. See traceback in the log for more information." 23:52:10 CmdDone: APO.doravetz 406 fliswarm reconnect Pinged 10.25.1.125 just to makes sure it was responsive and it returned pings. Tried the 'fliswarm reconnect' again, but now sdss-gfa4 was producing a connection timeout: 23:52:47 APO.doravetz 407 fliswarm f error="Command 'reconnect' failed because of an uncaught error 'ConnectTimeout: HTTPConnectionPool(host='sdss-gfa4', port=2375): Max retries exceeded with url: /v1.41/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22name%22%3A+%5B%22flicamera-gfa4%22%5D%2C+%22status%22%3A+%5B%22running%22%5D%7D (Caused by ConnectTimeoutError(, 'Connection to sdss-gfa4 timed out. (connect timeout=3)'))'. See traceback in the log for more information." 23:52:47 CmdDone: APO.doravetz 407 fliswarm reconnect 23:55:11 APO.doravetz 413 fliswarm w text="Node sdss-gfa4 is not pinging back." 23:56:35 APO.doravetz 414 fliswarm w text="Node gfa4 is not pinging back or the Docker daemon is not running. Try rebooting the computer." Tried a reboot of sdss-gfa4: 23:57:15 CmdStarted: APO.doravetz 416 fliswarm reboot -n sdss-gfa4 Still no luck; same sdss-gfa4 error as above. Tried a hard reboot: 23:59:13 CmdStarted: APO.doravetz 421 fliswarm reboot -n sdss-gfa4 -f That also didn't fix it. At the telescope I removed the IEB cover and verified all the connections to and from sdss-gfa4 were solid, then I manually rebooted sdss-gfa4. Next I issued a "soft" reboot of sdss-gfa4 after the power cycle because there had been an error about "No containers running to sdss-gfa4." 00:09:18 CmdStarted: APO.telescope 491 fliswarm reboot -n sdss-gfa4 The manual power cycle and the subsequent soft reboot evidently fixed the connection problem with sdss-gfa4 because sdss-gfa4 was back according to 'fliswarm status', but now there's a "No route to host." error instead: 00:12:57 APO.doravetz 424 fliswarm f error="Command 'reconnect' failed because of an uncaught error 'OSError: [Errno 113] No route to host'. See traceback in the log for more information." Restarted fliswarm from systemctl and now 'fliswarm reconnect' completes successfully. ********************************** FLISWARM RECONNECT ISSUES -- end ********************************** FPS powered on. Positioners are home/folded: 23:54:13 CmdStarted: APO.doravetz 409 jaeger unwind --status 23:54:14 APO.doravetz 409 jaeger i folded=T 23:54:14 APO.doravetz 409 jaeger : text="All positioners are folded." 23:54:14 CmdDone: APO.doravetz 409 jaeger unwind --status Loaded a manual folded configuration: 00:25:11 CmdStarted: APO.doravetz 426 jaeger configuration load --from-positions 00:25:17 APO.doravetz 426 jaeger d text="Focal plane scale: 1.0." 00:25:17 APO.doravetz 426 jaeger d configuration_loaded=7034,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/0070XX/confSummary-7034.par,F 00:25:21 APO.doravetz 426 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59867/configuration_snapshot_59867_7034.pdf 00:25:21 APO.doravetz 426 jaeger d design_preloaded=-999 00:25:21 APO.doravetz 426 jaeger : text="Configuration 7034 loaded." 00:25:21 CmdDone: APO.doravetz 426 jaeger configuration load --from-positions Took a '--no-apply' FVC image: 01:24:42 CmdStarted: APO.doravetz 430 jaeger fvc loop --no-apply fvc_rms= 26.21 fvc_deltarms= 26.2 fvc_filename=/data/fcam/59867/fimg-fvc1n-0001.fits fvc_perc_90= 40.30 fvc_percent_reached= 6.00 01:25:33 CmdDone: APO.doravetz 430 jaeger fvc loop --no-apply Scattered-light check: 01:26:35 CmdStarted: APO.doravetz 431 boss exposure science itime=300 01:26:42 CmdStarted: APO.doravetz 432 fliswarm talk -n fvc expose 50 fimg-fvc1n-0002.fits shows no evidence of scattered light. Ventilation not started due to weather. ============================================================================ NIGHT ============================================================================ We did not open on time due to clouds and associated radar returns. ********************************** HOME AND RELAX M1, HOME M2 - start ********************************** == M1 == desOrient=0.00,-59.50,-9.15,-512.00,1906.30,0.00 home E,F: "position error", "999999999, 999999999, 999999999, 999999999, 000000000, 000000000" home D: "position error", "999999999, 999999999, 999999999, 000000000, 999999999, 999999999" home A,B,C: "position error", "000000000, 000000000, 000000050, 999999999, 999999999, 999999999" 02:57:05 CmdStarted: APO.doravetz 458 tcc queue run=relax 03:00:01 .tcc 0 tcc : JobStatus=relax,Done == M2 == home D,E: "position error", "999999999, 999999999, 999999999, 000000000, -000000050" home A,B,C: "position error", "-000000159, -000000026, 000000018, 999999999, 999999999" Took mirror numbers and then saved new Nominal MIGs in Mirror Numbers. ********************************** HOME AND RELAX M1, HOME M2 - end ********************************** 09:44Z - Still clouded-out. Radar returns to the north. Satellite agrees with clear sky clock forecast that clouds will persist. Powered-off FPS. Started APOGEE morning cals. 10:34Z - Cloudy. Turning things off then calling it a night. ---- Mirror Numbers ---- Mirror numbers taken from checkout before the home/relax: PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0030 0.0060 Axial B: -0.0080 -0.0030 Axial C: -0.0040 0.0000 Trans D: 0.0100 0.0180 Lateral E: 0.3315 0.3315 Lateral F: 0.5350 0.5345 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 12.44 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.0510 -0.0490 Axial B: -0.0470 -0.0430 Axial C: -0.0660 -0.0670 Trans D: -0.2200 -0.2460 GALILS Commanded: 4914950 4891100 4988300 0 -3400 Actual: 4859085 4859173 4859079 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3856.92 0.02 0.02 0.01 41.63 Mirror numbers taken after home/relax: PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0060 0.0060 Axial B: -0.0030 -0.0030 Axial C: 0.0000 0.0000 Trans D: 0.0270 0.0180 Lateral E: 0.3315 0.3315 Lateral F: 0.5335 0.5345 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 12.44 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.0470 -0.0490 Axial B: -0.0440 -0.0430 Axial C: -0.0630 -0.0670 Trans D: -0.2810 -0.2460 GALILS Commanded: 4911150 4897100 4987200 -50 -3450 Actual: 4859217 4859236 4859293 -50 -3450 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.02 -0.03 0.0 0.0 42.89 Saved new Nominal MIGS: PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0060 0.0060 Axial B: -0.0030 -0.0030 Axial C: 0.0000 0.0000 Trans D: 0.0270 0.0270 Lateral E: 0.3315 0.3315 Lateral F: 0.5335 0.5335 GALILS Commanded: 12500 -1600 5300 47050 -8800 -8400 Actual: 12500 -1600 5300 47050 -8800 -8400 SETMIR VALUES PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 SECONDARY: ---------- Focus: 0.0000 Air temp: 12.44 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.0470 -0.0470 Axial B: -0.0440 -0.0440 Axial C: -0.0630 -0.0630 Trans D: -0.2810 -0.2810 GALILS Commanded: 4910800 4896900 4986900 0 -3400 Actual: 4859362 4859255 4859287 0 -3400 SETMIR VALUES SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 SecOrient: 3857.07 0.0 -0.04 -0.03 41.64 ---- Telescope Status ---- Status at: 10:36Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 695 696 714 713 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 240 lbs, 19.7 psi FPS Status ========== CP10 Loop = 7.22 L/min GFA Loop = 1.54 L/min Supply pressure = 41.2 psi Return pressure = 29.0 psi Chiller Status ============== Set temperarature = 11.0 C Flow = 5.2 Gpm ---- Software Used ---- STUI : 1.7.18b1 alerts : 2.3.0 boss : 6.0.2-alpha.0 -boss.specMec : 3.0.4 apo : 3.0.1 fliswarm : 0.3.1 jaeger : 1.0.2a0 kaiju : 1.2.4-alpha.0 coordio : 1.4.5-alpha.0 fps_calibrati : 2022.09.11+7ffa0ab cherno : 0.3.3a0 astrometrynet : 0.91 hal : 0.3.0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_148 -mcp.altFiduc : v1_148 -mcp.rotFiduc : v1_148 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev sdss-obstools : 3.9.1 TPM : 1.2-31-gd7eb MCP : 2.1 PLC : V35 Fiducials : v1_148 idlspec2d : v6_0_9 Kronos : 1.5.0 roboscheduler : 1.3.0 boson : 0.1.1 ---- APOGEE Data Summary ---- 59866 morning cals: APOGEE data list, mjd=59866, all raw_data: /data/apogee/utr_cdr/59866/apRaw-*.fits archive: /data/apogee/archive/59866/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59866/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 59866 75 12:37 43040075 Dark 60 A 13.0 ---------- a-b-c - - 59866 76 12:47 43040076 Dark 60 A 13.0 ---------- a-b-c - - 59866 77 12:58 43040077 Dark 60 A 13.0 ---------- a-b-c - - 59866 78 13:00 43040078 QuarFlat 10 A 13.0 ---------- a-b-c - - 59866 79 13:02 43040079 QuarFlat 10 A 13.0 ---------- a-b-c - - 59866 80 13:04 43040080 QuarFlat 10 A 13.0 ---------- a-b-c - - 59866 81 13:06 43040081 Arc-Thar 12 B 13.5 ---------- a-b-c 2.63 - 59866 82 13:13 43040082 Arc-Une 40 B 13.5 ---------- a-b-c - - 59866 83 13:15 43040083 Arc-Thar 12 A 13.0 ---------- a-b-c 2.11 - 59866 84 13:22 43040084 Arc-Une 40 A 13.0 ---------- a-b-c - - 59866 85 13:28 43040085 Dark 30 A 13.0 ---------- a-b-c - - 59866 86 13:33 43040086 InteFlat 30 A 13.0 ---------- a-b-c - - 59866 87 13:39 43040087 InteFlat 30 A 13.0 ---------- a-b-c - - 59866 88 13:44 43040088 InteFlat 30 A 13.0 ---------- a-b-c - - 59866 89 13:50 43040089 Dark 30 A 13.0 ---------- a-b-c - - 59866 90 13:55 43040090 Arc---- 30 A 13.0 ---------- a-b-c - 92 FPI ------------------------------------------------------------------------------ APOGEE data list, mjd=59867, all raw_data: /data/apogee/utr_cdr/59867/apRaw-*.fits archive: /data/apogee/archive/59867/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/59867/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 59867 1 23:48 43050001 Dark 3 A 13.0 ---------- a-b-c - - 59867 2 00:01 43050002 Dark 60 A 13.0 ---------- a-b-c - - 59867 3 00:03 43050003 QuarFlat 10 A 13.0 ---------- a-b-c - - 59867 4 00:05 43050004 Arc-Thar 12 B 13.5 ---------- a-b-c 2.61 - 59867 5 00:12 43050005 Arc-Une 40 B 13.5 ---------- a-b-c - - 59867 6 00:14 43050006 Arc-Thar 12 A 13.0 ---------- a-b-c 2.07 - 59867 7 00:22 43050007 Arc-Une 40 A 13.0 ---------- a-b-c - - 59867 8 00:33 43050008 Dark 60 A 13.0 ---------- a-b-c - - 59867 9 00:38 43050009 Arc---- 30 A 13.0 ---------- a-b-c - 89 FPI ------------------------------------------------------------------------------ ---- eBOSS Data Summary ---- Reading FITS from /data/spectro//59867 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 23:48 b1-00346944 ----------- ---- bia Clo 0.0 -1386 -1246 -2583 2 23:48 r1-00346944 ----------- ---- bia Clo 0.0 -1386 -1246 -2583 3 00:26 b1-00346945 -999--999 7034 arc Lef 4.1 -1386 -1246 -2583 HART 4 00:26 r1-00346945 -999--999 7034 arc Lef 4.1 -1386 -1246 -2583 HART 5 00:27 b1-00346946 -999--999 7034 arc Rig 4.1 -1386 -1246 -2583 HART 6 00:27 r1-00346946 -999--999 7034 arc Rig 4.1 -1386 -1246 -2583 HART 7 00:28 b1-00346947 -999--999 7034 arc Lef 4.1 96 237 -1101 HART 8 00:28 r1-00346947 -999--999 7034 arc Lef 4.1 96 237 -1101 HART 9 00:29 b1-00346948 -999--999 7034 arc Rig 4.1 96 237 -1101 HART 10 00:29 r1-00346948 -999--999 7034 arc Rig 4.1 96 237 -1101 HART 11 00:33 b1-00346949 -999--999 7034 arc Out 4.1 276 417 -921 12 00:33 r1-00346949 -999--999 7034 arc Out 4.1 276 417 -921 13 01:26 b1-00346950 -999--999 7034 sci Out 300.1 276 417 -921 14 01:26 r1-00346950 -999--999 7034 sci Out 300.1 276 417 -921 15 02:17 b1-00346951 -999--999 7034 arc Lef 4.1 276 417 -921 HART 16 02:17 r1-00346951 -999--999 7034 arc Lef 4.1 276 417 -921 HART 17 02:18 r1-00346952 -999--999 7034 arc Rig 4.1 276 417 -921 HART 18 02:18 b1-00346952 -999--999 7034 arc Rig 4.1 276 417 -921 HART 19 02:19 r1-00346953 -999--999 7034 arc Out 4.1 456 597 -741 20 02:19 b1-00346953 -999--999 7034 arc Out 4.1 456 597 -741 21 02:21 b1-00346954 -999--999 7034 fla Out 25.1 456 597 -741 22 02:21 r1-00346954 -999--999 7034 fla Out 25.1 456 597 -741 ---- BOSS Spectrographs Collimation ---- Afternoon focus check ===================== 00:28:35 boss exposure 4.0, file=346947 00:29:16 boss exposure 4.0, file=346948 00:29:57 sp1.motor.move= 180, 180, 180 00:29:57 Hartmann collimate output sp1: offset: r= -989 (In Focus); b= -4.2 (Out of focus) sp1: pred. move: spAverageMove= 180 sp1: pred. resid: r= -1169, b= -3.7, txt= OK, spTemp = 7.3 00:29:54 APO.doravetz 428 hartmann i r1MeanOffset=-0.15,"In Focus" 00:29:54 APO.doravetz 428 hartmann w b1MeanOffset=0.21,"Out of focus" ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 59867 Fill night time in Hours below *=*=*= Total bright time : 8.7 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 8.7 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 2.1 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 2.1 *=*=*= Dark time lost to technical issues : 0