Subject: 2.5m obslog 20250401 (SJD 60767) From: doravetz@apo.nmsu.edu Date: Wed, 2 Apr 2025 11:46:46 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday April 01, 2025 (SJD 60767) ---- Observing Team ---- Dan Oravetz ---- Observing Plan ---- Tue 01 Apr 2025 60767 dark (9.1 h) plus 0.9 h for 8 deg design ---- Observing Summary ---- The night was lost to weather. It was incredibly dusty (1.0um counts peaked >200k), windy, and cloudy. By the latter half of the night the clouds were sometimes accompanied by precipitation radar returns. During the last couple hours the wind gusts had died down and the dust was acceptable, but additional waves of clouds encroaching from the W hampered our chances. Notes ===== During afternoon checkout I experienced another instance of the jaeger time-outs (with resultant "Kaiju-collided grid" errors) that I experienced March 16th. Rather than relying on power-cycling the entire IEB, however, powering-off, and then on, the CANs from jaeger (followed by a jaeger restart and then exploding any reportedly software-collided positioners) restored things. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ All systems alive and responsive. Disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 467G 1.6T 23% /home /dev/sdd1 22T 13T 8.5T 60% /data FPS powered on. Positioners are home/folded: 20:45:20 APO.doravetz 40 jaeger i folded=T 20:45:20 APO.doravetz 40 jaeger : text="All positioners are folded." Loaded a manual folded configuration: 20:47:27 CmdStarted: APO.doravetz 2 jaeger configuration load --from-positions 20:47:29 APO.doravetz 2 jaeger d text="Resetting cherno offsets." 20:47:29 APO.doravetz 2 jaeger d text="Focal plane scale: 1.0." 20:47:29 APO.doravetz 2 jaeger d configuration_loaded=20170,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/020XXX/0201XX/confSummary-20170.par,F 20:47:34 APO.doravetz 2 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/60767/configuration_snapshot_60767_20170.pdf 20:47:34 APO.doravetz 2 jaeger d design_preloaded=-999 20:47:34 APO.doravetz 2 jaeger : text="Configuration 20170 loaded." 20:47:34 CmdDone: APO.doravetz 2 jaeger configuration load --from-positions Took a '--no-apply' FVC image: FVC loop results: fvc_rms= 72.78 fvc_deltarms= 35.64 fvc_filename=/data/fcam/60767/fimg-fvc1n-0001.fits fvc_perc_90= 94.50 fvc_percent_reached= 1.40 fvc iterations= 1 Scattered-light check: 20:48:34 CmdStarted: APO.doravetz 4 fliswarm talk -n fvc expose 50 fimg-fvc1n-0002.fits shows no evidence of light artifacts. Exercised positioners: permanently_disabled=201,275,460,478,503,535,639,716,802,846,935,984,985,995,997,1026,1043,1049,1202,1231,1246,1290 manually_disabled=1187 20:52:17 CmdStarted: APO.doravetz_3 3 jaeger configuration random --danger 20:53:37 .jaeger 0 jaeger e text="A collision was detected in positioner 1347. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 20:53:37 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 20:53:39 APO.doravetz_3 3 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/60766/trajectory-60766-0031.json 20:53:39 APO.doravetz_3 3 jaeger f error="Trajectory failed with error: Something went wrong during the trajectory: The FPS got locked during the trajectory. Trajectory failed 41.80 seconds after start" 20:53:39 CmdDone: APO.doravetz_3 3 jaeger configuration random --danger 20:53:39 .jaeger 0 jaeger e locked=T; locked_by=1347; locked_axes=alpha; locked_alpha=9.991988316178322; locked_beta=169.993257522583 20:56:26 .jaeger 0 jaeger w text="GET_STATUS timed out. Retrying." 21:00:38 CmdStarted: APO.doravetz 6 jaeger power off cm1 cm2 cm3 cm4 cm5 cm6 21:00:44 CmdDone: APO.doravetz 6 jaeger power off cm1 cm2 cm3 cm4 cm5 cm6 21:01:45 CmdStarted: APO.doravetz 7 jaeger power on cm1 cm2 cm3 cm4 cm5 cm6 21:02:02 CmdDone: APO.doravetz 7 jaeger power on cm1 cm2 cm3 cm4 cm5 cm6 Restarted jaeger and re-disabled positioner 1187. Still no luck with the 'unwind': 21:04:47 APO.doravetz_3 16 jaeger f error="Command 'unwind' failed because of an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate paths'. See traceback in the log for more information." Positioner 493 was reportedly software-collided, so I exploded it and in snapshot #37 there are no collided positioners at all. Tried another 'unwind' which worked this time. 21:10:45 CmdStarted: APO.doravetz 12 jaeger configuration random --danger 21:12:03 APO.doravetz 12 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/60766/trajectory-60766-0037.json 21:12:03 CmdDone: APO.doravetz 12 jaeger configuration random --danger 21:12:08 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/60766/fps_snapshot_60766_0039.pdf 21:12:34 CmdStarted: APO.doravetz_3 20 jaeger configuration reverse GFA biases/darks: 21:13:58 CmdStarted: APO.doravetz_3 21 fliswarm talk -c gfa expose --bias --count 10 21:17:03 CmdStarted: APO.doravetz_3 22 fliswarm talk -c gfa expose --dark --count 5 5 BOSS arc and flat: 21:44:01 BOSS expNum=383722, expTime= 4.0s 21:45:54 BOSS expNum=383723, expTime= 25.0s Did not start ventilation because of very high dust. ============================================================================ NIGHT ============================================================================ Did not open at sunset. Very windy and dusty. 05:00Z - Still incredibly dusty, very windy, and cloudy. 08:09Z - Still windy and dusty and the clouds now have associated radar-return precip signatures. 09:18Z - Wind gusts are down and dust is now only >4k counts, but still dealing with clouds and another big wave of clouds is moving in from the W to add to the current cloud issue. 11:40Z - Wind gusts and dust are both still in the yellow, but certainly better compared to earlier. The clouds are beginning to swing north and actually it would almost be ok to open soon, but it's by the time the cloud trend improves enough it will be too late into twilight now. Called it a night. FPS is powered-off. Positioners are home/folded. Started APOGEE morning calibration sequence. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0020 0.0040 Axial B: -0.0008 -0.0010 Axial C: -0.0040 -0.0060 Trans D: -0.0080 -0.0100 Lateral E: 0.3380 0.3380 Lateral F: 0.5450 0.5455 GALILS Commanded: 9800 -300 6750 19150 -3550 -3250 Actual: 9800 -300 6750 19150 -3550 -3250 SETMIR VALUES PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.1010 -0.0970 Axial B: -0.0630 -0.0650 Axial C: -0.0420 -0.0420 Trans D: -0.0030 -0.0030 GALILS Commanded: 5888550 5831600 5961750 -1500 -7250 Actual: 5883485 5811712 5906112 -1500 -7250 SETMIR VALUES SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 SecOrient: 4656.95 -24.49 -17.44 17.73 125.25 ---- Telescope Status ---- Status at close: Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 647 655 652 650 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 271 lbs, 19.8 psi ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.5.1a0 jaeger : 1.9.2a0 kaiju : 1.4.0 coordio : 1.13.0 fps_calibrati : 2025.02.24 cherno : 1.1.0 astrometrynet : 0.93 hal : 1.4.1 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1 -mcp.plc : 35 -mcp.azFiduci : v1_150 -mcp.altFiduc : v1_150 -mcp.rotFiduc : v1_150 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev STUIAdditions : 3.1.18 boson : 0.4.9 sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 PLC : V35 Fiducials : v1_150 idlspec2d : v6_2_0 Kronos : 2.0.0 roboscheduler : 2.1.0 ---- APOGEE Data Summary ---- 60766 morning cals: num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- 52040047 101730 713625 20168 DARK 13.0 60 -27 True -- -- -- -- -- -- 52040048 101730 713625 20168 DARK 13.0 60 7 True -- -- -- -- -- -- 52040049 101730 713625 20168 DARK 13.0 60 0 True -- -- -- -- -- -- 52040050 101730 713625 20168 QUARTZFLAT 13.0 10 100 True -- -0.16 -- 0.86 -- -- 52040051 101730 713625 20168 QUARTZFLAT 13.0 10 101 True -- -0.16 -- 0.86 -- -- 52040052 101730 713625 20168 QUARTZFLAT 13.0 10 102 True -- -0.16 -- 0.87 -- -- 52040053 101730 713625 20168 ARCLAMP-THAR 13.5 12 103 True 1.97 0.64 1.07 0.90 -- -- 52040054 101730 713625 20168 ARCLAMP-UNE 13.5 40 98 True 1.54 0.49 1.05 0.88 -- -- 52040055 101730 713625 20168 ARCLAMP-THAR 13.0 12 95 True 1.46 0.60 1.07 0.88 -- -- 52040056 101730 713625 20168 ARCLAMP-UNE 13.0 40 99 True 1.04 0.44 1.08 0.89 -- -- 52040057 101730 713625 20168 DARK 13.0 30 23 False -- -- -- -- -- -- 52040058 101730 713625 20168 INTERNALFLAT 13.0 30 99 True -- -- -- -- -- -- 52040059 101730 713625 20168 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 52040060 101730 713625 20168 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 52040061 101730 713625 20168 DARK 13.0 30 263 True -- -- -- -- -- -- 52040062 101730 713625 20168 FPI 13.0 30 203 True 12.88 0.40 1.15 0.89 -- -- 52040063 101730 713625 20168 FPI 13.5 30 204 True -11.23 0.42 1.16 0.88 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 52040050 QUARTZFLAT 10 67 71 81 89 89 85 87 89 88 81 52040051 QUARTZFLAT 10 69 74 82 90 90 86 88 89 89 83 52040052 QUARTZFLAT 10 70 75 82 90 90 86 88 89 89 83 (observers39) [observer@sdss5-hub: ~]$ check_ap 60767 num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ---- ------- ------- ------ ------ ---- ---- 52050001 101730 713625 20168 DARK 13.5 3 100 True -- -- -- -- -- -- 52050002 101730 713625 20168 DARK 13.5 60 -2 True -- -- -- -- -- -- 52050003 -999 -999 20170 QUARTZFLAT 13.5 10 102 True -- -0.27 -- 0.86 -- -- 52050004 -999 -999 20170 ARCLAMP-THAR 13.5 12 99 True 1.99 0.54 1.07 0.89 -- -- 52050005 -999 -999 20170 ARCLAMP-UNE 13.5 40 104 True 1.58 0.42 1.09 0.90 -- -- 52050006 -999 -999 20170 ARCLAMP-THAR 13.0 12 92 True 1.46 0.51 1.05 0.88 -- -- 52050007 -999 -999 20170 ARCLAMP-UNE 13.0 40 105 True 1.08 0.35 1.10 0.89 -- -- 52050008 -999 -999 20171 DARK 13.0 60 7 True -- -- -- -- -- -- 52050009 -999 -999 20171 FPI 13.0 30 212 True 12.90 0.32 1.16 0.88 -- -- 52050010 -999 -999 20171 FPI 13.5 30 215 True -11.20 0.33 1.18 0.88 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 52050003 QUARTZFLAT 10 66 70 81 90 91 85 88 90 89 82 ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60767 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 20:37 b1-00383717 ----------- ---- bia Clo 0.0 887 1011 983 2 20:37 r1-00383717 ----------- ---- bia Clo 0.0 887 1011 983 3 21:38 r1-00383718 -999--999 20171 arc Lef 4.1 887 1011 983 HART 4 21:38 b1-00383718 -999--999 20171 arc Lef 4.1 887 1011 983 HART 5 21:39 r1-00383719 -999--999 20171 arc Rig 4.1 887 1011 983 HART 6 21:39 b1-00383719 -999--999 20171 arc Rig 4.1 887 1011 983 HART 7 21:41 b1-00383720 -999--999 20171 arc Lef 4.1 -390 -267 -295 HART 8 21:41 r1-00383720 -999--999 20171 arc Lef 4.1 -390 -267 -295 HART 9 21:42 r1-00383721 -999--999 20171 arc Rig 4.1 -390 -267 -295 HART 10 21:42 b1-00383721 -999--999 20171 arc Rig 4.1 -390 -267 -295 HART 11 21:44 r1-00383722 -999--999 20171 arc Out 4.1 -366 -243 -272 12 21:44 b1-00383722 -999--999 20171 arc Out 4.1 -366 -243 -272 13 21:45 b1-00383723 -999--999 20171 fla Out 25.1 -366 -243 -272 14 21:45 r1-00383723 -999--999 20171 fla Out 25.1 -366 -243 -272 ---- BOSS Spectrographs Collimation ---- Afternoon focus check ===================== 21:41:55 BOSS expNum=383720, expTime= 4.0s 21:42:35 BOSS expNum=383721, expTime= 4.0s 21:43:16 sp1.motor.move= 24, 24, 23 21:43:16 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -2.2 (In Focus) sp1: pred. move: spAverageMove= 24 sp1: pred. resid: r= -683, b= -2.1, txt= OK, spTemp = 5.5 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60767 Fill night time in Hours below *=*=*= Total bright time : 0.9 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0.9 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 9.1 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 9.1 *=*=*= Dark time lost to technical issues : 0