Subject: 2.5m obslog 20240702 (SJD 60494) From: doravetz@apo.nmsu.edu Date: Wed, 3 Jul 2024 09:24:15 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Tuesday July 02, 2024 (SJD 60494) ---- Observing Team ---- Dan Oravetz ---- Observing Plan ---- Tue 02 Jul 2024 60494 dark (7.0 h) plus 1.0 h for 8 deg design ---- Observing Summary ---- Another night of heavy clouds, nearby storms, and some rain. Notes ===== (1) The APOGEE Collimator status was reporting '1 1 1 1 1 1' this afternoon, but an 'apogee coll init' restored the normal 'OK' state. (2) Restarted apogeeql for the updated sdsscore changes to take effect. (3) I ran preload-load-FVC loop sequences on several designs from the Obs Queue this evening to make sure there weren't any obvious errors as a result of the sdsscore updates from this afternoon. I didn't see any issues - confSummary files were being written and saved according to the jaeger output. (4) I replicated the Kronos problem documented in the 60486 night log in which design 621638 was loaded in hal, but not indicated as loaded in Kronos. Details in the Log body, but the crux of it seems to be related to loading it after a cloned dark_rm design because I loaded it earlier tonight and Kronos handled it just fine. Jira ticket COS-116 filed for this issue. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ All systems alive and responsive. Disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 465G 1.6T 23% /home /dev/sdd1 22T 14T 6.9T 68% /data FPS powered on. Positioners are home/folded: 22:23:09 APO.doravetz_3 37 jaeger i folded=T 22:23:09 APO.doravetz_3 37 jaeger : text="All positioners are folded." Loaded a manual folded configuration: 22:15:23 CmdStarted: APO.doravetz_3 32 jaeger configuration load --from-positions 22:15:25 APO.doravetz_3 32 jaeger d text="Resetting cherno offsets." 22:15:25 APO.doravetz_3 32 jaeger d text="Focal plane scale: 1.0." 22:15:25 APO.doravetz_3 32 jaeger d configuration_loaded=15068,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/015XXX/0150XX/confSummary-15068.par,F 22:15:30 APO.doravetz_3 32 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/60494/configuration_snapshot_60494_15068.pdf 22:15:30 APO.doravetz_3 32 jaeger d design_preloaded=-999 22:15:30 APO.doravetz_3 32 jaeger : text="Configuration 15068 loaded." 22:15:30 CmdDone: APO.doravetz_3 32 jaeger configuration load --from-positions Took a '--no-apply' FVC image. 22:16:28 FVC Loop fvc_rms= 40.8 fvc_deltarms= None fvc_filename=/data/fcam/60494/fimg-fvc1n-0001.fits fvc_perc_90= 61.30 fvc_percent_reached= 1.00 fvc iterations= 1 Scattered-light check: 22:17:18 CmdStarted: APO.doravetz_3 34 fliswarm talk -n fvc expose 50 fimg-fvc1n-0002.fits shows no evidence of light artifacts. Exercised positioners: 22:18:24 CmdStarted: APO.doravetz_3 35 jaeger configuration random --danger 22:20:04 APO.doravetz_3 35 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/60493/trajectory-60493-0005.json 22:20:05 CmdDone: APO.doravetz_3 35 jaeger configuration random --danger 22:20:09 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/60493/fps_snapshot_60493_0005.pdf 22:21:15 CmdStarted: APO.doravetz_3 36 jaeger configuration reverse GFA biases/darks: 22:24:14 CmdStarted: APO.doravetz_3 38 fliswarm talk -c gfa expose --bias --count 10 22:25:43 CmdStarted: APO.doravetz_3 39 fliswarm talk -c gfa expose --dark --count 5 5 Ventilation delayed due to threatening clouds and approaching storms. ============================================================================ NIGHT ============================================================================ Did not open at sunset due to approaching storms. I ran preload, load, FVC loop on several designs from the Obs Queue this evening to make sure there weren't any obvious errors as a result of the sdsscore updates from this afternoon. I didn't see any issues - confSummary files were being written and saved according to the jaeger output. I took the opportunity tonight to try and reproduce the error from 60486 regarding Kronos not recognizing 621638 had been loaded. First, after having loaded the first two bright_time designs from the queue without trouble, I prepended 621638 to the queue and then loaded it. In this case, Kronos properly updated the Obs Queue with 621638 displayed with strike-through text. Next, as was the case on 60486, I loaded a dark_rm design, then loaded the next (cloned) design. At this point, I prepended 621638 to the queue and preloaded it. However, after I loaded it Kronos did not update the Obs Queue to reflect 621638 had been loaded (no strike-through text). I clicked the Plan Observing link again, but 621638 (which was loaded in hal) still was not displayed as having been loaded in Kronos. I tried refreshing the page with the browser refresh button and Kronos still did not display 621638 as having been loaded. Regardless, I tried a preload and, unsurprisingly, 621638 was cloned (which is what happened on 60486 and which is NOT what should happen). Given that 621638 did display as loaded previously during this test, it seems there's a problem with it having followed a cloned dark_rm design? I didn't want to do too much additional testing since this sequence replicated the problem. It's simple to test various orders of when to attempt loading this design, but I figured John Donor might want to investigate up to this point first before adding any additional layers. Here's the whole sequence of loading designs from tonight: **************************************************************** 16:21:23 Field-Design-Configuration: 101171-615281-15067 **************************************************************** 20:27:13 mcp.gang=17 (Disconnected w FPI) 20:29:04 mcp.gang=28 (Podium: Dense FPI) 22:12:03 BOSS expNum=370739, expTime= 0.0s 22:12:09 APOGEE expNum=49320001, reads=3, dithPos=B, Dark 22:15:02 APOGEE expNum=49320002, reads=60, dithPos=B, Dark **************************************************************** 22:15:25 Field-Design-Configuration: -999--999-15068 **************************************************************** 22:15:59 CmdStarted: FVC loop 22:16:28 CmdDone: FVC loop FVC loop results: fvc_rms= 40.8 fvc_deltarms= -17.54 fvc_filename=/data/fcam/60494/fimg-fvc1n-0001.fits fvc_perc_90= 61.30 fvc_percent_reached= 1.00 fvc iterations= 1 22:21:14 CmdStarted: jaeger configuration reverse 22:22:06 CmdDone: jaeger configuration reverse 22:25:54 APOGEE expNum=49320003, reads=10, dithPos=B, QuartzFlat 22:27:46 APOGEE expNum=49320004, reads=12, dithPos=B, ArcLamp 22:28:38 BOSS expNum=370740, expTime= 4.0s 22:29:18 BOSS expNum=370741, expTime= 4.0s 22:29:59 APOGEE expNum=49320005, reads=40, dithPos=B, ArcLamp 22:30:01 sp1.motor.move= 349, 350, 349 22:30:01 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= -3.2 (In Focus) sp1: pred. move: spAverageMove= 349 sp1: pred. resid: r= -678, b= -2.1, txt= OK, spTemp = 10.7 22:30:23 BOSS expNum=370742, expTime= 4.0s 22:31:03 BOSS expNum=370743, expTime= 4.0s 22:31:45 sp1.motor.move= -137, -137, -137 22:31:45 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -1.2 (In Focus) sp1: pred. move: spAverageMove= -137 sp1: pred. resid: r= -522, b= -1.6, txt= OK, spTemp = 10.7 22:32:45 BOSS expNum=370744, expTime= 4.0s 22:34:45 BOSS expNum=370745, expTime= 25.0s 22:37:16 APOGEE expNum=49320006, reads=12, dithPos=A, ArcLamp 22:38:28 CmdStarted: tcc track (slew begins) 22:38:32 CmdDone: Slew finished ('tcc track' command finished) 22:39:29 APOGEE expNum=49320007, reads=40, dithPos=A, ArcLamp 22:46:54 APOGEE expNum=49320008, reads=60, dithPos=A, Dark 22:57:45 APOGEE expNum=49320009, reads=30, dithPos=A, ArcLamp 23:03:09 APOGEE expNum=49320010, reads=30, dithPos=B, ArcLamp 02:52:27 APOGEE expNum=49320011, reads=3, dithPos=B, Dark **************************************************************** 03:03:48 Field-Design-Configuration: 101171-615281-15069 **************************************************************** 03:03:54 hal goto-field: reconfigure, fvc 03:03:57 CmdStarted: jaeger configuration execute 03:05:02 CmdDone: jaeger configuration execute 03:05:03 CmdStarted: FVC loop 03:06:09 CmdDone: FVC loop FVC loop results: fvc_rms= 10.29 fvc_deltarms= 43.47 fvc_filename=/data/fcam/60494/fimg-fvc1n-0004.fits fvc_perc_90= 11.90 fvc_percent_reached= 46.90 fvc iterations= 2 **************************************************************** 03:10:43 Field-Design-Configuration: 100476-611631-15070 **************************************************************** 03:10:50 hal goto-field: reconfigure, fvc 03:10:53 CmdStarted: jaeger configuration reverse 03:11:58 CmdDone: jaeger configuration reverse 03:11:58 CmdStarted: jaeger configuration execute 03:12:45 CmdDone: jaeger configuration execute 03:12:46 CmdStarted: FVC loop 03:13:51 CmdDone: FVC loop FVC loop results: fvc_rms= 9.91 fvc_deltarms= 47.24 fvc_filename=/data/fcam/60494/fimg-fvc1n-0006.fits fvc_perc_90= 12.30 fvc_percent_reached= 38.20 fvc iterations= 2 **************************************************************** 03:15:44 Field-Design-Configuration: 103240-621638-15071 **************************************************************** 03:15:50 hal goto-field: reconfigure, fvc 03:15:52 CmdStarted: jaeger configuration reverse 03:16:40 CmdDone: jaeger configuration reverse 03:16:40 CmdStarted: jaeger configuration execute 03:17:41 CmdDone: jaeger configuration execute 03:17:42 CmdStarted: FVC loop 03:18:47 CmdDone: FVC loop FVC loop results: fvc_rms= 7.46 fvc_deltarms= 43.37 fvc_filename=/data/fcam/60494/fimg-fvc1n-0008.fits fvc_perc_90= 11.00 fvc_percent_reached= 50.90 fvc iterations= 2 **************************************************************** 03:41:35 Field-Design-Configuration: 112359-403338-15072 **************************************************************** 03:41:51 hal goto-field: reconfigure, fvc 03:41:54 CmdStarted: jaeger configuration reverse 03:42:56 CmdDone: jaeger configuration reverse 03:42:56 CmdStarted: jaeger configuration execute 03:43:59 CmdDone: jaeger configuration execute 03:44:00 CmdStarted: FVC loop 03:45:05 CmdDone: FVC loop FVC loop results: fvc_rms= 8.31 fvc_deltarms= 41.35 fvc_filename=/data/fcam/60494/fimg-fvc1n-0010.fits fvc_perc_90= 11.40 fvc_percent_reached= 45.20 fvc iterations= 2 **************************************************************** 03:45:10 Field-Design-Configuration: 112359-403339-15073 **************************************************************** **************************************************************** 03:49:41 Field-Design-Configuration: 103240-621638-15074 **************************************************************** 03:49:49 hal goto-field: reconfigure, fvc 03:49:52 CmdStarted: jaeger configuration reverse 03:50:55 CmdDone: jaeger configuration reverse 03:50:55 CmdStarted: jaeger configuration execute 03:51:46 CmdDone: jaeger configuration execute 03:51:46 CmdStarted: FVC loop 03:52:51 CmdDone: FVC loop FVC loop results: fvc_rms= 8.48 fvc_deltarms= 41.59 fvc_filename=/data/fcam/60494/fimg-fvc1n-0012.fits fvc_perc_90= 10.60 fvc_percent_reached= 48.90 fvc iterations= 2 Next preload attempt was hal popping-off 621638 again, which would be a clone. Lots of clouds and radar returns. Tonight is a wash. FPS is powered-off. Positioners are home/folded. APOGEE morning cals started. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.0050 0.0050 Axial B: -0.0010 -0.0010 Axial C: 0.0640 0.0630 Trans D: -0.0100 -0.0060 Lateral E: 0.3380 0.3380 Lateral F: 0.5455 0.5455 GALILS Commanded: 9800 -300 6750 19150 -3550 -3250 Actual: 9800 -300 6750 19150 -3550 -3250 SETMIR VALUES PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -1.0390 -1.0380 Axial B: 0.2190 0.2210 Axial C: -1.1100 -1.1090 Trans D: -0.0040 -0.0050 GALILS Commanded: 5887200 5813350 5948200 -1500 -7250 Actual: 5883662 5811782 5906169 -1500 -7250 SETMIR VALUES SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 SecOrient: 4657.01 -24.48 -17.49 17.7 125.24 ---- Telescope Status ---- Status at: 09:00Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 693 695 712 712 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 255 lbs, 22 psi FPS Status ========== CP10 Loop = 7.55 L/min GFA Loop = 2.25 L/min Supply pressure = 42.5 psi Return pressure = 29.0 psi Chiller Status ============== Set temperarature = 15.1 C Flow = 5.1 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.5.1a0 jaeger : 1.7.5 kaiju : 1.4.0 coordio : 1.11.2 fps_calibrati : 2024.04.01+91998f8 cherno : 1.0.0 astrometrynet : 0.93 hal : 1.3.5 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1-1-ge461 -mcp.plc : 35 -mcp.azFiduci : v1_149 -mcp.altFiduc : v1_149 -mcp.rotFiduc : v1_149 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev STUIAdditions : 3.1.18 boson : 0.4.9 sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 PLC : V35 Fiducials : v1_149 idlspec2d : v6_1_3 Kronos : 1.7.0 roboscheduler : 1.4.0 ---- APOGEE Data Summary ---- 60493 morning cals: num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- 49310011 -999 -999 15066 DARK 13.5 60 -27 True -- -- -- -- -- -- 49310012 -999 -999 15066 DARK 13.5 60 -10 True -- -- -- -- -- -- 49310013 -999 -999 15066 DARK 13.5 60 -13 True -- -- -- -- -- -- 49310014 -999 -999 15066 QUARTZFLAT 13.5 10 108 True -- -0.18 -- 0.86 -- -- 49310015 -999 -999 15066 QUARTZFLAT 13.5 10 109 True -- -0.18 -- 0.87 -- -- 49310016 -999 -999 15066 QUARTZFLAT 13.5 10 110 True -- -0.17 -- 0.87 -- -- 49310017 -999 -999 15066 ARCLAMP-THAR 13.5 12 102 True 1.04 0.62 1.08 0.90 -- -- 49310018 -999 -999 15066 ARCLAMP-UNE 13.5 40 93 True 0.63 0.47 1.11 0.89 -- -- 49310019 -999 -999 15066 ARCLAMP-THAR 13.0 12 92 True 0.53 0.68 1.06 0.88 -- -- 49310020 -999 -999 15066 ARCLAMP-UNE 13.0 40 88 True 0.16 0.44 1.06 0.89 -- -- 49310021 -999 -999 15066 DARK 13.0 30 23 False -- -- -- -- -- -- 49310022 -999 -999 15066 INTERNALFLAT 13.0 30 99 True -- -- -- -- -- -- 49310023 -999 -999 15066 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 49310024 -999 -999 15066 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 49310025 -999 -999 15066 DARK 13.0 30 260 True -- -- -- -- -- -- 49310026 -999 -999 15066 FPI 13.0 30 130 True -0.31 0.43 1.17 0.88 -- -- 49310027 -999 -999 15066 FPI 13.5 30 131 True -12.14 0.39 1.18 0.88 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 49310014 QUARTZFLAT 10 68 74 83 92 93 88 89 91 91 83 49310015 QUARTZFLAT 10 71 78 84 93 94 89 89 91 91 85 49310016 QUARTZFLAT 10 72 79 85 94 95 89 90 92 92 86 (observers39) [observer@sdss5-hub: ~]$ check_ap 60494 num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ---- ------- ------- ------ ------ ---- ---- 49320001 101171 615281 15067 DARK 13.5 3 133 True -- -- -- -- -- -- 49320002 101171 615281 15067 DARK 13.5 60 -25 True -- -- -- -- -- -- 49320003 -999 -999 15068 QUARTZFLAT 13.5 10 107 True -- -0.27 -- 0.87 -- -- 49320004 -999 -999 15068 ARCLAMP-THAR 13.5 12 99 True 1.06 0.55 1.06 0.89 -- -- 49320005 -999 -999 15068 ARCLAMP-UNE 13.5 40 97 True 0.65 0.38 1.10 0.87 -- -- 49320006 -999 -999 15068 ARCLAMP-THAR 13.0 12 92 True 0.53 0.59 1.05 0.87 -- -- 49320007 -999 -999 15068 ARCLAMP-UNE 13.0 40 95 True 0.16 0.35 1.11 0.90 -- -- 49320008 -999 -999 15068 DARK 13.0 60 -5 True -- -- -- -- -- -- 49320009 -999 -999 15068 FPI 13.0 30 135 True 11.99 0.32 1.19 0.89 -- -- 49320010 -999 -999 15068 FPI 13.5 30 135 True -12.12 0.32 1.18 0.87 -- -- 49320011 -999 -999 15068 DARK 13.5 3 100 True -- -- -- -- -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 49320003 QUARTZFLAT 10 68 73 83 92 93 88 90 91 91 83 ---- BOSS Spectrographs Collimation ---- Afternoon focus check ===================== 22:30:24 BOSS expNum=370742, expTime= 4.0s 22:31:04 BOSS expNum=370743, expTime= 4.0s 22:31:45 sp1.motor.move= -137, -137, -137 22:31:45 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -1.2 (In Focus) sp1: pred. move: spAverageMove= -137 sp1: pred. resid: r= -522, b= -1.6, txt= OK, spTemp = 10.7 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60494 Fill night time in Hours below *=*=*= Total bright time : 1.0 *=*=*= Bright time, queued observations : 0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 1.0 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 7.0 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 7.0 *=*=*= Dark time lost to technical issues : 0