Subject: 2.5m obslog 20250220 (SJD 60727) From: cscoresby@apo.nmsu.edu Date: Fri, 21 Feb 2025 13:03:08 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Thursday February 20, 2025 (SJD 60727) ---- Observing Team ---- Conner Scoresby ---- Observing Plan ---- Thu 20 Feb 2025 60727 0.45 01:47 02:01 12:31 05:03 15:35 --- dark until 08:43 ( 6.7 h), then bright ( 4.1 h, w/ 0.3 h for 12 deg design) ---- Observing Summary ---- First half was lost to cloud cover. Opened in the second half and observed five designs. However the HgCd and Ne lamps stopped working at ~10:00Z. After some troubleshooting, decided I was unable to resolve the issue myself, and being toward the end of the night anyway, we closed at 12:00Z. ATTN Day Engineers: HgCd and Ne lamps unresponsive. A power module fault in the lamp power supply rack is likely the the culprit; see Engineering Issues for more details. Notes: 1. Dust collected ~9000 dust-hours 2. Configurations 19469--19472 inclusive have no arcs. ---- Automatic Summary ---- Time Field-Design-Conf DesignType BOSS/APOGEE Seeing Overhead Complete? Comment 08:03:06 101228-713004-19463 dark_monit_no_apogee_skies 1x900/1xAB 1.4" 0:00 Epoch Not Done 08:44:37 101310-713031-19465 bright_time_no_apogee_skies 1x730/1xAB 1.6" 13:30 Done 09:04:11 101664-713602-19466 bright_time_no_apogee_skies 1x730/1xAB 1.7" 7:39 Done 09:24:06 101373-713077-19467 bright_time_no_apogee_skies 1x730/1xAB 2.2" 7:41 Done 09:43:58 102570-714392-19468 bright_time_no_apogee_skies 1x730/1xAB 2.1" 7:53 Done 10:04:03 102556-714385-19469 bright_time_no_apogee_skies 1x730/1xAB 2.0" 7:01 * 10:22:59 102317-714010-19470 bright_time_no_apogee_skies 1x730/1xAB 1.5" 6:15 * 10:41:45 103171-714776-19471 bright_time_no_apogee_skies 1x730/1xAB 2.1" 7:25 * 11:01:05 103181-714788-19472 bright_time 1x730/1xAB 2.1" 5:49 * *Configurations with no arc files taken and not reduced, though signal on science object was good. ---- Engineering Issues ---- Spotted the probable culprit in the lamp power supply rack closest to the stairs. APC power supply beeps and shows fault with message 'Power Module Fault - Main Frame Module:L4 - Level 4 from top' ---- HAL / STUI stopped sending ne_on commands correctly around 10:00Z. HAL throws an error upon beginning Expose: "Some lamps are on." Seeing this, and seeing the lamps checked in STUI MCP, I unchecked and proceeded with the exposures. MCP GUI shows the lamps are on. Several configurations later I saw on SOS that the arcs were showing no arc lines and couldn't be reduced. On a test configuration, STUI MCP says 'hgcd lamps' are checked but displays 'All Off'. MCP SDSS 2.5M Status Monitor says the lamps are on. Likewise for 'ne lamps'. Verified that flat field lamps still work. Verified that hgcd and ne lamps do not turn on with either semaphore. Verified that supposed arc images appear as darks. Configurations 19469--19472 inclusive have no arcs. ---- Observing Log ---- Disk space check: /dev/sdb1 2.0T 456G 1.6T 23% /home /dev/sdd1 22T 13T 8.8T 58% /data 22:47:10 BOSS expNum=381905, expTime= 0.0s 22:48:43 BOSS expNum=381906, expTime= 20.0s 22:50:56 BOSS expNum=381907, expTime= 4.0s 22:51:36 BOSS expNum=381908, expTime= 4.0s 22:52:18 sp1.motor.move= 510, 509, 510 22:52:18 Hartmann collimate output sp1: offset: r= -329 (In Focus); b= -4.2 (Out of focus) sp1: pred. move: spAverageMove= 510 sp1: pred. resid: r= -839, b= -2.6, txt= OK, spTemp = 4.1 22:52:43 BOSS expNum=381909, expTime= 4.0s 22:54:28 BOSS expNum=381910, expTime= 25.0s 22:57Z FPS powered on, robots are folded. **************************************************************** 22:57:43 Field-Design-Configuration: -999--999-19462 **************************************************************** 22:58:07 CmdStarted: FVC loop 22:58:34 CmdDone: FVC loop FVC loop results: fvc_rms= 73.21 fvc_deltarms= 23.15 fvc_filename=/data/fcam/60727/fimg-fvc1n-0001.fits fvc_perc_90= 94.20 fvc_percent_reached= 1.40 fvc iterations= 1 -- Confirmed no bright spots 00:10Z Ventilation delayed due to some radar and cloud cover. Finished APOGEE evening cals: ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 60727 1 22:41 51650001 Dark 3 B 13.5 ---------- a-b-c - - 60727 2 22:52 51650002 Dark 60 B 13.5 ---------- a-b-c - - 60727 3 22:54 51650003 QuarFlat 10 B 13.5 ---------- a-b-c - - 60727 4 22:56 51650004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.97 - 60727 5 23:03 51650005 Arc-Une 40 B 13.5 ---------- a-b-c - - 60727 6 23:06 51650006 Arc-Thar 12 A 13.0 ---------- a-b-c 3.41 - 60727 7 23:13 51650007 Arc-Une 40 A 13.0 ---------- a-b-c - - 60727 8 23:24 51650008 Dark 60 A 13.0 ---------- a-b-c - - 60727 9 23:29 51650009 Arc---- 30 A 13.0 ---------- a-b-c - 72 FPI A 60727 10 23:35 51650010 Arc---- 30 B 13.5 ---------- a-b-c - 73 FPI B ------------------------------------------------------------------------------ 07:15Z Ventilation started. Forced a fill. 07:47Z Opened to partly cloudy skies. 07:51:15 hal apogee-dome-flat: FFS, shutter, expose 07:51:42 APOGEE expNum=51650011, reads=5, dithPos=B, DomeFlat 07:52:50 hal apogee-dome-flat: FFS, shutter, expose 07:53:16 APOGEE expNum=51650012, reads=5, dithPos=B, DomeFlat 07:54:09 hal apogee-dome-flat: FFS, shutter, expose 07:54:36 APOGEE expNum=51650013, reads=5, dithPos=B, DomeFlat Globally Normalized MJD DomeFlat MTPs:1 2 3 4 5 6 7 8 9 10 AZ ALT ROT 60727 51650011 0.98 1.02 1.03 1.02 1.03 1.00 0.98 0.99 0.99 0.94 60.0 60.0 60.0 60727 51650012 1.00 1.03 1.03 1.01 1.03 1.00 0.97 0.99 0.99 0.94 60.0 60.0 60.0 60727 51650013 1.01 1.03 1.02 1.01 1.03 1.00 0.97 0.99 0.99 0.94 60.0 60.0 60.0 07:56:38 BOSS expNum=381911, expTime= 4.0s 07:57:18 BOSS expNum=381912, expTime= 4.0s 07:58:09 sp1.motor.move= 2953, 2953, 2952 07:58:09 Hartmann collimate output sp1: offset: r= 2308 (Out of focus); b= -11.3 (Out of focus) sp1: pred. move: spAverageMove= 2953 sp1: pred. resid: r= -645, b= -2.0, txt= OK, spTemp = 3.3 07:58:45 BOSS expNum=381913, expTime= 25.0s 08:00:34 BOSS expNum=381914, expTime= 25.0s 08:02:24 BOSS expNum=381915, expTime= 25.0s **************************************************************** 08:03:07 Field-Design-Configuration: 101228-713004-19463 **************************************************************** 08:03:14 hal goto-field: slew, reconfigure, fvc, reslew, flat, arc, acquire, guide 08:03:17 HgCd lamps turned OFF 08:03:17 Ne lamps turned OFF 08:03:18 CmdStarted: jaeger configuration reverse 08:03:22 CmdStarted: tcc track (slew begins) 08:04:15 CmdDone: jaeger configuration reverse 08:04:15 CmdStarted: jaeger configuration execute 08:04:49 CmdDone: Slew finished ('tcc track' command finished) 08:05:04 CmdDone: jaeger configuration execute 08:05:05 CmdStarted: FVC loop 08:06:09 CmdDone: FVC loop FVC loop results: fvc_rms= 10.6 fvc_deltarms= 63.37 fvc_filename=/data/fcam/60727/fimg-fvc1n-0004.fits fvc_perc_90= 11.80 fvc_percent_reached= 46.70 fvc iterations= 2 08:06:11 CmdStarted: tcc track (slew begins) 08:06:19 CmdDone: Slew finished ('tcc track' command finished) 08:06:38 BOSS expNum=381916, expTime= 25.0s 08:07:08 HgCd lamps turned ON 08:07:08 Ne lamps turned ON 08:09:14 BOSS expNum=381917, expTime= 4.0s 08:09:24 HgCd lamps turned OFF 08:09:24 Ne lamps turned OFF 08:09:34 CmdStarted: cherno acquire 08:12:01 CmdDone: cherno acquire 08:12:03 CmdStarted: 'cherno guide' 08:12:21 hal goto-field: slew, acquire, guide 08:12:28 CmdStarted: tcc track (slew begins) 08:12:30 CmdDone: Slew finished ('tcc track' command finished) 08:12:44 CmdStarted: tcc track (slew begins) 08:12:51 CmdDone: Slew finished ('tcc track' command finished) 08:12:58 hal goto-field: acquire, guide 08:22:40 hal expose: BOSS science, APOGEE science 08:23:06 APOGEE expNum=51650014, reads=43, dithPos=B, Object 08:23:12 BOSS expNum=381918, expTime= 900.0s 08:30:52 APOGEE expNum=51650015, reads=43, dithPos=A, Object **************************************************************** 08:38:30 Field-Design-Configuration: 101462-713199-19464 **************************************************************** 08:38:37 hal goto-field: slew, reconfigure, fvc, reslew, flat, arc, acquire, guide 08:38:41 CmdStarted: jaeger configuration reverse 08:38:45 CmdStarted: tcc track (slew begins) 08:39:16 CmdDone: Slew finished ('tcc track' command finished) 08:39:31 CmdDone: jaeger configuration reverse 08:39:31 CmdStarted: jaeger configuration execute 08:40:17 CmdDone: jaeger configuration execute 08:40:18 CmdStarted: FVC loop 08:41:22 CmdDone: FVC loop FVC loop results: fvc_rms= 11.52 fvc_deltarms= 53.95 fvc_filename=/data/fcam/60727/fimg-fvc1n-0006.fits fvc_perc_90= 11.40 fvc_percent_reached= 45.30 fvc iterations= 2 08:41:24 CmdStarted: tcc track (slew begins) 08:41:34 CmdDone: Slew finished ('tcc track' command finished) 08:41:53 BOSS expNum=381919, expTime= 25.0s 08:42:15 HgCd lamps turned OFF 08:42:15 Ne lamps turned OFF **************************************************************** 08:44:37 Field-Design-Configuration: 101310-713031-19465 **************************************************************** 08:44:45 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 08:44:50 CmdStarted: jaeger configuration reverse 08:44:53 CmdStarted: tcc track (slew begins) 08:44:57 HgCd lamps turned ON 08:45:36 CmdDone: jaeger configuration reverse 08:45:36 CmdStarted: jaeger configuration execute 08:46:21 CmdDone: jaeger configuration execute 08:47:38 CmdDone: Slew finished ('tcc track' command finished) 08:47:39 CmdStarted: FVC loop 08:48:43 CmdDone: FVC loop FVC loop results: fvc_rms= 13.9 fvc_deltarms= 55.39 fvc_filename=/data/fcam/60727/fimg-fvc1n-0008.fits fvc_perc_90= 10.60 fvc_percent_reached= 53.10 fvc iterations= 2 08:48:44 CmdStarted: tcc track (slew begins) 08:48:45 Ne lamps turned ON 08:48:51 CmdDone: Slew finished ('tcc track' command finished) 08:49:32 BOSS expNum=381920, expTime= 4.0s 08:49:41 HgCd lamps turned OFF 08:49:41 Ne lamps turned OFF 08:49:50 CmdStarted: cherno acquire 08:51:07 CmdDone: cherno acquire 08:51:09 CmdStarted: 'cherno guide' 08:51:11 New-field overhead: 0:12:42 08:51:11 hal expose: BOSS science, APOGEE science 08:51:37 APOGEE expNum=51650016, reads=35, dithPos=A, Object 08:51:42 BOSS expNum=381921, expTime= 730.0s 08:57:53 APOGEE expNum=51650017, reads=35, dithPos=B, Object **************************************************************** 09:04:11 Field-Design-Configuration: 101664-713602-19466 **************************************************************** 09:04:20 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 09:04:20 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 09:04:24 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 09:04:28 CmdStarted: jaeger configuration reverse 09:04:32 CmdStarted: tcc track (slew begins) 09:04:36 HgCd lamps turned ON 09:05:13 CmdDone: jaeger configuration reverse 09:05:13 CmdStarted: jaeger configuration execute 09:06:03 CmdDone: jaeger configuration execute 09:07:33 CmdDone: Slew finished ('tcc track' command finished) 09:07:34 CmdStarted: FVC loop 09:08:38 CmdDone: FVC loop FVC loop results: fvc_rms= 9.6 fvc_deltarms= 57.86 fvc_filename=/data/fcam/60727/fimg-fvc1n-0010.fits fvc_perc_90= 11.20 fvc_percent_reached= 48.60 fvc iterations= 2 09:08:40 CmdStarted: tcc track (slew begins) 09:08:41 Ne lamps turned ON 09:08:46 CmdDone: Slew finished ('tcc track' command finished) 09:09:24 BOSS expNum=381922, expTime= 4.0s 09:09:33 HgCd lamps turned OFF 09:09:33 Ne lamps turned OFF 09:09:42 CmdStarted: cherno acquire 09:10:57 CmdDone: cherno acquire 09:10:59 CmdStarted: 'cherno guide' 09:11:00 hal expose: BOSS science, APOGEE science 09:11:32 BOSS expNum=381923, expTime= 730.0s 09:11:34 APOGEE expNum=51650018, reads=35, dithPos=A, Object 09:17:51 APOGEE expNum=51650019, reads=35, dithPos=B, Object 09:24:03 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 09:24:07 Field-Design-Configuration: 101373-713077-19467 **************************************************************** 09:24:13 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 09:24:20 CmdStarted: jaeger configuration reverse 09:24:23 CmdStarted: tcc track (slew begins) 09:24:26 HgCd lamps turned ON 09:25:09 CmdDone: jaeger configuration reverse 09:25:09 CmdStarted: jaeger configuration execute 09:25:56 CmdDone: jaeger configuration execute 09:27:25 CmdDone: Slew finished ('tcc track' command finished) 09:27:26 CmdStarted: FVC loop 09:28:30 CmdDone: FVC loop FVC loop results: fvc_rms= 56.65 fvc_deltarms= 8.79 fvc_filename=/data/fcam/60727/fimg-fvc1n-0012.fits fvc_perc_90= 11.70 fvc_percent_reached= 48.00 fvc iterations= 2 09:28:32 CmdStarted: tcc track (slew begins) 09:28:33 Ne lamps turned ON 09:28:38 CmdDone: Slew finished ('tcc track' command finished) 09:29:15 BOSS expNum=381924, expTime= 4.0s 09:29:23 HgCd lamps turned OFF 09:29:23 Ne lamps turned OFF 09:29:33 CmdStarted: cherno acquire 09:30:48 CmdDone: cherno acquire 09:30:50 CmdStarted: 'cherno guide' 09:30:52 hal expose: BOSS science, APOGEE science 09:31:24 BOSS expNum=381925, expTime= 730.0s 09:31:26 APOGEE expNum=51650020, reads=35, dithPos=A, Object 09:37:42 APOGEE expNum=51650021, reads=35, dithPos=B, Object 09:43:54 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 09:43:58 Field-Design-Configuration: 102570-714392-19468 **************************************************************** 09:44:05 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 09:44:11 CmdStarted: jaeger configuration reverse 09:44:15 CmdStarted: tcc track (slew begins) 09:44:17 HgCd lamps turned ON 09:44:58 CmdDone: jaeger configuration reverse 09:44:58 CmdStarted: jaeger configuration execute 09:45:56 CmdDone: jaeger configuration execute 09:47:14 CmdDone: Slew finished ('tcc track' command finished) 09:47:15 CmdStarted: FVC loop 09:48:20 CmdDone: FVC loop FVC loop results: fvc_rms= 7.97 fvc_deltarms= 56.49 fvc_filename=/data/fcam/60727/fimg-fvc1n-0014.fits fvc_perc_90= 10.60 fvc_percent_reached= 53.50 fvc iterations= 2 09:48:21 CmdStarted: tcc track (slew begins) 09:48:22 Ne lamps turned ON 09:48:28 CmdDone: Slew finished ('tcc track' command finished) 09:49:09 BOSS expNum=381926, expTime= 4.0s 09:49:23 HgCd lamps turned OFF 09:49:23 Ne lamps turned OFF 09:49:32 CmdStarted: cherno acquire 09:50:54 CmdDone: cherno acquire 09:50:56 CmdStarted: 'cherno guide' 09:50:57 hal expose: BOSS science, APOGEE science 09:51:28 BOSS expNum=381927, expTime= 730.0s 09:51:32 APOGEE expNum=51650022, reads=35, dithPos=A, Object 09:57:47 APOGEE expNum=51650023, reads=35, dithPos=B, Object 10:03:59 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 10:04:03 Field-Design-Configuration: 102556-714385-19469 **************************************************************** 10:04:09 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:04:16 CmdStarted: jaeger configuration reverse 10:04:20 CmdStarted: tcc track (slew begins) 10:04:23 HgCd lamps turned ON 10:04:45 CmdDone: Slew finished ('tcc track' command finished) 10:05:11 CmdDone: jaeger configuration reverse 10:05:11 CmdStarted: jaeger configuration execute 10:05:58 CmdDone: jaeger configuration execute 10:05:59 CmdStarted: FVC loop 10:07:04 CmdDone: FVC loop FVC loop results: fvc_rms= 93.06 fvc_deltarms= -28.0 fvc_filename=/data/fcam/60727/fimg-fvc1n-0016.fits fvc_perc_90= 11.90 fvc_percent_reached= 43.00 fvc iterations= 2 10:07:05 CmdStarted: tcc track (slew begins) 10:07:06 Ne lamps turned ON 10:07:11 CmdDone: Slew finished ('tcc track' command finished) 10:07:45 BOSS expNum=381928, expTime= 4.0s 10:08:03 CmdStarted: cherno acquire 10:09:18 CmdDone: cherno acquire 10:09:19 CmdStarted: 'cherno guide' 10:09:21 hal expose: BOSS science, APOGEE science 10:09:55 New-field overhead: 1:05:48 10:09:55 hal expose: BOSS science, APOGEE science 10:10:04 HgCd lamps turned OFF 10:10:06 Ne lamps turned OFF 10:10:09 hal expose: BOSS science, APOGEE science 10:10:35 APOGEE expNum=51650024, reads=35, dithPos=B, Object 10:10:40 BOSS expNum=381929, expTime= 730.0s 10:16:56 APOGEE expNum=51650025, reads=35, dithPos=A, Object 10:20:41 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 10:20:42 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 10:23:00 Field-Design-Configuration: 102317-714010-19470 **************************************************************** 10:23:06 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:23:11 CmdStarted: jaeger configuration reverse 10:23:14 CmdStarted: tcc track (slew begins) 10:23:18 HgCd lamps turned ON 10:23:49 CmdDone: Slew finished ('tcc track' command finished) 10:23:58 CmdDone: jaeger configuration reverse 10:23:58 CmdStarted: jaeger configuration execute 10:24:46 CmdDone: jaeger configuration execute 10:24:46 CmdStarted: FVC loop 10:25:51 CmdDone: FVC loop FVC loop results: fvc_rms= 15.67 fvc_deltarms= 60.38 fvc_filename=/data/fcam/60727/fimg-fvc1n-0018.fits fvc_perc_90= 13.70 fvc_percent_reached= 46.30 fvc iterations= 2 10:25:53 CmdStarted: tcc track (slew begins) 10:25:54 Ne lamps turned ON 10:26:00 CmdDone: Slew finished ('tcc track' command finished) 10:26:33 BOSS expNum=381930, expTime= 4.0s 10:26:50 CmdStarted: cherno acquire 10:28:07 CmdDone: cherno acquire 10:28:08 CmdStarted: 'cherno guide' 10:28:10 hal expose: BOSS science, APOGEE science 10:28:26 Ne lamps turned OFF 10:28:27 HgCd lamps turned OFF 10:28:34 New-field overhead: 0:05:26 10:28:34 hal expose: BOSS science, APOGEE science 10:29:01 APOGEE expNum=51650026, reads=35, dithPos=A, Object 10:29:06 BOSS expNum=381931, expTime= 730.0s 10:35:17 APOGEE expNum=51650027, reads=35, dithPos=B, Object **************************************************************** 10:41:45 Field-Design-Configuration: 103171-714776-19471 **************************************************************** 10:41:53 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:41:57 CmdStarted: jaeger configuration reverse 10:42:01 CmdStarted: tcc track (slew begins) 10:42:04 HgCd lamps turned ON 10:42:41 CmdDone: Slew finished ('tcc track' command finished) 10:42:45 CmdDone: jaeger configuration reverse 10:42:45 CmdStarted: jaeger configuration execute 10:43:34 CmdDone: jaeger configuration execute 10:43:34 CmdStarted: FVC loop 10:44:39 CmdDone: FVC loop FVC loop results: fvc_rms= 60.72 fvc_deltarms= 7.05 fvc_filename=/data/fcam/60727/fimg-fvc1n-0020.fits fvc_perc_90= 12.20 fvc_percent_reached= 51.20 fvc iterations= 2 10:44:41 CmdStarted: tcc track (slew begins) 10:44:42 Ne lamps turned ON 10:44:49 CmdDone: Slew finished ('tcc track' command finished) 10:45:24 BOSS expNum=381932, expTime= 4.0s 10:45:42 CmdStarted: cherno acquire 10:46:57 CmdDone: cherno acquire 10:46:59 CmdStarted: 'cherno guide' 10:48:01 New-field overhead: 0:06:29 10:48:01 hal expose: BOSS science, APOGEE science 10:48:07 Ne lamps turned OFF 10:48:07 HgCd lamps turned OFF 10:48:11 hal expose: BOSS science, APOGEE science 10:48:37 APOGEE expNum=51650028, reads=35, dithPos=B, Object 10:48:43 BOSS expNum=381933, expTime= 730.0s 10:54:58 APOGEE expNum=51650029, reads=35, dithPos=A, Object 11:00:02 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 11:00:02 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 11:01:05 Field-Design-Configuration: 103181-714788-19472 **************************************************************** 11:01:12 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 11:01:16 CmdStarted: jaeger configuration reverse 11:01:20 CmdStarted: tcc track (slew begins) 11:01:23 HgCd lamps turned ON 11:01:37 CmdDone: Slew finished ('tcc track' command finished) 11:02:05 CmdDone: jaeger configuration reverse 11:02:05 CmdStarted: jaeger configuration execute 11:02:53 CmdDone: jaeger configuration execute 11:02:53 CmdStarted: FVC loop 11:03:58 CmdDone: FVC loop FVC loop results: fvc_rms= 22.14 fvc_deltarms= 46.5 fvc_filename=/data/fcam/60727/fimg-fvc1n-0022.fits fvc_perc_90= 12.40 fvc_percent_reached= 50.00 fvc iterations= 2 11:03:59 CmdStarted: tcc track (slew begins) 11:04:00 Ne lamps turned ON 11:04:07 CmdDone: Slew finished ('tcc track' command finished) 11:04:39 BOSS expNum=381934, expTime= 4.0s 11:04:57 CmdStarted: cherno acquire 11:05:55 CmdDone: cherno acquire 11:05:56 CmdStarted: 'cherno guide' 11:05:58 hal expose: BOSS science, APOGEE science 11:06:06 Ne lamps turned OFF 11:06:07 HgCd lamps turned OFF 11:06:11 New-field overhead: 0:05:01 11:06:11 hal expose: BOSS science, APOGEE science 11:06:36 APOGEE expNum=51650030, reads=35, dithPos=A, Object 11:06:42 BOSS expNum=381935, expTime= 730.0s 11:12:52 APOGEE expNum=51650031, reads=35, dithPos=B, Object 11:20Z Noticed the lack of good arcs for the last few configurations. Going to watch a configurations to see what happens: *************************************************************** 11:21:05 Field-Design-Configuration: 103175-714782-19473 **************************************************************** 11:21:14 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 11:21:19 CmdStarted: jaeger configuration reverse 11:21:22 CmdStarted: tcc track (slew begins) 11:21:26 HgCd lamps turned ON 11:21:38 CmdDone: Slew finished ('tcc track' command finished) 11:22:04 CmdDone: jaeger configuration reverse 11:22:04 CmdStarted: jaeger configuration execute 11:22:55 CmdDone: jaeger configuration execute 11:22:56 CmdStarted: FVC loop 11:24:00 CmdDone: FVC loop FVC loop results: fvc_rms= 20.58 fvc_deltarms= 58.76 fvc_filename=/data/fcam/60727/fimg-fvc1n-0024.fits fvc_perc_90= 11.40 fvc_percent_reached= 50.30 fvc iterations= 2 11:24:01 CmdStarted: tcc track (slew begins) 11:24:02 Ne lamps turned ON 11:24:09 CmdDone: Slew finished ('tcc track' command finished) 11:24:41 BOSS expNum=381936, expTime= 4.0s 11:25:00 CmdStarted: cherno acquire 11:25:55 CmdDone: cherno acquire 11:25:57 CmdStarted: 'cherno guide' -- HgCd lamps are not turning on at all; 'arc' image came in looks like a dark. FF lamps still work. 12:00Z With limited time until morning, decided to call it and close. 12:01Z FPS folded and powered off. 12:05:17 hal apogee-dome-flat: FFS, shutter, expose 12:05:44 APOGEE expNum=51650032, reads=5, dithPos=B, DomeFlat 12:06:37 hal apogee-dome-flat: FFS, shutter, expose 12:07:03 APOGEE expNum=51650033, reads=5, dithPos=B, DomeFlat 12:07:56 hal apogee-dome-flat: FFS, shutter, expose 12:08:23 APOGEE expNum=51650034, reads=5, dithPos=B, DomeFlat 60727 51650032 0.99 1.02 1.03 1.02 1.04 1.00 0.97 0.99 0.98 0.96 121.0 30.0 0.0 60727 51650033 1.01 1.03 1.03 1.02 1.04 1.00 0.97 0.99 0.98 0.96 121.0 30.0 0.0 60727 51650034 1.02 1.03 1.03 1.02 1.03 1.00 0.97 0.99 0.99 0.95 121.0 30.0 0.0 12:09:35 BOSS expNum=381937, expTime= 25.0s -- No end-of-night arc 12:17Z Enclosure closed. Ventilation halted. 12:30Z Lamps confirmed still not responding 13:35Z Noticed the likely culprit, a power supply on the lamp rack in the lower enclosure beeps with message 'Power Module Fault - Main Frame Module:L4 - Level 4 from top' 12:48Z Started APOGEE morning cals. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: -0.0020 0.0040 Axial B: -0.0008 -0.0010 Axial C: -0.0060 -0.0060 Trans D: -0.0110 -0.0100 Lateral E: 0.3380 0.3380 Lateral F: 0.5450 0.5455 GALILS Commanded: 9800 -300 6750 19150 -3550 -3250 Actual: 9800 -300 6750 19150 -3550 -3250 SETMIR VALUES PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -0.1040 -0.0970 Axial B: -0.0680 -0.0650 Axial C: -0.0430 -0.0420 Trans D: -0.0040 -0.0030 GALILS Commanded: 5893000 5837700 5965350 -1500 -7250 Actual: 5883592 5811693 5906213 -1500 -7250 SETMIR VALUES SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 SecOrient: 4656.98 -24.49 -17.49 17.7 125.25 ---- Telescope Offsets and Scale ---- Time Field-Design Az Alt Rot RA Off Dec Off Rot Off Guide RMS (um) ================================================================================ 08:38:17 nan-nan -39.2 67.3 191.3 -0.003 -0.001 -0.269 0.197 08:57:37 101310-713031 118.5 52.1 -78.0 0.001 0.002 -0.055 0.235 09:15:00 101664-713602 82.5 61.4 259.7 0.002 0.004 -0.107 0.251 09:31:14 101373-713077 122.4 34.5 -78.2 0.002 -0.001 0.009 0.632 09:51:19 102570-714392 69.5 35.8 256.0 0.001 -0.002 -0.075 0.387 10:10:23 102556-714385 71.3 31.1 257.5 0.001 -0.003 -0.067 0.319 10:28:48 102317-714010 55.4 53.3 239.6 0.004 -0.001 -0.090 0.348 10:45:00 103171-714776 54.9 30.2 257.7 0.004 -0.000 -0.091 0.349 11:00:58 103171-714776 51.5 32.9 255.2 0.001 -0.002 -0.075 0.353 11:18:53 103181-714788 51.3 32.9 255.0 0.001 -0.002 -0.073 0.286 ---- Focus Log ---- Time Field-Design M1 M2 Focus Az Alt Temp Wind Dir ================================================================================ 08:38:17 nan-nan 0.00 4770.63 235 -39.2 67.3 0.0 13 229 08:57:37 101310-713031 0.00 4808.99 235 118.5 52.1 0.0 13 243 09:15:00 101664-713602 0.00 4770.57 222 82.5 61.4 0.0 10 257 09:31:14 101373-713077 0.00 4839.16 201 122.4 34.5 0.0 15 250 09:51:19 102570-714392 0.00 4746.31 114 69.5 35.8 0.0 13 245 10:10:23 102556-714385 0.00 4765.74 114 71.3 31.1 0.0 17 243 10:28:48 102317-714010 0.00 4662.21 92 55.4 53.3 0.0 14 258 10:45:00 103171-714776 0.00 4735.16 79 54.9 30.2 0.0 14 246 11:00:58 103171-714776 0.00 4694.15 50 51.5 32.9 0.0 17 244 11:18:53 103181-714788 0.00 4694.29 50 51.3 32.9 0.0 13 257 ---- Telescope Status ---- Opened at 07:47:39 Closed at 12:17:48 Status at: 2:46:34Z Telescope Position: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 653.6, 655.1, 652.5, 650.8 LN2 autofill systems: Connected and turned on 180L LN2 dewar scale: SP1 98.0 lbs, 22.8 psi FPS Status ========== CP10 Loop = 8.09 L/min GFA Loop = 2.74 L/min Supply pressure = 44.6 psi Return pressure = 29.2 psi Chiller Status ============== Set temperarature = 1.2 C Flow = 4.9 Gpm ---- Software Used ---- -- STUI software: STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.5.1a0 jaeger : 1.9.1a0 kaiju : 1.4.0 coordio : 1.13.0 fps_calibrati : 2024.10.16+97840f3 cherno : 1.0.1a0 astrometrynet : 0.93 hal : 1.3.1a0 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1-1-ge461 -mcp.plc : 35 -mcp.azFiduci : v1_150 -mcp.altFiduc : v1_150 -mcp.rotFiduc : v1_150 tcc : 1.4.0 apogee : not availble apogeecal : not availble apogeeql : v1_7_0dev STUIAdditions : FAILED Boson : 0.4.9 --------------Other Versions-------------- sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 idlspec2d : v6_2_0 Kronos : 2.0.0 roboscheduler : 2.1.0 ---- APOGEE Data Summary ---- MJD UTC Field-Design-Config Exposure Type Dith Reads Arch -------------------------------------------------------------------------------- 60726 22:40:53 100522-712171-19461 51650001 Dark B 3 a-b-c 60726 22:42:15 100522-712171-19461 51650002 Dark B 60 a-b-c 60726 22:53:06 100522-712171-19461 51650003 Quartzflat B 10 a-b-c 60726 22:54:59 100522-712171-19461 51650004 ThAr Arc B 12 a-b-c 60726 22:57:11 100522-712171-19461 51650005 UNe Arc B 40 a-b-c 60726 23:04:28 -999- -999-19462 51650006 ThAr Arc A 12 a-b-c 60726 23:06:40 -999- -999-19462 51650007 UNe Arc A 40 a-b-c 60726 23:14:06 -999- -999-19462 51650008 Dark A 60 a-b-c 60726 23:24:57 -999- -999-19462 51650009 FPI Lamp A 30 a-b-c 60726 23:30:21 -999- -999-19462 51650010 FPI Lamp B 30 a-b-c 60727 07:51:30 -999- -999-19462 51650011 Domeflat B 5 a-b-c 60727 07:53:04 -999- -999-19462 51650012 Domeflat B 5 a-b-c 60727 07:54:24 -999- -999-19462 51650013 Domeflat B 5 a-b-c 60727 08:22:54 101228-713004-19463 51650014 Object B 43 a-b-c 60727 08:30:40 101228-713004-19463 51650015 Object A 43 a-b-c 60727 08:51:24 101310-713031-19465 51650016 Object A 35 a-b-c 60727 08:57:41 101310-713031-19465 51650017 Object B 35 a-b-c 60727 09:11:22 101664-713602-19466 51650018 Object A 35 a-b-c 60727 09:17:38 101664-713602-19466 51650019 Object B 35 a-b-c 60727 09:31:14 101373-713077-19467 51650020 Object A 35 a-b-c 60727 09:37:30 101373-713077-19467 51650021 Object B 35 a-b-c 60727 09:51:19 102570-714392-19468 51650022 Object A 35 a-b-c 60727 09:57:35 102570-714392-19468 51650023 Object B 35 a-b-c 60727 10:10:23 102556-714385-19469 51650024 Object B 35 a-b-c 60727 10:16:44 102556-714385-19469 51650025 Object A 35 a-b-c 60727 10:28:48 102317-714010-19470 51650026 Object A 35 a-b-c 60727 10:35:05 102317-714010-19470 51650027 Object B 35 a-b-c 60727 10:48:25 103171-714776-19471 51650028 Object B 35 a-b-c 60727 10:54:46 103171-714776-19471 51650029 Object A 35 a-b-c 60727 11:06:24 103181-714788-19472 51650030 Object A 35 a-b-c 60727 11:12:40 103181-714788-19472 51650031 Object B 35 a-b-c 60727 12:05:31 103175-714782-19473 51650032 Domeflat B 5 a-b-c 60727 12:06:51 103175-714782-19473 51650033 Domeflat B 5 a-b-c 60727 12:08:11 103175-714782-19473 51650034 Domeflat B 5 a-b-c ThAr Offsets: ['-0.51'] UNe Offsets: ['-0.54'] Object Offsets: Max: 0.53, Min: 0.47, Mean: 0.50 num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ---- ------- ------- ------ ------ ---- ---- 51650001 100522 712171 19461 DARK 13.5 3 33 True -- -- -- -- -- -- 51650002 100522 712171 19461 DARK 13.5 60 -37 True -- -- -- -- -- -- 51650003 100522 712171 19461 QUARTZFLAT 13.5 10 103 True -- -0.29 -- 0.86 -- -- 51650004 100522 712171 19461 ARCLAMP-THAR 13.5 12 97 True 3.91 0.50 1.06 0.88 -- -- 51650005 100522 712171 19461 ARCLAMP-UNE 13.5 40 110 True 3.65 0.37 1.09 0.86 -- -- 51650006 -999 -999 19462 ARCLAMP-THAR 13.0 12 90 True 3.52 0.54 1.06 0.85 -- -- 51650007 -999 -999 19462 ARCLAMP-UNE 13.0 40 108 True 3.14 0.34 1.11 0.91 -- -- 51650008 -999 -999 19462 DARK 13.0 60 -2 True -- -- -- -- -- -- 51650009 -999 -999 19462 FPI 13.0 30 128 True 14.97 0.31 1.17 0.88 -- -- 51650010 -999 -999 19462 FPI 13.5 30 128 True -9.17 0.30 1.19 0.89 -- -- 51650011 -999 -999 19462 DOMEFLAT 13.5 5 108 True -- -0.24 -- 0.85 -- -- 51650012 -999 -999 19462 DOMEFLAT 13.5 5 108 True -- -0.22 -- 0.84 -- -- 51650013 -999 -999 19462 DOMEFLAT 13.5 5 111 True -- -0.22 -- 0.87 -- -- 51650014 101228 713004 19463 OBJECT 13.5 43 63 True 4.00 -0.22 1.75 0.85 93 104 51650015 101228 713004 19463 OBJECT 13.0 43 65 True 3.49 -0.23 1.74 0.88 92 106 51650016 101310 713031 19465 OBJECT 13.0 35 68 True 3.47 -0.22 1.74 0.88 92 103 51650017 101310 713031 19465 OBJECT 13.5 35 66 True 3.98 -0.22 1.75 0.84 90 104 51650018 101664 713602 19466 OBJECT 13.0 35 58 True 3.48 -0.23 1.74 0.88 91 103 51650019 101664 713602 19466 OBJECT 13.5 35 57 True 3.99 -0.21 1.76 0.85 93 100 51650020 101373 713077 19467 OBJECT 13.0 35 99 True 3.47 -0.22 1.72 0.89 91 99 51650021 101373 713077 19467 OBJECT 13.5 35 96 True 3.99 -0.21 1.70 0.86 91 96 51650022 102570 714392 19468 OBJECT 13.0 35 91 True 3.48 -0.22 1.71 0.89 91 100 51650023 102570 714392 19468 OBJECT 13.5 35 90 True 3.99 -0.20 1.71 0.85 89 102 51650024 102556 714385 19469 OBJECT 13.5 35 90 True 4.01 -0.21 1.77 0.86 91 101 51650025 102556 714385 19469 OBJECT 13.0 35 91 True 3.49 -0.21 1.75 0.88 93 102 51650026 102317 714010 19470 OBJECT 13.0 35 64 True 3.49 -0.22 1.75 0.88 90 103 51650027 102317 714010 19470 OBJECT 13.5 35 63 True 3.99 -0.21 1.75 0.85 93 103 51650028 103171 714776 19471 OBJECT 13.5 35 88 True 3.99 -0.20 1.77 0.85 92 101 51650029 103171 714776 19471 OBJECT 13.0 35 87 True 3.49 -0.21 1.75 0.89 90 103 51650030 103181 714788 19472 OBJECT 13.0 35 97 True 3.49 -0.22 1.73 0.88 91 104 51650031 103181 714788 19472 OBJECT 13.5 35 95 True 3.99 -0.20 1.75 0.86 91 103 51650032 103175 714782 19473 DOMEFLAT 13.5 5 110 True -- -0.20 -- 0.85 -- -- 51650033 103175 714782 19473 DOMEFLAT 13.5 5 109 True -- -0.20 -- 0.85 -- -- 51650034 103175 714782 19473 DOMEFLAT 13.5 5 111 True -- -0.21 -- 0.87 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 51650003 QUARTZFLAT 10 66 69 81 90 90 85 87 89 89 82 51650011 DOMEFLAT 5 75 82 93 101 108 96 101 105 101 94 51650012 DOMEFLAT 5 77 85 95 102 109 97 102 106 102 95 51650013 DOMEFLAT 5 78 87 96 103 110 97 102 106 102 96 51650032 DOMEFLAT 5 76 83 94 101 108 95 100 105 99 95 51650033 DOMEFLAT 5 78 87 96 102 109 97 101 106 101 97 51650034 DOMEFLAT 5 79 88 95 101 108 96 101 105 100 97 ---- BOSS Data Summary ---- MJD UTC Field-Design-Config Exposure Type SOS ETime Hart -------------------------------------------------------------------------------- 60727 22:47:10 100522-712171-19461 00381905 Bias r1-b1 0 Closed 60727 22:48:43 100522-712171-19461 00381906 Dark r1-b1 20 Closed 60727 22:50:56 100522-712171-19461 00381907 Hart r1-b1 4 Left 60727 22:51:36 100522-712171-19461 00381908 Hart r1-b1 4 Right 60727 22:52:43 100522-712171-19461 00381909 Arc r1-b1 4 Open 60727 22:54:29 100522-712171-19461 00381910 Flat r1-b1 25 Open 60727 07:56:38 -999- -999-19462 00381911 Hart r1-b1 4 Left 60727 07:57:18 -999- -999-19462 00381912 Hart r1-b1 4 Right 60727 07:58:45 -999- -999-19462 00381913 Flat r1-b1 25 Open 60727 08:00:34 -999- -999-19462 00381914 Flat r1-b1 25 Open 60727 08:02:24 -999- -999-19462 00381915 Flat r1-b1 25 Open 60727 08:06:38 101228-713004-19463 00381916 Flat r1-b1 25 Open 60727 08:09:14 101228-713004-19463 00381917 Arc r1-b1 4 Open 60727 08:23:11 101228-713004-19463 00381918 Science r1-b1 900 Open 60727 08:41:52 101462-713199-19464 00381919 Flat r1-b1 25 Open 60727 08:49:32 101310-713031-19465 00381920 Arc r1-b1 4 Open 60727 08:51:42 101310-713031-19465 00381921 Science r1-b1 730 Open 60727 09:09:24 101664-713602-19466 00381922 Arc r1-b1 4 Open 60727 09:11:32 101664-713602-19466 00381923 Science r1-b1 730 Open 60727 09:29:14 101373-713077-19467 00381924 Arc r1-b1 4 Open 60727 09:31:24 101373-713077-19467 00381925 Science r1-b1 730 Open 60727 09:49:08 102570-714392-19468 00381926 Arc r1-b1 4 Open 60727 09:51:28 102570-714392-19468 00381927 Science r1-b1 730 Open 60727 10:07:45 102556-714385-19469 00381928 Arc r1-b1 4 Open 60727 10:10:40 102556-714385-19469 00381929 Science r1-b1 730 Open 60727 10:26:32 102317-714010-19470 00381930 Arc r1-b1 4 Open 60727 10:29:06 102317-714010-19470 00381931 Science r1-b1 730 Open 60727 10:45:24 103171-714776-19471 00381932 Arc r1-b1 4 Open 60727 10:48:42 103171-714776-19471 00381933 Science r1-b1 730 Open 60727 11:04:39 103181-714788-19472 00381934 Arc r1-b1 4 Open 60727 11:06:42 103181-714788-19472 00381935 Science r1-b1 730 Open 60727 11:24:41 103175-714782-19473 00381936 Arc r1-b1 4 Open 60727 12:09:35 103175-714782-19473 00381937 Flat r1-b1 25 Open ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 22:52:10 3.8 -329 -4.2 510 -2.6 07:57:50 3.1 2308 -11.3 2953 -2.0 ---- Weather Log ---- Time Field-Design Temp DP Diff Humid Wind Dir 1μm Dust IRSC σ IRSC μ ================================================================================ 08:38:17 nan-nan nan nan 0.0 0.0 0.0 100 13 229.2 nan 08:57:37 101310-713031 0.0 0.0 0.0 100.0 13.1 243 nan 35.9 342 09:15:00 101664-713602 0.0 0.0 0.0 100.0 10.2 257 nan 10.9 342 09:31:14 101373-713077 0.0 0.0 0.0 100.0 15.1 250 nan 2.5 341 09:51:19 102570-714392 0.0 0.0 0.0 100.0 13.3 245 nan 1.9 342 10:10:23 102556-714385 0.0 0.0 0.0 100.0 17.1 243 nan 1.8 342 10:28:48 102317-714010 0.0 0.0 0.0 100.0 14.3 258 nan 1.8 342 10:45:00 103171-714776 0.0 0.0 0.0 100.0 13.5 246 nan 1.8 342 11:00:58 103171-714776 0.0 0.0 0.0 100.0 17.1 244 nan 1.7 342 11:18:53 103181-714788 0.0 0.0 0.0 100.0 12.7 257 nan 2.2 341 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60727 Fill night time in Hours below *=*=*= Total bright time : 4.1 *=*=*= Bright time, queued observations : 1.2 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0 *=*=*= Bright time lost to technical issues : 2.9 *=*=*= Total dark time : 6.7 *=*=*= Dark time, queued observations : 0.3 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 6.4 *=*=*= Dark time lost to technical issues : 0 Thu 20 Feb 2025 60727 0.45 01:47 02:01 12:31 05:03 15:35 --- dark until 08:43 ( 6.7 h), then bright ( 4.1 h, w/ 0.3 h for 12 deg design)