Subject: 2.5m obslog 20240610 (SJD 60472) From: doravetz@apo.nmsu.edu Date: Tue, 11 Jun 2024 11:23:29 +0000 (UTC) APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Monday June 10, 2024 (SJD 60472) ---- Observing Team ---- Dan Oravetz ---- Observing Plan ---- Mon 10 Jun 2024 60472 dark (7.0 h) plus 1.0 h for 8 deg design ---- Observing Summary ---- Similar weather conditions to last night: partly cloudy to begin with after a delayed opening due to virga, then a few hours lost to being totally clouded-out in the middle of the night, and finally clear skies to end the night. Time Field-Design-Conf DesignType BOSS/APOGEE Seeing Overhead Complete? Comment 03:02:10 101208-594871-14819 bright_time 1x730/1xAB 1.8" 0:00 DONE 03:20:55 101208-594872-14820 bright_time 1x730/1xAB 1.9" 4:48 DONE 03:37:42 112359-403333-14821 dark_rm 2x900/2xAB 1.6" 10:07 not done 07:11:28 101126-594021-14822 dark_plane 1x900/1xAB 1.4" 180:10 DONE 07:34:47 104507-604350-14823 dark_plane 1x900/1xAB 1.8" 6:21 not done 07:55:57 100964-593809-14824 dark_plane 1x900/1xAB 1.4" 6:54 not done 08:17:55 104455-604192-14825 dark_plane 1x900/1xAB 1.5" 8:33 not done 09:25:37 100971-593839-14826 dark_plane 2x900/2xAB 1.5" 52:04 DONE 10:09:25 100965-593815-14827 dark_plane 1x900/1xAB 1.3" 10:51 DONE 10:38:56 100570-592570-14829 bright_time 1x730/1xAB 1.4" 15:43 DONE Notes ===== (1) There was an unusual problem with BOSS that began with a CCD temperature alert (even though physically the temperature was OK), then a deluge of voltage warnings. During this time the camera exposed and read-out a couple of exposures while I was trying to figure out what was going on. Those exposures do have high-read-noise warnings associated with them in SOS (expNum 370070 and 71). It isn't clear to me what caused the alerts and warnings and what restored things back to normal, but details are in the Problems in Detail section at the end of the Log. (2) /data is 96% full (which appears to be the threshold at which SOS includes warning messages for each exposure that /data is getting to be quite full) (3) Dust wasn't too bad tonight, but getDust.py didn't work this morning. ---- Software Issues ---- Looks like something changed today with python environment in the observers account? (3.12.1) [observer@sdss5-hub: 60472]$ getDust.py pyenv: getDust.py: command not found The `getDust.py' command exists in these Python versions: 3.9.4/envs/observers39 observers39 Note: See 'pyenv help global' for tips on allowing both python2 and python3 to be found. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ All systems alive and responsive. Disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 2.0T 461G 1.6T 23% /home /dev/sdd1 22T 20T 1.2T 95% /data FPS powered on. Positioners are home/folded: 21:38:24 APO.doravetz_3 30 jaeger i folded=T 21:38:24 APO.doravetz_3 30 jaeger : text="All positioners are folded." Loaded a manual folded configuration: 21:45:43 CmdStarted: APO.doravetz_3 31 jaeger configuration load --from-positions 21:45:45 APO.doravetz_3 31 jaeger d text="Resetting cherno offsets." 21:45:45 APO.doravetz_3 31 jaeger d text="Focal plane scale: 1.0." 21:45:45 APO.doravetz_3 31 jaeger d configuration_loaded=14818,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/0148XX/confSummary-14818.par,F 21:45:50 APO.doravetz_3 31 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/60472/configuration_snapshot_60472_14818.pdf 21:45:50 APO.doravetz_3 31 jaeger d design_preloaded=-999 21:45:50 APO.doravetz_3 31 jaeger : text="Configuration 14818 loaded." 21:45:50 CmdDone: APO.doravetz_3 31 jaeger configuration load --from-positions Took a '--no-apply' FVC image. 21:46:42 FVC Loop fvc_rms= 40.67 fvc_deltarms= 43.29 fvc_filename=/data/fcam/60472/fimg-fvc1n-0001.fits fvc_perc_90= 60.70 fvc_percent_reached= 1.60 fvc iterations= 1 Scattered-light check: 21:47:11 CmdStarted: APO.doravetz_3 33 fliswarm talk -n fvc expose 50 fimg-fvc1n-0002.fits shows no evidence of light artifacts. Exercised positioners: 21:48:24 CmdStarted: APO.doravetz_3 35 jaeger configuration random --danger 21:50:27 APO.doravetz_3 35 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/60471/trajectory-60471-0040.json 21:50:28 CmdDone: APO.doravetz_3 35 jaeger configuration random --danger 21:50:32 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/60471/fps_snapshot_60471_0040.pdf 21:51:58 CmdStarted: APO.doravetz_3 36 jaeger configuration reverse GFA biases/darks: 21:53:42 CmdStarted: APO.doravetz_3 37 fliswarm talk -c gfa expose --bias --count 10 21:54:48 CmdStarted: APO.doravetz_3 38 fliswarm talk -c gfa expose --dark --count 5 5 01:25Z - Ventilation started. ============================================================================ NIGHT ============================================================================ Opening was delayed (just like last night) waiting for some residual cloudiness to improve. 02:48:33 2.5m Enclosure: OPEN. Force-filled BOSS LN2 dears. Partly cloudy sky. APOGEE dome flats: 02:51:01 APOGEE expNum=49100011, reads=5, dithPos=B, DomeFlat 02:52:20 APOGEE expNum=49100012, reads=5, dithPos=B, DomeFlat 02:53:39 APOGEE expNum=49100013, reads=5, dithPos=B, DomeFlat MJD DomeFlat MTPs:1 2 3 4 5 6 7 8 9 10 AZ ALT ROT 60472 49100011 0.98 1.00 1.02 1.01 1.02 1.00 0.97 1.00 0.99 0.94 121.0 30.0 -0.0 60472 49100012 1.00 1.00 1.02 1.01 1.02 1.00 0.97 1.00 0.99 0.95 121.0 30.0 0.0 60472 49100013 1.01 1.00 1.02 1.01 1.02 1.00 0.97 1.00 0.99 0.94 121.0 30.0 0.0 BOSS hartmanns: 02:55:50 BOSS expNum=370049, expTime= 4.0s 02:56:30 BOSS expNum=370050, expTime= 4.0s 02:57:17 sp1.motor.move= 3279, 3279, 3278 02:57:17 Hartmann collimate output sp1: offset: r= 2638 (Out of focus); b= -12.3 (Out of focus) sp1: pred. move: spAverageMove= 3279 sp1: pred. resid: r= -641, b= -2.0, txt= OK, spTemp = 9.0 BOSS flats: 02:57:59 BOSS expNum=370051, expTime= 25.0s 02:59:47 BOSS expNum=370052, expTime= 25.0s 03:01:35 BOSS expNum=370053, expTime= 25.0s **************************************************************** 03:02:11 Field-Design-Configuration: 101208-594871-14819 **************************************************************** 03:02:19 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 03:02:23 CmdStarted: jaeger configuration execute 03:02:27 CmdStarted: tcc track (slew begins) 03:02:30 HgCd lamps turned ON 03:03:08 CmdDone: Slew finished ('tcc track' command finished) 03:03:13 CmdDone: jaeger configuration execute 03:03:14 CmdStarted: FVC loop 03:04:21 CmdDone: FVC loop FVC loop results: fvc_rms= 38.57 fvc_deltarms= 16.15 fvc_filename=/data/fcam/60472/fimg-fvc1n-0004.fits fvc_perc_90= 12.80 fvc_percent_reached= 39.80 fvc iterations= 2 03:04:22 CmdStarted: tcc track (slew begins) 03:04:23 Ne lamps turned ON 03:04:28 CmdDone: Slew finished ('tcc track' command finished) 03:05:08 BOSS expNum=370054, expTime= 4.0s 03:05:16 HgCd lamps turned OFF 03:05:16 Ne lamps turned OFF 03:05:26 CmdStarted: cherno acquire 03:07:44 CmdDone: cherno acquire 03:07:45 CmdStarted: 'cherno guide' 03:07:46 hal expose: BOSS science, APOGEE science 03:08:12 APOGEE expNum=49100014, reads=35, dithPos=B, Object 03:08:17 BOSS expNum=370055, expTime= 730.0s 03:14:32 APOGEE expNum=49100015, reads=35, dithPos=A, Object **************************************************************** 03:20:56 Field-Design-Configuration: 101208-594872-14820 **************************************************************** 03:21:03 hal goto-field: slew, reconfigure, fvc, reslew, acquire, guide 03:21:07 CmdStarted: jaeger configuration reverse 03:21:10 CmdStarted: tcc track (slew begins) 03:21:16 CmdDone: Slew finished ('tcc track' command finished) 03:21:56 CmdDone: jaeger configuration reverse 03:21:56 CmdStarted: jaeger configuration execute 03:22:47 CmdDone: jaeger configuration execute 03:22:48 CmdStarted: FVC loop 03:23:53 CmdDone: FVC loop FVC loop results: fvc_rms= 11.83 fvc_deltarms= 42.75 fvc_filename=/data/fcam/60472/fimg-fvc1n-0006.fits fvc_perc_90= 12.90 fvc_percent_reached= 36.70 fvc iterations= 2 03:23:54 CmdStarted: tcc track (slew begins) 03:24:01 CmdDone: Slew finished ('tcc track' command finished) 03:24:03 CmdStarted: cherno acquire 03:24:23 New-field overhead: 0:03:36 03:24:30 CmdDone: cherno acquire 03:24:32 CmdStarted: 'cherno guide' 03:24:44 hal expose: BOSS science, APOGEE science 03:25:11 APOGEE expNum=49100016, reads=35, dithPos=A, Object 03:25:17 BOSS expNum=370056, expTime= 730.0s 03:31:27 APOGEE expNum=49100017, reads=35, dithPos=B, Object Getting cloudier again with clouds coming in from the north now. **************************************************************** 03:37:43 Field-Design-Configuration: 112359-403333-14821 **************************************************************** 03:37:49 hal goto-field: slew, reconfigure, fvc, reslew, flat, hartmanns, arc, acquire, guide 03:37:53 CmdStarted: jaeger configuration reverse 03:37:57 CmdStarted: tcc track (slew begins) 03:38:43 CmdDone: Slew finished ('tcc track' command finished) 03:38:43 CmdDone: jaeger configuration reverse 03:38:43 CmdStarted: jaeger configuration execute 03:39:44 CmdDone: jaeger configuration execute 03:39:45 CmdStarted: FVC loop 03:40:53 CmdDone: FVC loop FVC loop results: fvc_rms= 7.44 fvc_deltarms= 39.14 fvc_filename=/data/fcam/60472/fimg-fvc1n-0008.fits fvc_perc_90= 9.90 fvc_percent_reached= 53.20 fvc iterations= 2 03:40:54 CmdStarted: tcc track (slew begins) 03:41:03 CmdDone: Slew finished ('tcc track' command finished) 03:41:23 BOSS expNum=370057, expTime= 25.0s 03:41:52 HgCd lamps turned ON 03:41:52 Ne lamps turned ON 03:43:15 BOSS expNum=370058, expTime= 4.0s 03:43:55 BOSS expNum=370059, expTime= 4.0s 03:44:37 sp1.motor.move= -296, -297, -298 03:44:37 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -0.2 (In Focus) sp1: pred. move: spAverageMove= -297 sp1: pred. resid: r= -361, b= -1.1, txt= OK, spTemp = 8.4 03:44:56 BOSS expNum=370060, expTime= 4.0s 03:45:05 HgCd lamps turned OFF 03:45:05 Ne lamps turned OFF 03:45:14 CmdStarted: cherno acquire 03:46:59 CmdDone: cherno acquire 03:47:01 CmdStarted: 'cherno guide' 03:47:04 New-field overhead: 0:09:22 03:47:04 hal expose: BOSS science, APOGEE science 03:47:30 APOGEE expNum=49100018, reads=46, dithPos=B, Object 03:47:35 BOSS expNum=370061, expTime= 900.0s 03:55:47 APOGEE expNum=49100019, reads=46, dithPos=A, Object 04:03:57 APOGEE expNum=49100020, reads=43, dithPos=A, Object 04:04:02 BOSS expNum=370062, expTime= 900.0s 04:11:37 APOGEE expNum=49100021, reads=43, dithPos=B, Object 04:23Z - Both exposures were useless due to clouds. Slewed to stow and closed FF petals waiting for the sky conditions to improve. **************************************************************** 07:11:29 Field-Design-Configuration: 101126-594021-14822 **************************************************************** 07:11:35 hal goto-field: slew, reconfigure, fvc, reslew, hartmanns, arc, acquire, guide 07:11:40 CmdStarted: jaeger configuration execute 07:11:43 CmdStarted: tcc track (slew begins) 07:11:47 HgCd lamps turned ON 07:12:38 CmdDone: Slew finished ('tcc track' command finished) 07:12:39 CmdDone: jaeger configuration execute 07:12:40 CmdStarted: FVC loop 07:13:47 CmdDone: FVC loop FVC loop results: fvc_rms= 7.21 fvc_deltarms= 44.23 fvc_filename=/data/fcam/60472/fimg-fvc1n-0010.fits fvc_perc_90= 10.40 fvc_percent_reached= 51.40 fvc iterations= 2 07:13:48 CmdStarted: tcc track (slew begins) 07:13:49 Ne lamps turned ON 07:13:58 CmdDone: Slew finished ('tcc track' command finished) 07:14:16 BOSS expNum=370063, expTime= 4.0s 07:14:56 BOSS expNum=370064, expTime= 4.0s 07:15:41 sp1.motor.move= -2411, -2411, -2409 07:15:41 Hartmann collimate output sp1: offset: r= -2638 (Out of focus); b= 6.8 (Out of focus) sp1: pred. move: spAverageMove= -2411 sp1: pred. resid: r= -226, b= -0.7, txt= OK, spTemp = 8.5 07:16:00 BOSS expNum=370065, expTime= 4.0s 07:16:09 HgCd lamps turned OFF 07:16:09 Ne lamps turned OFF 07:16:18 CmdStarted: cherno acquire 07:17:47 CmdDone: cherno acquire 07:17:49 CmdStarted: 'cherno guide' 07:18:41 New-field overhead: 2:59:24 07:18:41 hal expose: BOSS science, APOGEE science 07:19:07 APOGEE expNum=49100022, reads=43, dithPos=B, Object 07:19:12 BOSS expNum=370066, expTime= 900.0s 07:26:53 APOGEE expNum=49100023, reads=43, dithPos=A, Object **************************************************************** 07:34:48 Field-Design-Configuration: 104507-604350-14823 **************************************************************** 07:34:58 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 07:34:58 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 07:35:02 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 07:35:07 CmdStarted: jaeger configuration reverse 07:35:10 CmdStarted: tcc track (slew begins) 07:35:13 HgCd lamps turned ON 07:35:45 CmdDone: Slew finished ('tcc track' command finished) 07:36:08 CmdDone: jaeger configuration reverse 07:36:08 CmdStarted: jaeger configuration execute 07:36:53 CmdDone: jaeger configuration execute 07:36:54 CmdStarted: FVC loop 07:38:01 CmdDone: FVC loop FVC loop results: fvc_rms= 40.7 fvc_deltarms= 8.57 fvc_filename=/data/fcam/60472/fimg-fvc1n-0012.fits fvc_perc_90= 10.00 fvc_percent_reached= 51.70 fvc iterations= 2 07:38:02 CmdStarted: tcc track (slew begins) 07:38:03 Ne lamps turned ON 07:38:12 CmdDone: Slew finished ('tcc track' command finished) 07:38:45 BOSS expNum=370067, expTime= 4.0s 07:38:54 HgCd lamps turned OFF 07:38:54 Ne lamps turned OFF 07:39:03 CmdStarted: cherno acquire 07:40:01 CmdDone: cherno acquire 07:40:03 CmdStarted: 'cherno guide' 07:40:04 hal expose: BOSS science, APOGEE science 07:40:30 APOGEE expNum=49100024, reads=43, dithPos=A, Object 07:40:35 BOSS expNum=370068, expTime= 900.0s 07:48:11 APOGEE expNum=49100025, reads=43, dithPos=B, Object 07:55:52 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 07:55:58 Field-Design-Configuration: 100964-593809-14824 **************************************************************** 07:56:06 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 07:56:10 CmdStarted: jaeger configuration reverse 07:56:14 CmdStarted: tcc track (slew begins) 07:56:18 HgCd lamps turned ON 07:56:56 CmdDone: jaeger configuration reverse 07:56:56 CmdStarted: jaeger configuration execute 07:57:45 CmdDone: jaeger configuration execute 07:58:22 CmdDone: Slew finished ('tcc track' command finished) 07:58:23 CmdStarted: FVC loop 07:59:29 CmdDone: FVC loop FVC loop results: fvc_rms= 10.26 fvc_deltarms= 49.42 fvc_filename=/data/fcam/60472/fimg-fvc1n-0014.fits fvc_perc_90= 11.50 fvc_percent_reached= 50.70 fvc iterations= 2 07:59:31 CmdStarted: tcc track (slew begins) 07:59:31 Ne lamps turned ON 07:59:37 CmdDone: Slew finished ('tcc track' command finished) 08:00:14 BOSS expNum=370069, expTime= 4.0s 08:00:23 HgCd lamps turned OFF 08:00:23 Ne lamps turned OFF 08:00:32 CmdStarted: cherno acquire 08:01:56 CmdDone: cherno acquire 08:01:57 CmdStarted: 'cherno guide' 08:01:59 hal expose: BOSS science, APOGEE science 08:02:30 BOSS expNum=370070, expTime= 900.0s 08:02:32 APOGEE expNum=49100026, reads=43, dithPos=A, Object 08:10:13 APOGEE expNum=49100027, reads=43, dithPos=B, Object 08:17:50 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 08:17:56 Field-Design-Configuration: 104455-604192-14825 **************************************************************** 08:18:03 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 08:18:07 CmdStarted: jaeger configuration reverse 08:18:11 CmdStarted: tcc track (slew begins) 08:18:14 HgCd lamps turned ON 08:18:57 CmdDone: jaeger configuration reverse 08:18:57 CmdStarted: jaeger configuration execute 08:20:05 CmdDone: jaeger configuration execute 08:20:58 CmdDone: Slew finished ('tcc track' command finished) 08:20:59 CmdStarted: FVC loop 08:22:07 CmdDone: FVC loop FVC loop results: fvc_rms= 10.61 fvc_deltarms= 43.96 fvc_filename=/data/fcam/60472/fimg-fvc1n-0016.fits fvc_perc_90= 12.20 fvc_percent_reached= 39.80 fvc iterations= 2 08:22:08 CmdStarted: tcc track (slew begins) 08:22:09 Ne lamps turned ON 08:22:16 CmdDone: Slew finished ('tcc track' command finished) 08:22:51 BOSS expNum=370071, expTime= 4.0s 08:23:00 HgCd lamps turned OFF 08:23:00 Ne lamps turned OFF 08:23:09 CmdStarted: cherno acquire 08:25:30 CmdDone: cherno acquire 08:25:31 CmdStarted: 'cherno guide' 08:25:33 hal expose: BOSS science, APOGEE science 08:26:04 BOSS expNum=370072, expTime= 900.0s 08:26:07 APOGEE expNum=49100028, reads=43, dithPos=A, Object 08:33:48 APOGEE expNum=49100029, reads=43, dithPos=B, Object 09:02:18 CmdStarted: tcc track (slew begins) 09:03:09 CmdDone: Slew finished ('tcc track' command finished) 09:06:55 CmdStarted: tcc track (slew begins) 09:07:33 CmdStarted: tcc track (slew begins) 09:08:15 CmdDone: Slew finished ('tcc track' command finished) 09:08:34 CmdStarted: tcc track (slew begins) 09:09:02 CmdDone: Slew finished ('tcc track' command finished) 09:09:07 CmdStarted: tcc track (slew begins) 09:09:35 CmdDone: Slew finished ('tcc track' command finished) 09:22:42 BOSS expNum=370073, expTime= 0.0s **************************************************************** 09:25:38 Field-Design-Configuration: 100971-593839-14826 **************************************************************** 09:25:50 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 09:25:54 CmdStarted: jaeger configuration reverse 09:25:58 CmdStarted: tcc track (slew begins) 09:26:01 HgCd lamps turned ON 09:27:01 CmdDone: jaeger configuration reverse 09:27:01 CmdStarted: jaeger configuration execute 09:27:13 CmdDone: Slew finished ('tcc track' command finished) 09:27:44 CmdDone: jaeger configuration execute 09:27:45 CmdStarted: FVC loop 09:28:50 CmdDone: FVC loop FVC loop results: fvc_rms= 9.86 fvc_deltarms= 49.94 fvc_filename=/data/fcam/60472/fimg-fvc1n-0018.fits fvc_perc_90= 11.70 fvc_percent_reached= 45.40 fvc iterations= 2 09:28:51 CmdStarted: tcc track (slew begins) 09:28:52 Ne lamps turned ON 09:28:59 CmdDone: Slew finished ('tcc track' command finished) 09:29:34 BOSS expNum=370074, expTime= 4.0s 09:29:42 HgCd lamps turned OFF 09:29:42 Ne lamps turned OFF 09:29:52 CmdStarted: cherno acquire 09:31:20 CmdDone: cherno acquire 09:31:22 CmdStarted: 'cherno guide' 09:32:38 New-field overhead: 1:58:08 09:32:38 hal expose: BOSS science, APOGEE science 09:33:04 APOGEE expNum=49100030, reads=43, dithPos=B, Object 09:33:09 BOSS expNum=370075, expTime= 900.0s 09:40:49 APOGEE expNum=49100031, reads=43, dithPos=A, Object 09:48:27 APOGEE expNum=49100032, reads=43, dithPos=A, Object 09:49:37 BOSS expNum=370076, expTime= 900.0s 09:56:08 APOGEE expNum=49100033, reads=43, dithPos=B, Object **************************************************************** 10:09:26 Field-Design-Configuration: 100965-593815-14827 **************************************************************** 10:10:08 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 10:10:08 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 10:10:12 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:10:16 CmdStarted: jaeger configuration reverse 0:10:20 CmdStarted: tcc track (slew begins) 10:10:24 HgCd lamps turned ON 10:10:40 CmdDone: Slew finished ('tcc track' command finished) 10:10:59 CmdDone: jaeger configuration reverse 10:10:59 CmdStarted: jaeger configuration execute 10:11:49 CmdDone: jaeger configuration execute 10:11:50 CmdStarted: FVC loop 10:12:56 CmdDone: FVC loop FVC loop results: fvc_rms= 18.66 fvc_deltarms= 40.42 fvc_filename=/data/fcam/60472/fimg-fvc1n-0020.fits fvc_perc_90= 12.10 fvc_percent_reached= 45.30 fvc iterations= 2 10:12:57 CmdStarted: tcc track (slew begins) 10:12:58 Ne lamps turned ON 10:13:08 CmdDone: Slew finished ('tcc track' command finished) 10:13:40 BOSS expNum=370077, expTime= 4.0s 10:13:48 HgCd lamps turned OFF 10:13:48 Ne lamps turned OFF 10:13:57 CmdStarted: cherno acquire 10:14:54 CmdDone: cherno acquire 10:14:56 CmdStarted: 'cherno guide' 10:14:57 hal expose: BOSS science, APOGEE science 10:15:28 BOSS expNum=370078, expTime= 900.0s 10:15:31 APOGEE expNum=49100034, reads=43, dithPos=A, Object 10:23:12 APOGEE expNum=49100035, reads=43, dithPos=B, Object 10:30:49 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 10:30:53 Field-Design-Configuration: 104657-605730-14828 **************************************************************** 10:31:00 hal goto-field: slew, reconfigure, fvc, reslew, flat, arc, acquire, guide 10:31:04 CmdStarted: jaeger configuration reverse 10:31:08 CmdStarted: tcc track (slew begins) 10:31:48 CmdDone: Slew finished ('tcc track' command finished) 10:31:54 CmdDone: jaeger configuration reverse 10:31:54 CmdStarted: jaeger configuration execute 10:32:44 CmdDone: jaeger configuration execute 10:32:44 CmdStarted: FVC loop 10:33:53 CmdDone: FVC loop FVC loop results: fvc_rms= 37.42 fvc_deltarms= 20.57 fvc_filename=/data/fcam/60472/fimg-fvc1n-0022.fits fvc_perc_90= 12.40 fvc_percent_reached= 47.00 fvc iterations= 2 10:33:54 CmdStarted: tcc track (slew begins) 10:34:01 CmdDone: Slew finished ('tcc track' command finished) Skipped this design because it's dark_monit and it's twilight, so switching to a bright_time design. **************************************************************** 10:38:57 Field-Design-Configuration: 100570-592570-14829 **************************************************************** 10:39:08 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:39:23 CmdStarted: jaeger configuration reverse 10:39:27 CmdStarted: tcc track (slew begins) 10:39:30 HgCd lamps turned ON 10:40:11 CmdDone: jaeger configuration reverse 10:40:11 CmdStarted: jaeger configuration execute 10:40:21 CmdDone: Slew finished ('tcc track' command finished) 10:40:58 CmdDone: jaeger configuration execute 10:40:59 CmdStarted: FVC loop 10:42:04 CmdDone: FVC loop FVC loop results: fvc_rms= 14.07 fvc_deltarms= 44.48 fvc_filename=/data/fcam/60472/fimg-fvc1n-0024.fits fvc_perc_90= 12.40 fvc_percent_reached= 46.40 fvc iterations= 2 10:42:06 CmdStarted: tcc track (slew begins) 10:42:06 Ne lamps turned ON 10:42:13 CmdDone: Slew finished ('tcc track' command finished) 10:42:49 BOSS expNum=370081, expTime= 4.0s 10:42:58 HgCd lamps turned OFF 10:42:58 Ne lamps turned OFF 10:43:07 CmdStarted: cherno acquire 10:44:31 CmdDone: cherno acquire 10:44:32 CmdStarted: 'cherno guide' 10:45:41 New-field overhead: 1:01:54 10:45:41 hal expose: BOSS science, APOGEE science 10:46:07 APOGEE expNum=49100036, reads=35, dithPos=B, Object 10:46:13 BOSS expNum=370082, expTime= 730.0s 10:52:28 APOGEE expNum=49100037, reads=35, dithPos=A, Object APOGEE dome flats: 11:02:07 APOGEE expNum=49100038, reads=5, dithPos=A, DomeFlat 11:03:27 APOGEE expNum=49100039, reads=5, dithPos=A, DomeFlat 11:04:46 APOGEE expNum=49100040, reads=5, dithPos=A, DomeFlat MJD DomeFlat MTPs:1 2 3 4 5 6 7 8 9 10 AZ ALT ROT 60472 49100038 0.99 1.00 1.02 1.01 1.02 1.00 0.97 1.00 0.99 0.95 121.0 30.0 0.0 60472 49100039 1.01 1.01 1.02 1.01 1.02 1.00 0.97 1.00 0.98 0.94 121.0 30.0 0.0 60472 49100040 1.01 1.01 1.01 1.00 1.01 0.99 0.97 1.00 0.99 0.95 121.0 30.0 0.0 11:12:08 2.5m Enclosure: CLOSED. FPS powered-off. Positioners are home/folded. APOGEE morning cals started. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: 0.2030 0.1970 Axial B: -0.0012 -0.0012 Axial C: 0.0580 0.0530 Trans D: 0.4150 0.4100 Lateral E: 0.3380 0.3380 Lateral F: 0.5455 0.5455 GALILS Commanded: 9800 -300 6750 19150 -3550 -3250 Actual: 9800 -300 6750 19150 -3550 -3250 SETMIR VALUES PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -1.0360 -1.0380 Axial B: 0.2210 0.2180 Axial C: -1.1080 -1.1050 Trans D: -0.0250 -0.0330 GALILS Commanded: 5888150 5814850 5951550 -1500 -7300 Actual: 5883573 5811725 5906099 -1500 -7300 SETMIR VALUES SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 SecOrient: 4656.97 -24.47 -17.47 18.33 125.87 ---- Telescope Status ---- Status at: 11:20Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: 692 695 712 712 LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 275 lbs, 22 psi FPS Status ========== CP10 Loop = 7.24 L/min GFA Loop = 2.17 L/min Supply pressure = 41.6 psi Return pressure = 29.4 psi Chiller Status ============== Set temperarature = 11.9 C Flow = 5.0 Gpm ---- Software Used ---- STUI : 1.7.19 alerts : 2.3.0 boss : 6.0.2 -boss.specMec : 3.0.4 apo : 4.0.1a0 fliswarm : 0.5.1a0 jaeger : 1.7.4 kaiju : 1.4.0 coordio : 1.11.2 fps_calibrati : 2024.04.01+91998f8 cherno : 1.0.0 astrometrynet : 0.93 hal : 1.3.4 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.1-1-ge461 -mcp.plc : 35 -mcp.azFiduci : v1_149 -mcp.altFiduc : v1_149 -mcp.rotFiduc : v1_149 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : v1_7_0dev STUIAdditions : 3.1.18 boson : 0.4.9 sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.1-1-ge461 PLC : V35 Fiducials : v1_149 idlspec2d : v6_1_3 Kronos : 1.7.0 roboscheduler : 1.4.0 ---- APOGEE Data Summary ---- 60471 morning cals: num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- 49090052 100570 592569 14817 DARK 13.5 60 20 True -- -- -- -- -- -- 49090053 100570 592569 14817 DARK 13.5 60 13 True -- -- -- -- -- -- 49090054 100570 592569 14817 DARK 13.5 60 3 True -- -- -- -- -- -- 49090055 100570 592569 14817 QUARTZFLAT 13.5 10 109 True -- -0.19 -- 0.86 -- -- 49090056 100570 592569 14817 QUARTZFLAT 13.5 10 109 True -- -0.18 -- 0.86 -- -- 49090057 100570 592569 14817 QUARTZFLAT 13.5 10 109 True -- -0.19 -- 0.85 -- -- 49090058 100570 592569 14817 ARCLAMP-THAR 13.5 12 103 True 1.08 0.61 1.07 0.90 -- -- 49090059 100570 592569 14817 ARCLAMP-UNE 13.5 40 95 True 0.68 0.44 1.10 0.88 -- -- 49090060 100570 592569 14817 ARCLAMP-THAR 13.0 12 97 True 0.53 0.66 1.06 0.88 -- -- 49090061 100570 592569 14817 ARCLAMP-UNE 13.0 40 92 True 0.14 0.42 1.10 0.90 -- -- 49090062 100570 592569 14817 DARK 13.0 30 7 False -- -- -- -- -- -- 49090063 100570 592569 14817 INTERNALFLAT 13.0 30 99 True -- -- -- -- -- -- 49090064 100570 592569 14817 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 49090065 100570 592569 14817 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- 49090066 100570 592569 14817 DARK 13.0 30 263 True -- -- -- -- -- -- 49090067 100570 592569 14817 FPI 13.0 30 133 True 12.00 0.38 1.15 0.86 -- -- 49090068 100570 592569 14817 FPI 13.5 30 138 True -12.11 0.39 1.17 0.88 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 49090055 QUARTZFLAT 10 69 76 83 92 93 88 88 91 90 84 49090056 QUARTZFLAT 10 72 78 84 93 94 89 89 91 91 85 49090057 QUARTZFLAT 10 72 80 84 93 94 89 89 92 91 85 (3.12.1) [observer@sdss5-hub: 60472]$ check_ap 60472 connect /private/tmp/com.apple.launchd.tp4CbWBPj2/org.xquartz:0: Connection refused Using nfowler=1 for DOMEFLAT Using nfowler=1 for DOMEFLAT Using nfowler=1 for DOMEFLAT Using nfowler=1 for DOMEFLAT Using nfowler=1 for DOMEFLAT Using nfowler=1 for DOMEFLAT num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 -------- ------ ------ ------ ------------ ------- ----- ------- ---- ------- ------- ------ ------ ---- ---- 49100001 100570 592569 14817 DARK 13.5 3 100 True -- -- -- -- -- -- 49100002 -999 -999 14818 DARK 13.5 60 -23 True -- -- -- -- -- -- 49100003 -999 -999 14818 QUARTZFLAT 13.5 10 106 True -- -0.28 -- 0.86 -- -- 49100004 -999 -999 14818 ARCLAMP-THAR 13.5 12 101 True 1.11 0.53 1.07 0.89 -- -- 49100005 -999 -999 14818 ARCLAMP-UNE 13.5 40 98 True 0.71 0.38 1.09 0.86 -- -- 49100006 -999 -999 14818 ARCLAMP-THAR 13.0 12 95 True 0.54 0.58 1.04 0.86 -- -- 49100007 -999 -999 14818 ARCLAMP-UNE 13.0 40 96 True 0.17 0.37 1.14 0.91 -- -- 49100008 -999 -999 14818 DARK 13.0 60 3 True -- -- -- -- -- -- 49100009 -999 -999 14818 FPI 13.0 30 139 True 12.01 0.30 1.16 0.86 -- -- 49100010 -999 -999 14818 FPI 13.5 30 137 True -12.10 0.31 1.17 0.89 -- -- 49100011 -999 -999 14818 DOMEFLAT 13.5 5 107 True -- -0.25 -- 0.84 -- -- 49100012 -999 -999 14818 DOMEFLAT 13.5 5 107 True -- -0.23 -- 0.86 -- -- 49100013 -999 -999 14818 DOMEFLAT 13.5 5 107 True -- -0.26 -- 0.84 -- -- 49100014 101208 594871 14819 OBJECT 13.5 35 103 True 1.09 -0.25 1.76 0.87 102 101 49100015 101208 594871 14819 OBJECT 13.0 35 109 True 0.55 -0.27 1.77 0.90 104 100 49100016 101208 594872 14820 OBJECT 13.0 35 100 True 0.55 -0.26 1.77 0.91 105 101 49100017 101208 594872 14820 OBJECT 13.5 35 97 True 1.08 -0.26 1.76 0.87 99 100 49100018 112359 403333 14821 OBJECT 13.5 46 69 True 1.09 -0.22 1.76 0.88 103 103 49100019 112359 403333 14821 OBJECT 13.0 46 58 True 0.55 -0.26 1.75 0.90 106 104 49100020 112359 403333 14821 OBJECT 13.0 43 41 True 0.54 -0.26 1.75 0.89 108 104 49100021 112359 403333 14821 OBJECT 13.5 43 24 True 1.10 -0.22 1.75 0.87 104 103 49100022 101126 594021 14822 OBJECT 13.5 43 65 True 1.08 -0.20 1.76 0.89 102 101 49100023 101126 594021 14822 OBJECT 13.0 43 60 True 0.54 -0.23 1.74 0.90 105 102 49100024 104507 604350 14823 OBJECT 13.0 43 64 True 0.54 -0.23 1.76 0.90 108 102 49100025 104507 604350 14823 OBJECT 13.5 43 56 True 1.09 -0.21 1.75 0.86 104 104 49100026 100964 593809 14824 OBJECT 13.0 43 61 True 0.52 -0.23 1.72 0.90 108 102 49100027 100964 593809 14824 OBJECT 13.5 43 55 True 1.08 -0.21 1.74 0.86 105 102 49100028 104455 604192 14825 OBJECT 13.0 43 60 True 0.55 -0.22 1.75 0.90 104 102 49100029 104455 604192 14825 OBJECT 13.5 43 59 True 1.08 -0.21 1.75 0.85 104 103 49100030 100971 593839 14826 OBJECT 13.5 43 51 True 1.10 -0.20 1.75 0.86 105 104 49100031 100971 593839 14826 OBJECT 13.0 43 50 True 0.54 -0.21 1.75 0.90 106 103 49100032 100971 593839 14826 OBJECT 13.0 43 53 True 0.53 -0.21 1.76 0.90 108 106 49100033 100971 593839 14826 OBJECT 13.5 43 55 True 1.09 -0.20 1.74 0.85 105 102 49100034 100965 593815 14827 OBJECT 13.0 43 38 True 0.53 -0.21 1.75 0.89 107 102 49100035 100965 593815 14827 OBJECT 13.5 43 37 True 1.08 -0.19 1.74 0.86 106 103 49100036 100570 592570 14829 OBJECT 13.5 35 73 True 1.13 -0.18 1.74 0.88 102 101 49100037 100570 592570 14829 OBJECT 13.0 35 76 True 0.59 -0.21 1.75 0.89 107 103 49100038 100570 592570 14829 DOMEFLAT 13.0 5 111 True -- -0.19 -- 0.87 -- -- 49100039 100570 592570 14829 DOMEFLAT 13.0 5 111 True -- -0.19 -- 0.85 -- -- 49100040 100570 592570 14829 DOMEFLAT 13.0 5 112 True -- -0.19 -- 0.86 -- -- num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- 49100003 QUARTZFLAT 10 68 74 83 92 93 88 89 91 91 83 49100011 DOMEFLAT 5 77 84 95 104 111 99 104 109 104 97 49100012 DOMEFLAT 5 76 84 94 101 109 96 101 106 101 95 49100013 DOMEFLAT 5 79 88 96 104 111 99 104 109 103 98 49100038 DOMEFLAT 5 78 88 96 104 111 99 104 109 104 99 49100039 DOMEFLAT 5 81 90 97 104 112 100 105 110 105 99 49100040 DOMEFLAT 5 82 91 98 105 113 101 106 111 106 101 ---- BOSS Data Summary ---- Reading FITS from /data/spectro//60472 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 21:37 b1-00370042 ----------- ---- bia Clo 0.0 4086 4088 4073 2 21:37 r1-00370042 ----------- ---- bia Clo 0.0 4086 4088 4073 3 21:59 b1-00370043 -999--999 14818 arc Lef 4.1 4086 4088 4073 HART 4 21:59 r1-00370043 -999--999 14818 arc Lef 4.1 4086 4088 4073 HART 5 21:59 r1-00370044 -999--999 14818 arc Rig 4.1 4086 4088 4073 HART 6 21:59 b1-00370044 -999--999 14818 arc Rig 4.1 4086 4088 4073 HART 7 22:01 b1-00370045 -999--999 14818 arc Lef 4.1 3789 3791 3779 HART 8 22:01 r1-00370045 -999--999 14818 arc Lef 4.1 3789 3791 3779 HART 9 22:02 b1-00370046 -999--999 14818 arc Rig 4.1 3789 3791 3779 HART 10 22:02 r1-00370046 -999--999 14818 arc Rig 4.1 3789 3791 3779 HART 11 22:03 b1-00370047 -999--999 14818 arc Out 4.1 3652 3654 3643 12 22:03 r1-00370047 -999--999 14818 arc Out 4.1 3652 3654 3643 13 22:05 b1-00370048 -999--999 14818 fla Out 25.1 3652 3654 3643 14 22:05 r1-00370048 -999--999 14818 fla Out 25.1 3652 3654 3643 15 02:55 r1-00370049 -999--999 14818 arc Lef 4.1 3652 3654 3643 HART 16 02:55 b1-00370049 -999--999 14818 arc Lef 4.1 3652 3654 3643 HART 17 02:56 b1-00370050 -999--999 14818 arc Rig 4.1 3652 3654 3643 HART 18 02:56 r1-00370050 -999--999 14818 arc Rig 4.1 3652 3654 3643 HART 19 02:57 r1-00370051 -999--999 14818 fla Out 25.1 6931 6933 6921 20 02:57 b1-00370051 -999--999 14818 fla Out 25.1 6931 6933 6921 21 02:59 b1-00370052 -999--999 14818 fla Out 25.1 6931 6933 6921 22 02:59 r1-00370052 -999--999 14818 fla Out 25.1 6931 6933 6921 23 03:01 r1-00370053 -999--999 14818 fla Out 25.1 6931 6933 6921 24 03:01 b1-00370053 -999--999 14818 fla Out 25.1 6931 6933 6921 25 03:05 r1-00370054 594871-101208 14819 arc Out 4.1 6931 6933 6921 26 03:05 b1-00370054 594871-101208 14819 arc Out 4.1 6931 6933 6921 27 03:08 r1-00370055 594871-101208 14819 sci Out 730.1 6931 6933 6921 28 03:08 b1-00370055 594871-101208 14819 sci Out 730.1 6931 6933 6921 29 03:25 r1-00370056 594872-101208 14820 sci Out 730.1 6931 6933 6921 30 03:25 b1-00370056 594872-101208 14820 sci Out 730.1 6931 6933 6921 31 03:41 b1-00370057 403333-112359 14821 fla Out 25.1 6931 6933 6921 32 03:41 r1-00370057 403333-112359 14821 fla Out 25.1 6931 6933 6921 33 03:43 r1-00370058 403333-112359 14821 arc Lef 4.1 6931 6933 6921 HART 34 03:43 b1-00370058 403333-112359 14821 arc Lef 4.1 6931 6933 6921 HART 35 03:43 b1-00370059 403333-112359 14821 arc Rig 4.1 6931 6933 6921 HART 36 03:43 r1-00370059 403333-112359 14821 arc Rig 4.1 6931 6933 6921 HART 37 03:44 b1-00370060 403333-112359 14821 arc Out 4.1 6635 6636 6623 38 03:44 r1-00370060 403333-112359 14821 arc Out 4.1 6635 6636 6623 39 03:47 r1-00370061 403333-112359 14821 sci Out 900.1 6635 6636 6623 40 03:47 b1-00370061 403333-112359 14821 sci Out 900.1 6635 6636 6623 41 04:04 r1-00370062 403333-112359 14821 sci Out 900.1 6635 6636 6623 42 04:04 b1-00370062 403333-112359 14821 sci Out 900.1 6635 6636 6623 43 07:14 b1-00370063 594021-101126 14822 arc Lef 4.1 6635 6636 6623 HART 44 07:14 r1-00370063 594021-101126 14822 arc Lef 4.1 6635 6636 6623 HART 45 07:14 b1-00370064 594021-101126 14822 arc Rig 4.1 6635 6636 6623 HART 46 07:14 r1-00370064 594021-101126 14822 arc Rig 4.1 6635 6636 6623 HART 47 07:16 r1-00370065 594021-101126 14822 arc Out 4.1 4224 4225 4214 48 07:16 b1-00370065 594021-101126 14822 arc Out 4.1 4224 4225 4214 49 07:19 b1-00370066 594021-101126 14822 sci Out 900.1 4224 4225 4214 50 07:19 r1-00370066 594021-101126 14822 sci Out 900.1 4224 4225 4214 51 07:38 b1-00370067 604350-104507 14823 arc Out 4.1 4224 4225 4214 52 07:38 r1-00370067 604350-104507 14823 arc Out 4.1 4224 4225 4214 53 07:40 b1-00370068 604350-104507 14823 sci Out 900.1 4224 4225 4214 54 07:40 r1-00370068 604350-104507 14823 sci Out 900.1 4224 4225 4214 55 08:00 b1-00370069 593809-100964 14824 arc Out 4.1 4224 4225 4214 56 08:00 r1-00370069 593809-100964 14824 arc Out 4.1 4224 4225 4214 57 08:02 b1-00370070 593809-100964 14824 sci Out 900.1 4224 4225 4214 58 08:02 r1-00370070 593809-100964 14824 sci Out 900.1 4224 4225 4214 59 08:22 r1-00370071 604192-104455 14825 arc Out 4.1 4224 4225 4214 60 08:22 b1-00370071 604192-104455 14825 arc Out 4.1 4224 4225 4214 61 08:26 r1-00370072 604192-104455 14825 sci Out 900.1 4224 4225 4214 62 08:26 b1-00370072 604192-104455 14825 sci Out 900.1 4224 4225 4214 63 09:22 r1-00370073 ----------- ---- bia Clo 0.0 4224 4225 4214 64 09:22 b1-00370073 ----------- ---- bia Clo 0.0 4224 4225 4214 65 09:29 b1-00370074 593839-100971 14826 arc Out 4.1 4224 4225 4214 66 09:29 r1-00370074 593839-100971 14826 arc Out 4.1 4224 4225 4214 67 09:33 b1-00370075 593839-100971 14826 sci Out 900.1 4224 4225 4214 68 09:33 r1-00370075 593839-100971 14826 sci Out 900.1 4224 4225 4214 69 09:49 r1-00370076 593839-100971 14826 sci Out 900.1 4224 4225 4214 70 09:49 b1-00370076 593839-100971 14826 sci Out 900.1 4224 4225 4214 71 10:13 r1-00370077 593815-100965 14827 arc Out 4.1 4224 4225 4214 72 10:13 b1-00370077 593815-100965 14827 arc Out 4.1 4224 4225 4214 73 10:15 b1-00370078 593815-100965 14827 sci Out 900.1 4224 4225 4214 74 10:15 r1-00370078 593815-100965 14827 sci Out 900.1 4224 4225 4214 75 10:34 b1-00370079 605730-104657 14828 fla Out 25.1 4224 4225 4214 76 10:34 r1-00370079 605730-104657 14828 fla Out 25.1 4224 4225 4214 77 10:36 b1-00370080 605730-104657 14828 arc Out 4.1 4224 4225 4214 78 10:36 r1-00370080 605730-104657 14828 arc Out 4.1 4224 4225 4214 79 10:42 b1-00370081 592570-100570 14829 arc Out 4.1 4224 4225 4214 80 10:42 r1-00370081 592570-100570 14829 arc Out 4.1 4224 4225 4214 81 10:46 r1-00370082 592570-100570 14829 sci Out 730.1 4224 4225 4214 82 10:46 b1-00370082 592570-100570 14829 sci Out 730.1 4224 4225 4214 ---- BOSS Spectrographs Collimation ---- Afternoon focus check ===================== 22:01:50 BOSS expNum=370045, expTime= 4.0s 22:02:30 BOSS expNum=370046, expTime= 4.0s 22:03:10 APOGEE expNum=49100005, reads=40, dithPos=B, ArcLamp 22:03:11 sp1.motor.move= -137, -137, -136 22:03:11 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -1.2 (In Focus) sp1: pred. move: spAverageMove= -137 sp1: pred. resid: r= -522, b= -1.6, txt= OK, spTemp = 9.7 22:03:08 APO.doravetz_3 45 hartmann i r1MeanOffset=-0.10,"In Focus" 22:03:08 APO.doravetz_3 45 hartmann i b1MeanOffset=0.06,"In Focus" Night ===== Time TSP1 R1 B1 Move1 B1Resid ============================================= 02:57:00 8.8 2638 -12.3 3279 -2.0 03:44:30 8.2 -659 -0.2 -297 -1.1 07:15:30 8.2 -2638 6.8 -2411 -0.7 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 60472 Fill night time in Hours below *=*=*= Total bright time : 1.0 *=*=*= Bright time, queued observations : 1.0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0 *=*=*= Bright time lost to technical issues : 0 *=*=*= Total dark time : 7.0 *=*=*= Dark time, queued observations : 3.0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 3.5 *=*=*= Dark time lost to technical issues : 0.5 ---- Problems in Detail ---- A critical alert about r1 CCD temperature spawned about 08:14Z. The 10L LN2 pressure was an acceptable 11 psi and I didn't have reason to believe a lack of LN2 was the problem because the reported CCD temperature was jumping around in an unphysical way when I queried the status, so this felt like an electronics or scrambled-communication kind of issue. There was an exposure in-progress at the time (expNum 370070) and with only a few minutes left on the integration time I let it continue, curious to see if the readout would choke. The readout completed without any issues and I decided to let the next design load and continue in the background while I looked at the image. The image looked OK "by eye" and then when SOS reported, the SN2 values weren't unreasonable given the partly-cloudy sky, however, there was a high-read-noise warning: sdR-r1-00370070.fit.gz: WARNING: r1 Amp 3 high read noise = 7.76 By this point, in addition to the SP1R0CCDTempRead alert, there were several voltage warnings reported by camCheck: Critical camCheck.SP1R0CCDTempRead at 2024-06-11 08:14:37 UT found 'Reported by camCheck' Warn camCheck.SP1R1VLG3Read at 2024-06-11 08:14:43 UT found 'Reported by camCheck' Warn camCheck.SP1R1VLG2Read at 2024-06-11 08:14:42 UT found 'Reported by camCheck' Warn camCheck.SP1R0VLG1Read at 2024-06-11 08:14:39 UT found 'Reported by camCheck' Warn camCheck.SP1R1VRedPurgeRead at 2024-06-11 08:14:44 UT found 'Reported by camCheck' Warn camCheck.SP1R1VRonRead at 2024-06-11 08:14:41 UT found 'Reported by camCheck' Warn camCheck.SP1R1INegTrim4Read at 2024-06-11 08:14:40 UT found 'Reported by camCheck' Warn camCheck.SP1R0VSWbRead at 2024-06-11 08:14:38 UT found 'Reported by camCheck' Also curious was that when I did an aliveness check of BOSS with a 'ping', sp1cam reported a 'DEAD' status, but it didn't last and soon reported an 'OK' state without any intervention on my part. Here's a time stamp for that: 08:18:19 APO.doravetz 56 boss w text="sp1cam: DEAD" Next, there was a b1 CCD temperature alert add to the list of alerts: Critical camCheck.SP1B2CCDTempRead at 2024-06-11 08:20:12 UT found 'Reported by camCheck' The reported temperatures were still bouncing around in an unphysical way: b1 reported -48C, then -90C, for example, on the next status request; shortly thereafter, r1 reported -15.2C and b1 -21.9C, before they jumped up again by 50+ deg C. In addition, a whole lot more voltage warnings: Warn camCheck.SP1B3VRoffRead at 2024-06-11 08:22:29 UT found 'Reported by camCheck' Warn camCheck.SP1B3VRD4Read at 2024-06-11 08:22:36 UT found 'Reported by camCheck' Warn camCheck.SP1B3VSb2Read at 2024-06-11 08:22:35 UT found 'Reported by camCheck' Warn camCheck.SP1B3VPcCRead at 2024-06-11 08:22:34 UT found 'Reported by camCheck' Warn camCheck.SP1B3VPcDRead at 2024-06-11 08:22:33 UT found 'Reported by camCheck' Warn camCheck.SP1B3VPbCRead at 2024-06-11 08:22:32 UT found 'Reported by camCheck' Warn camCheck.SP1B3VPbDRead at 2024-06-11 08:22:31 UT found 'Reported by camCheck' Warn camCheck.SP1B3VLG34Read at 2024-06-11 08:22:30 UT found 'Reported by camCheck' Warn camCheck.SP1B3IPosTrim3Read at 2024-06-11 08:22:28 UT found 'Reported by camCheck' Warn camCheck.SP1B2TZeroRead at 2024-06-11 08:22:13 UT found 'Reported by camCheck' Warn camCheck.SP1B2VDD1Read at 2024-06-11 08:22:26 UT found 'Reported by camCheck' Warn camCheck.SP1B2VDD2Read at 2024-06-11 08:22:25 UT found 'Reported by camCheck' Warn camCheck.SP1B2VSb1Read at 2024-06-11 08:22:22 UT found 'Reported by camCheck' Warn camCheck.SP1B2VPbARead at 2024-06-11 08:22:19 UT found 'Reported by camCheck' Warn camCheck.SP1B2VLG12Read at 2024-06-11 08:22:16 UT found 'Reported by camCheck' Warn camCheck.SP1R1VRD4Read at 2024-06-11 08:22:08 UT found 'Reported by camCheck' Warn camCheck.SP1R1VSb2Read at 2024-06-11 08:22:07 UT found 'Reported by camCheck' Warn camCheck.SP1B3IPosTrim4Read at 2024-06-11 08:22:27 UT found 'Reported by camCheck' Warn camCheck.SP1B2VRD1Read at 2024-06-11 08:22:24 UT found 'Reported by camCheck' Warn camCheck.SP1B2VRD2Read at 2024-06-11 08:22:23 UT found 'Reported by camCheck' Warn camCheck.SP1B2VPcARead at 2024-06-11 08:22:21 UT found 'Reported by camCheck' Warn camCheck.SP1B2VPcBRead at 2024-06-11 08:22:20 UT found 'Reported by camCheck' Warn camCheck.SP1B2VPbBRead at 2024-06-11 08:22:18 UT found 'Reported by camCheck' Warn camCheck.SP1B2VTcRead at 2024-06-11 08:22:17 UT found 'Reported by camCheck' Warn camCheck.SP1R1VRD3Read at 2024-06-11 08:22:09 UT found 'Reported by camCheck' Warn camCheck.SP1R1VPbCRead at 2024-06-11 08:22:06 UT found 'Reported by camCheck' Warn camCheck.SP1R1VLG4Read at 2024-06-11 08:22:03 UT found 'Reported by camCheck' Warn camCheck.SP1R1IPosTrim3Read at 2024-06-11 08:21:59 UT found 'Reported by camCheck' Warn camCheck.SP1R1INegTrim3Read at 2024-06-11 08:21:57 UT found 'Reported by camCheck' Warn camCheck.SP1R1IPosTrim4Read at 2024-06-11 08:21:58 UT found 'Reported by camCheck' Warn camCheck.SP1R1VLG3Read at 2024-06-11 08:22:02 UT found 'Reported by camCheck' Warn camCheck.SP1R1VLG2Read at 2024-06-11 08:22:01 UT found 'Reported by camCheck' Warn camCheck.SP1R1VRonRead at 2024-06-11 08:22:00 UT found 'Reported by camCheck' Warn camCheck.SP1R1VRedPurgeRead at 2024-06-11 08:22:04 UT found 'Reported by camCheck' Warn camCheck.SP1B2IPosTrim1Read at 2024-06-11 08:22:15 UT found 'Reported by camCheck' Warn camCheck.SP1B2IPosTrim2Read at 2024-06-11 08:22:14 UT found 'Reported by camCheck' Warn camCheck.SP1R1VRedEraseRead at 2024-06-11 08:22:05 UT found 'Reported by camCheck' Warn camCheck.SP1R1VDD3Read at 2024-06-11 08:22:11 UT found 'Reported by camCheck' Warn camCheck.SP1R1VDD4Read at 2024-06-11 08:22:10 UT found 'Reported by camCheck' Warn camCheck.SP1R0VDD1Read at 2024-06-11 08:21:55 UT found 'Reported by camCheck' Warn camCheck.SP1R0VDD2Read at 2024-06-11 08:21:54 UT found 'Reported by camCheck' Warn camCheck.SP1R0VRD2Read at 2024-06-11 08:21:52 UT found 'Reported by camCheck' Warn camCheck.SP1R0VRD1Read at 2024-06-11 08:21:53 UT found 'Reported by camCheck' Warn camCheck.SP1R1INegTrim4Read at 2024-06-11 08:21:56 UT found 'Reported by camCheck' Warn camCheck.SP1B3VDD3Read at 2024-06-11 08:21:40 UT found 'Reported by camCheck' Warn camCheck.SP1B3VDD4Read at 2024-06-11 08:21:39 UT found 'Reported by camCheck' Warn camCheck.SP1R0VTcRead at 2024-06-11 08:21:45 UT found 'Reported by camCheck' Warn camCheck.SP1R0VSb1Read at 2024-06-11 08:21:51 UT found 'Reported by camCheck' Warn camCheck.SP1R0VPbARead at 2024-06-11 08:21:50 UT found 'Reported by camCheck' Warn camCheck.SP1R0VPbBRead at 2024-06-11 08:21:49 UT found 'Reported by camCheck' Warn camCheck.SP1R0VPcARead at 2024-06-11 08:21:48 UT found 'Reported by camCheck' Warn camCheck.SP1R0VPcBRead at 2024-06-11 08:21:47 UT found 'Reported by camCheck' Warn camCheck.SP1R0VTbRead at 2024-06-11 08:21:46 UT found 'Reported by camCheck' Warn camCheck.SP1R0IPosTrim1Read at 2024-06-11 08:21:42 UT found 'Reported by camCheck' Warn camCheck.SP1R0IPosTrim2Read at 2024-06-11 08:21:41 UT found 'Reported by camCheck' Warn camCheck.SP1R0VLG1Read at 2024-06-11 08:21:44 UT found 'Reported by camCheck' Warn camCheck.SP1R0VSWbRead at 2024-06-11 08:21:43 UT found 'Reported by camCheck' Warn camCheck.SP1R0INegTrim1Read at 2024-06-11 08:21:40 UT found 'Reported by camCheck' Warn camCheck.SP1B3VRD3Read at 2024-06-11 08:21:38 UT found 'Reported by camCheck' Warn camCheck.SP1R0INegTrim2Read at 2024-06-11 08:21:39 UT found 'Reported by camCheck' I didn't see a DACS_SET_ERROR, which in the past has sometimes accompanied this kind of thing (an unphysical temperature alert and a bunch of messed-up voltages) and indicated some problem with a communication link. I tried an 'iack' a couple of times (since there wasn't a DACS_SET_ERROR I figured the 'iack' should at least propgate all the way through and wouldn't get stuck due to a comm problem), but the 'iack' didn't fix the problem even though it completed. I also tried some 'boss ackErrors' after the 'iack' just in case anything was still stuck camCheck-error-reporting-wise, but that also didn't clear the alerts/warnings. Below is a comparison of NOMINAL voltages and READ voltages (and I'll continue writing below that block of text): 08:50:54 CmdStarted: APO.doravetz 124 boss sp1cam raw=shownomvolts 08:50:54 APO.doravetz 124 boss i text="Sending raw command SHOWNOMVOLTS to sp1cam" 08:50:54 APO.doravetz 124 boss d text="Sending SHOWNOMVOLTS to sp1cam" 08:50:57 APO.doravetz 124 boss d text="Completed message from sp1cam" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'# CAMERA 0 ZB CAM# 0 '" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_LN2TEMP_NOM 584'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_HEATERV_NOM NONE THIS CAM'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_CCDTEMP_NOM 188'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_TSET_NOM 575'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_TZERO_NOM 374'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_I-TRIM2_NOM 313'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_I-TRIM1_NOM 313'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_I+TRIM2_NOM 253'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_I+TRIM1_NOM 253'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VSW-_NOM 258'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VLG_NOM 288'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VT-_NOM 360'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VT+_NOM 770'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VP3-_NOM 360'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VP12-_NOM 360'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VP3+_NOM 770'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VP12+_NOM 770'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VS-_NOM 104'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VS+_NOM 720'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VR_NOM 696'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VRD2_NOM 833'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VRD1_NOM 833'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VDD2_NOM 882'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VDD1_NOM 882'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b''" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'# CAMERA 1 ZB CAM# 1 '" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_LN2TEMP_NOM NONE THIS CAM'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_HEATERV_NOM 768'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_CCDTEMP_NOM NONE THIS CAM'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_TSET_NOM 561'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_TZERO_NOM 354'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_I-TRIM2_NOM 313'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_I-TRIM1_NOM 313'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_I+TRIM2_NOM 253'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_I+TRIM1_NOM 253'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VSW-_NOM 206'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VLG_NOM 288'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VT-_NOM 288'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VT+_NOM 739'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VP3-_NOM 57'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VP12-_NOM 360'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VP3+_NOM 976'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VP12+_NOM 770'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VS-_NOM 174'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VS+_NOM 716'" 08:50:57 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VR_NOM 714'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VRD2_NOM 833'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VRD1_NOM 833'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VDD2_NOM 882'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1R1_VDD1_NOM 882'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b''" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'# CAMERA 2 ZB CAM# 2 '" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_LN2TEMP_NOM 584'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_HEATERV_NOM NONE THIS CAM'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_CCDTEMP_NOM 256'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_TSET_NOM 509'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_TZERO_NOM 290'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_I-TRIM2_NOM 626'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_I-TRIM1_NOM 626'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_I+TRIM2_NOM 253'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_I+TRIM1_NOM 274'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VSW-_NOM 360'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VLG_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VT-_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VT+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VP3-_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VP12-_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VP3+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VP12+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VS-_NOM 360'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VS+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VR_NOM 714'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VRD2_NOM 841'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VRD1_NOM 841'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VDD2_NOM 929'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B2_VDD1_NOM 929'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b''" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'# CAMERA 3 ZB CAM# 3 '" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_LN2TEMP_NOM NONE THIS CAM'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_HEATERV_NOM 768'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_CCDTEMP_NOM NONE THIS CAM'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_TSET_NOM 509'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_TZERO_NOM 358'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_I-TRIM2_NOM 626'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_I-TRIM1_NOM 626'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_I+TRIM2_NOM 270'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_I+TRIM1_NOM 274'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VSW-_NOM 360'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VLG_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VT-_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VT+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VP3-_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VP12-_NOM 309'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VP3+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VP12+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VS-_NOM 360'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VS+_NOM 821'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VR_NOM 714'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VRD2_NOM 841'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VRD1_NOM 841'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VDD2_NOM 929'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1B3_VDD1_NOM 929'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b''" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_LOWVS+_NOM 520'" 08:50:58 APO.doravetz 124 boss i sp1camRawText="b'SP1R0_VPERASE_NOM 19 ok'" 08:50:58 APO.doravetz 124 boss : 08:50:58 CmdDone: APO.doravetz 124 boss sp1cam raw=shownomvolts 08:51:09 CmdStarted: APO.doravetz 125 boss sp1cam raw=showreadvolts 08:51:09 APO.doravetz 125 boss i text="Sending raw command SHOWREADVOLTS to sp1cam" 08:51:09 APO.doravetz 125 boss d text="Sending SHOWREADVOLTS to sp1cam" 08:51:12 APO.doravetz 125 boss d text="Completed message from sp1cam" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'2NDARY_DEWAR_PRESS 10 # PSI GAUGE'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'RED_ION_PUMP_LOGP -6.51'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'BLU_ION_PUMP_LOGP -6.24'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'# CAMERA 0 ZB CAM# 0 '" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_LN2TEMP_READ 538'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_HEATERV_READ NONE THIS CAM'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_CCDTEMP_READ 370*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_TSET_READ 558*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_TZERO_READ 414*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_I-TRIM2_READ 394*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_I-TRIM1_READ 397*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_I+TRIM2_READ 387*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_I+TRIM1_READ 388*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VSW-_READ 390*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VLG_READ 396*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VT-_READ 422*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VT+_READ 666*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VP3-_READ 431*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VP12-_READ 428*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VP3+_READ 669*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VP12+_READ 670*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VS-_READ 398*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VS+_READ 518'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VR_READ 646*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VRD2_READ 690*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VRD1_READ 691*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VDD2_READ 698*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R0_VDD1_READ 699*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b''" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'# CAMERA 1 ZB CAM# 1 '" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_LN2TEMP_READ NONE THIS CAM'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_HEATERV_READ 642'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_CCDTEMP_READ NONE THIS CAM'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_TSET_READ 546*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_TZERO_READ 412*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_I-TRIM2_READ 401*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_I-TRIM1_READ 399*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_I+TRIM2_READ 390*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_I+TRIM1_READ 392*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VSW-_READ 391*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VLG_READ 399*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VT-_READ 399*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VT+_READ 656*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VP3-_READ 397*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VP12-_READ 425*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VP3+_READ 687*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VP12+_READ 669*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VS-_READ 397*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VS+_READ 517'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VR_READ 658*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VRD2_READ 693*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VRD1_READ 693*'"08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VDD2_READ 699*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1R1_VDD1_READ 702*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b''" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'# CAMERA 2 ZB CAM# 2 '" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_LN2TEMP_READ 540'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_HEATERV_READ NONE THIS CAM'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_CCDTEMP_READ 390*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_TSET_READ 513'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_TZERO_READ 393*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_I-TRIM2_READ 583*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_I-TRIM1_READ 587*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_I+TRIM2_READ 384*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_I+TRIM1_READ 388*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VSW-_READ 412*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VLG_READ 398*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VT-_READ 398*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VT+_READ 686*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VP3-_READ 407*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VP12-_READ 403*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VP3+_READ 688*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VP12+_READ 690*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VS-_READ 417*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VS+_READ 519'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VR_READ 640*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VRD2_READ 708*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VRD1_READ 715*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VDD2_READ 729*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B2_VDD1_READ 733*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b''" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'# CAMERA 3 ZB CAM# 3 '" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_LN2TEMP_READ NONE THIS CAM'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_HEATERV_READ 736'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_CCDTEMP_READ NONE THIS CAM'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_TSET_READ 512'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_TZERO_READ 397*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_I-TRIM2_READ 602*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_I-TRIM1_READ 603*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_I+TRIM2_READ 371*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_I+TRIM1_READ 385*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VSW-_READ 409*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VLG_READ 395*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VT-_READ 396*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VT+_READ 685*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VP3-_READ 407*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VP12-_READ 402*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VP3+_READ 685*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VP12+_READ 687*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VS-_READ 418*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VS+_READ 519'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VR_READ 638*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VRD2_READ 706*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VRD1_READ 707*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VDD2_READ 722*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b'SP1B3_VDD1_READ 753*'" 08:51:12 APO.doravetz 125 boss i sp1camRawText="b' ok'" 08:51:12 APO.doravetz 125 boss : 08:51:12 CmdDone: APO.doravetz 125 boss sp1cam raw=showreadvolts I went down to the telescope to have a look at the camera power supplies. First, the +95, -30, and PWR OK LEDs were all lit. Then I toggled the dials and stepped-through all the other CCD, reference, etc., supplies. All LEDs were green there and I didn't see anything obviously bad. When I returned to the control room there were no longer any voltage warnings and the critical alerts were now reporting 'Ok' as well, although there were a bunch of 'alerts status' errors that required a restart of alertsActor. Despite the alertsActor needing a restart, camCheck stopped reporting any voltage warnings, so for clarity's sake, restarting alertsActor wasn't the solution here, it just sometimes gets into this weird state after alerts and needs a restart. Maybe it's purely coincidental, but is there any reason why physically dialing through the power supplies to check their status could have kicked things back into reporting voltages that were now within tolerance? I know it's quite the stretch, but seeing as how that's the only thing I did between there being a bunch of voltage warnings and then none at all it seemed worthy of posing the question. Bizarre and unexpected, but at least things were back to normal. I took a bias and it also looked fine with normal SOS-reported PERCENTILE98 values. And no more high-read-noise errors from expNum 370072 onwards.