APACHE POINT OBSERVATORY SDSS 2.5M OBSERVING LOG Friday August 28, 2026 (SJD 61281) ---- Observing Team ---- Dan Oravetz ---- Observing Plan ---- Fri 28 Aug 2026 61281 bright (9.9 h, w/ 1.2 h for 8 deg design) ---- Observing Summary ---- Clear for much of the night, but there were a few clouds to battle early in the night and the first 0.4 hours was lost waiting for the remnant threatening clouds from the afternoon storms to clear. Observed from the queue the rest of the night after that brief delay. Time Field-Design-Conf DesignType BOSS/APOGEE Seeing Overhead Complete? Comment 02:23:55 100537-752741-27448 bright_time 1x730/1xAB 1.4" 0:00 DONE 02:43:18 100497-752544-27449 bright_time 1x730/1xAB 1.2" 6:41 DONE 03:02:09 100567-752915-27450 bright_time 1x730/1xAB 1.9" 6:09 DONE 03:20:30 100548-752791-27451 bright_time 3x730/3xAB 2.1" 5:44 DONE; heavier clouds 04:05:34 100454-752267-27452 bright_time 1x730/1xAB 1.3" 5:39 DONE 04:52:46 102435-754592-27453 bright_time 1x730/1xAB 1.8" 36:00 DONE; see Note (1) 05:11:41 100467-752381-27454 bright_time 1x730/1xAB 1.4" 6:08 DONE 05:29:59 100488-752514-27455 bright_time 1x730/1xAB 1.4" 5:36 DONE 05:57:03 100955-753239-27457 bright_time 1x730/1xAB 1.7" 15:39 DONE; see Note (2) 06:15:36 100450-752224-27458 bright_time 1x730/1xAB 1.8" 5:51 DONE 06:33:38 100990-753363-27459 bright_time 1x730/1xAB 2.0" 6:00 DONE 06:51:50 102403-754407-27460 bright_time 1x730/1xAB 1.9" 5:42 DONE 07:09:43 100959-753304-27461 bright_time 1x730/1xAB 2.0" 5:39 DONE 07:27:32 102403-754417-27462 bright_time 1x730/1xAB 1.8" 5:42 DONE 07:45:25 104435-756519-27463 bright_time 1x730/1xAB 2.2" 6:14 DONE 08:03:51 100986-753359-27464 bright_time 1x730/1xAB 2.1" 5:54 DONE 08:21:56 100977-753346-27465 bright_time 1x730/1xAB 2.7" 5:31 DONE 08:39:38 104455-756588-27466 bright_time 1x730/1xAB 2.7" 6:17 DONE 08:58:05 104481-756625-27467 bright_time 1x730/1xAB 3.0" 6:18 DONE 09:16:35 102418-754523-27468 bright_time 1x730/1xAB 2.5" 6:10 DONE 09:34:56 102458-754641-27469 bright_time 1x730/1xAB 2.4" 6:21 DONE 09:53:27 100631-753084-27470 bright_time 1x730/1xAB 2.6" 5:47 DONE 10:11:22 102473-754669-27471 bright_time 1x730/1xAB 2.5" 6:04 DONE 10:29:40 102502-754730-27472 bright_time 1x730/1xAB 2.5" 5:28 DONE 10:47:18 100681-753138-27473 bright_time 1x730/1xAB 2.1" 6:01 DONE 11:05:31 100679-753135-27474 bright_time 1x730/1xAB 2.7" 5:53 DONE Notes ===== (1) The 'jaeger preload' attempts timed-out for field 10050, design 752802. Turned out cherno had crashed and then needed to be restarted and also jaeger required a restart as well. Details in the Software Issues section and Problems in Detail section. (0.5 hours lost total including the 10 minutes for the two timeouts) (2) After restarting jaeger I had moved on to the next field before re-disabling 1339 and 885. Both positioners behaved on the first design post-jaeger restart, but then 1339 complained of a collision and it didn't want to move. However, moving the rotator 180 degrees seemed enough to coax it into moving enough that I could fold the array and then disable 1339 (and 885) in the folded state. Can we please move 885 and 1339 from the manually_disabled list to the permanently_disabled list? These two have been on the manually_disabled list for quite some time now. ---- Software Issues ---- (1) Two attempts at 'jaeger preload' timed-out around 04:10Z-04:20Z. They timed-out because the 'preload' calls 'cherno get-scale' and that 'get-scale' command never returned within cherno, so the 'preload' timed-out after the preset 5 minutes. As it happened, cherno was altogether unresponsive and required a restart. (Details in Problems Section -- there were two dozen chernoActor 'start' processes on hub!) After restarting cherno, the 'preload' worked beyond the 'get-scale' command, but it wouldn't actually successfully complete because there was an error message that there was already a 'jaeger preload' running (despite it already having completed with a CmdDone "timeout" message). That led to the need to restart jaeger. FYI, I think there's a ticket about the 'jaeger preload' not working after a timeout and requires a whole restart of jaeger, but jira.sdss.org "sent back an empty page" and I could not connect to check on tickets. (2) There were several actors/services that were unresponsive/disconnected upon arrival and starting checkout this afternoon. (3) There were no APOGEE Quickred plots populating on the http://sdss5-hub.apo.nmsu.edu/data/apogee/quickred/61281/61281.html page tonight; just a blank page. (4) I encountered some Gateway Timeout errors when I was trying to get the exposure summaries to populate the LUI data sections. ---- Observing Log ---- ============================================================================ AFTERNOON CHECKOUT ============================================================================ Several software issues upon arrival: (1) alertsActor requiring a restart; (2) TCC axisConnState was stuck in a "Disconnecting,Disconnecting,Disconnecting" state and required a 'tcc axis connect'; and (3) Grafana was hanging and when it did finally time out it produced a "failed to load application files" error page. Restarted cherno. Disk space: Filesystem Size Used Avail Use% Mounted on /dev/sdb1 1.0T 420G 604G 42% /home /dev/sdd1 22T 15T 6.2T 71% /data FPS powered on. Positioners are home/folded: 22:14:44 APO.doravetz 120 jaeger i folded=T 22:14:44 APO.doravetz 120 jaeger : text="All positioners are folded." 22:15:05 APO.doravetz 121 jaeger : permanently_disabled=463,463,463,647,986; manually_disabled=885,1339 Loaded a manual folded configuration: 22:15:58 CmdStarted: APO.doravetz 123 jaeger configuration load --from-positions 22:16:01 APO.doravetz 123 jaeger d text="Resetting cherno offsets." 22:16:01 APO.doravetz 123 jaeger d text="Focal plane scale: 1.0." 22:16:01 APO.doravetz 123 jaeger d configuration_loaded=27447,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/027XXX/0274XX/confSummary-27447.par,F 22:16:07 APO.doravetz 123 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/61281/configuration_snapshot_61281_27447.pdf 22:16:07 APO.doravetz 123 jaeger d design_preloaded=-999 22:16:07 APO.doravetz 123 jaeger : text="Configuration 27447 loaded." 22:16:07 CmdDone: APO.doravetz 123 jaeger configuration load --from-positions Took a '--no-apply' FVC image: 22:17:08 FVC Loop fvc_rms= 34.21 fvc_deltarms= 26.72 fvc_filename=/data/fcam/61281/fimg-fvc1n-0001.fits fvc_perc_90= 47.30 fvc_percent_reached= 4.60 fvc iterations= 1 Scattered-light check: 22:17:35 CmdStarted: APO.doravetz 125 fliswarm talk -n fvc expose 50 No evidence of light artifacts. GFA biases, darks, and flat: 22:26:54 CmdStarted: APO.doravetz 127 fliswarm talk -c gfa expose --bias --count 10 22:28:56 CmdStarted: APO.doravetz 128 fliswarm talk -c gfa expose --dark --count 5 5 22:31:52 CmdStarted: APO.doravetz 130 fliswarm talk -c gfa expose --flat 1 Exercised positioners: 22:32:20 CmdStarted: APO.doravetz 132 jaeger configuration random --danger 22:34:20 APO.doravetz 132 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/61281/trajectory-61281-0001.json 22:34:20 CmdDone: APO.doravetz 132 jaeger configuration random --danger 22:34:25 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/61280/fps_snapshot_61280_0001.pdf 22:35:14 CmdStarted: APO.doravetz 133 jaeger configuration reverse BOSS hartmanns, arc, flat: 22:47:21 BOSS expNum=403926, expTime= 4.0s 22:48:01 BOSS expNum=403927, expTime= 4.0s 22:48:42 sp1.motor.move= 24, 23, 24 22:48:42 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -2.2 (In Focus) sp1: pred. move: spAverageMove= 24 sp1: pred. resid: r= -683, b= -2.1, txt= OK, spTemp = 10.3 22:49:00 BOSS expNum=403928, expTime= 4.0s 22:50:34 BOSS expNum=403929, expTime= 25.0s Ventilation delayed due to remnant thunderstorm clouds. ============================================================================ NIGHT ============================================================================ 02:03:51 2.5m Enclosure: OPEN. Force-filled BOSS LN2 dewars. Mostly clear. APOGEE dome flats: 02:06:46 APOGEE expNum=57190011, reads=5, dithPos=B, DomeFlat 02:08:01 APOGEE expNum=57190012, reads=5, dithPos=B, DomeFlat 02:09:17 APOGEE expNum=57190013, reads=5, dithPos=B, DomeFlat MJD DomeFlat MTPs:1 2 3 4 5 6 7 8 9 10 AZ ALT ROT 61281 57190011 0.97 1.00 0.99 1.00 1.02 1.03 0.97 1.01 1.01 0.95 121.0 30.0 0.0 61281 57190012 0.98 1.00 0.99 1.00 1.02 1.03 0.97 1.02 1.01 0.94 121.0 30.0 0.0 61281 57190013 0.99 0.99 0.99 1.00 1.02 1.03 0.97 1.01 1.01 0.94 121.0 30.0 0.0 BOSS hartmanns: 02:14:59 BOSS expNum=403932, expTime= 4.0s 02:15:39 BOSS expNum=403933, expTime= 4.0s 02:16:20 sp1.motor.move= 184, 184, 183 02:16:20 Hartmann collimate output sp1: offset: r= -659 (In Focus); b= -3.2 (In Focus) sp1: pred. move: spAverageMove= 184 sp1: pred. resid: r= -843, b= -2.6, txt= OK, spTemp = 10.5 BOSS flats: 02:16:45 BOSS expNum=403934, expTime= 25.0s 02:18:32 BOSS expNum=403935, expTime= 25.0s 02:20:20 BOSS expNum=403936, expTime= 25.0s **************************************************************** 02:23:55 Field-Design-Configuration: 100537-752741-27448 **************************************************************** 02:24:04 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 02:24:04 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 02:24:06 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 02:24:08 CmdStarted: jaeger configuration execute 02:24:12 CmdStarted: tcc track (slew begins) 02:24:16 HgCd lamps turned ON 02:25:30 CmdDone: jaeger configuration execute 02:26:16 CmdDone: Slew finished ('tcc track' command finished) 02:26:16 CmdStarted: FVC loop 02:27:16 CmdDone: FVC loop FVC loop results: fvc_rms= 11.9 fvc_deltarms= 25.37 fvc_filename=/data/fcam/61281/fimg-fvc1n-0004.fits fvc_perc_90= 9.60 fvc_percent_reached= 63.60 fvc iterations= 2 02:27:16 CmdStarted: tcc track (slew begins) 02:27:17 Ne lamps turned ON 02:27:24 CmdDone: Slew finished ('tcc track' command finished) 02:27:58 BOSS expNum=403937, expTime= 4.0s 02:28:06 HgCd lamps turned OFF 02:28:06 Ne lamps turned OFF 02:28:15 CmdStarted: cherno acquire 02:30:16 CmdDone: cherno acquire 02:30:17 CmdStarted: 'cherno guide' 02:30:17 hal expose: BOSS science, APOGEE science 02:30:47 BOSS expNum=403938, expTime= 730.0s 02:30:49 APOGEE expNum=57190014, reads=35, dithPos=A, Object 02:37:05 APOGEE expNum=57190015, reads=35, dithPos=B, Object 02:43:16 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 02:43:18 Field-Design-Configuration: 100497-752544-27449 **************************************************************** 02:43:26 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 02:43:27 CmdStarted: jaeger configuration reverse 02:43:31 CmdStarted: tcc track (slew begins) 02:43:36 HgCd lamps turned ON 02:43:53 CmdDone: Slew finished ('tcc track' command finished) 02:44:52 CmdDone: jaeger configuration reverse 02:44:52 CmdStarted: jaeger configuration execute 02:45:56 CmdDone: jaeger configuration execute 02:45:57 CmdStarted: FVC loop 02:46:57 CmdDone: FVC loop FVC loop results: fvc_rms= 15.52 fvc_deltarms= 22.69 fvc_filename=/data/fcam/61281/fimg-fvc1n-0006.fits fvc_perc_90= 9.50 fvc_percent_reached= 63.30 fvc iterations= 2 02:46:57 CmdStarted: tcc track (slew begins) 02:46:57 Ne lamps turned ON 02:47:07 CmdDone: Slew finished ('tcc track' command finished) 02:47:42 BOSS expNum=403939, expTime= 4.0s 02:47:50 HgCd lamps turned OFF 02:47:50 Ne lamps turned OFF 02:48:00 CmdStarted: cherno acquire 02:49:07 CmdDone: cherno acquire 02:49:08 CmdStarted: 'cherno guide' 02:49:08 hal expose: BOSS science, APOGEE science 02:49:39 BOSS expNum=403940, expTime= 730.0s 02:49:40 APOGEE expNum=57190016, reads=35, dithPos=A, Object 02:55:55 APOGEE expNum=57190017, reads=35, dithPos=B, Object 03:02:06 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 03:02:08 Field-Design-Configuration: 100567-752915-27450 **************************************************************** 03:02:16 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 03:02:18 CmdStarted: jaeger configuration reverse 03:02:22 CmdStarted: tcc track (slew begins) 03:02:26 HgCd lamps turned ON 03:02:55 CmdDone: Slew finished ('tcc track' command finished) 03:03:24 CmdDone: jaeger configuration reverse 03:03:24 CmdStarted: jaeger configuration execute 03:04:16 CmdDone: jaeger configuration execute 03:04:16 CmdStarted: FVC loop 03:05:18 CmdDone: FVC loop FVC loop results: fvc_rms= 6.81 fvc_deltarms= 32.64 fvc_filename=/data/fcam/61281/fimg-fvc1n-0008.fits fvc_perc_90= 8.20 fvc_percent_reached= 69.00 fvc iterations= 2 03:05:19 CmdStarted: tcc track (slew begins) 03:05:19 Ne lamps turned ON 03:05:27 CmdDone: Slew finished ('tcc track' command finished) 03:06:04 BOSS expNum=403941, expTime= 4.0s 03:06:12 HgCd lamps turned OFF 03:06:12 Ne lamps turned OFF 03:06:21 CmdStarted: cherno acquire 03:07:28 CmdDone: cherno acquire 03:07:29 CmdStarted: 'cherno guide' 03:07:30 hal expose: BOSS science, APOGEE science 03:07:59 BOSS expNum=403942, expTime= 730.0s 03:08:01 APOGEE expNum=57190018, reads=35, dithPos=A, Object 03:14:17 APOGEE expNum=57190019, reads=35, dithPos=B, Object 03:20:28 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 03:20:30 Field-Design-Configuration: 100548-752791-27451 **************************************************************** 03:20:36 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 03:20:37 CmdStarted: jaeger configuration reverse 03:20:41 CmdStarted: tcc track (slew begins) 03:20:46 HgCd lamps turned ON 03:20:58 CmdDone: Slew finished ('tcc track' command finished) 03:21:29 CmdDone: jaeger configuration reverse 03:21:29 CmdStarted: jaeger configuration execute 03:22:20 CmdDone: jaeger configuration execute 03:22:20 CmdStarted: FVC loop 03:23:22 CmdDone: FVC loop FVC loop results: fvc_rms= 6.27 fvc_deltarms= 30.33 fvc_filename=/data/fcam/61281/fimg-fvc1n-0010.fits fvc_perc_90= 9.40 fvc_percent_reached= 64.00 fvc iterations= 2 03:23:22 CmdStarted: tcc track (slew begins) 03:23:23 Ne lamps turned ON 03:23:31 CmdDone: Slew finished ('tcc track' command finished) 03:24:05 BOSS expNum=403943, expTime= 4.0s 03:24:13 HgCd lamps turned OFF 03:24:13 Ne lamps turned OFF 03:24:23 CmdStarted: cherno acquire 03:25:23 CmdDone: cherno acquire 03:25:24 CmdStarted: 'cherno guide' 03:25:24 hal expose: BOSS science, APOGEE science 03:25:54 BOSS expNum=403944, expTime= 730.0s 03:25:55 APOGEE expNum=57190020, reads=35, dithPos=A, Object 03:32:12 APOGEE expNum=57190021, reads=35, dithPos=B, Object 03:38:25 APOGEE expNum=57190022, reads=35, dithPos=B, Object 03:39:31 BOSS expNum=403945, expTime= 730.0s 03:44:45 APOGEE expNum=57190023, reads=35, dithPos=A, Object 03:50:58 APOGEE expNum=57190024, reads=35, dithPos=A, Object 03:53:08 BOSS expNum=403946, expTime= 730.0s 03:57:13 APOGEE expNum=57190025, reads=35, dithPos=B, Object 04:05:32 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 04:05:34 Field-Design-Configuration: 100454-752267-27452 **************************************************************** 04:05:40 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 04:05:41 CmdStarted: jaeger configuration reverse 04:05:46 CmdStarted: tcc track (slew begins) 04:05:50 HgCd lamps turned ON 04:06:27 CmdDone: Slew finished ('tcc track' command finished) 04:06:33 CmdDone: jaeger configuration reverse 04:06:33 CmdStarted: jaeger configuration execute 04:07:23 CmdDone: jaeger configuration execute 04:07:24 CmdStarted: FVC loop 04:08:24 CmdDone: FVC loop FVC loop results: fvc_rms= 10.8 fvc_deltarms= 28.13 fvc_filename=/data/fcam/61281/fimg-fvc1n-0012.fits fvc_perc_90= 9.00 fvc_percent_reached= 66.70 fvc iterations= 2 04:08:24 CmdStarted: tcc track (slew begins) 04:08:25 Ne lamps turned ON 04:08:34 CmdDone: Slew finished ('tcc track' command finished) 04:09:06 BOSS expNum=403947, expTime= 4.0s 04:09:14 HgCd lamps turned OFF 04:09:14 Ne lamps turned OFF 04:09:23 CmdStarted: cherno acquire 04:10:27 CmdDone: cherno acquire 04:10:28 CmdStarted: 'cherno guide' 04:10:29 hal expose: BOSS science, APOGEE science 04:10:58 BOSS expNum=403948, expTime= 730.0s 04:11:00 APOGEE expNum=57190026, reads=35, dithPos=A, Object 04:17:15 APOGEE expNum=57190027, reads=35, dithPos=B, Object 04:23:27 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup The preload timed-out for field 10050, design 752802. I re-tried, but when that timed-out I skipped it and moved on. 04:18:19 CmdStarted: jaeger.jaeger 1833 cherno get-scale --max-age 3600 04:37:53 CmdStarted: jaeger configuration reverse 04:38:40 CmdDone: jaeger configuration reverse **************************************************************** 04:52:46 Field-Design-Configuration: 102435-754592-27453 **************************************************************** 04:52:56 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 04:52:56 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 04:52:58 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 04:52:59 CmdStarted: jaeger configuration execute 04:53:03 CmdStarted: tcc track (slew begins) 04:53:08 HgCd lamps turned ON 04:53:49 CmdDone: jaeger configuration execute 04:54:57 CmdDone: Slew finished ('tcc track' command finished) 04:54:57 CmdStarted: FVC loop 04:55:57 CmdDone: FVC loop FVC loop results: fvc_rms= 11.99 fvc_deltarms= 23.08 fvc_filename=/data/fcam/61281/fimg-fvc1n-0014.fits fvc_perc_90= 8.20 fvc_percent_reached= 71.10 fvc iterations= 2 04:55:57 CmdStarted: tcc track (slew begins) 04:55:58 Ne lamps turned ON 04:56:06 CmdDone: Slew finished ('tcc track' command finished) 04:56:40 BOSS expNum=403949, expTime= 4.0s 04:56:48 HgCd lamps turned OFF 04:56:48 Ne lamps turned OFF 04:57:00 CmdStarted: cherno acquire 04:58:39 CmdDone: cherno acquire 04:58:40 CmdStarted: 'cherno guide' 04:58:40 hal expose: BOSS science, APOGEE science 04:59:09 BOSS expNum=403950, expTime= 730.0s 04:59:11 APOGEE expNum=57190028, reads=35, dithPos=A, Object 05:05:27 APOGEE expNum=57190029, reads=35, dithPos=B, Object 05:11:39 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 05:11:41 Field-Design-Configuration: 100467-752381-27454 **************************************************************** 05:11:46 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 05:11:48 CmdStarted: jaeger configuration reverse 05:11:52 CmdStarted: tcc track (slew begins) 05:11:57 HgCd lamps turned ON 05:12:39 CmdDone: jaeger configuration reverse 05:12:39 CmdStarted: jaeger configuration execute 05:13:27 CmdDone: jaeger configuration execute 05:13:31 CmdDone: Slew finished ('tcc track' command finished) 05:13:31 CmdStarted: FVC loop 05:14:33 CmdDone: FVC loop FVC loop results: fvc_rms= 8.94 fvc_deltarms= 28.55 fvc_filename=/data/fcam/61281/fimg-fvc1n-0016.fits fvc_perc_90= 9.10 fvc_percent_reached= 66.20 fvc iterations= 2 05:14:33 CmdStarted: tcc track (slew begins) 05:14:33 Ne lamps turned ON 05:14:40 CmdDone: Slew finished ('tcc track' command finished) 05:15:15 BOSS expNum=403951, expTime= 4.0s 05:15:23 HgCd lamps turned OFF 05:15:23 Ne lamps turned OFF 05:15:33 CmdStarted: cherno acquire 05:16:58 CmdDone: cherno acquire 05:16:59 CmdStarted: 'cherno guide' 05:16:59 hal expose: BOSS science, APOGEE science 05:17:28 BOSS expNum=403952, expTime= 730.0s 05:17:30 APOGEE expNum=57190030, reads=35, dithPos=A, Object 05:23:46 APOGEE expNum=57190031, reads=35, dithPos=B, Object 05:29:57 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 05:29:59 Field-Design-Configuration: 100488-752514-27455 **************************************************************** 05:30:05 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 05:30:07 CmdStarted: jaeger configuration reverse 05:30:11 CmdStarted: tcc track (slew begins) 05:30:16 HgCd lamps turned ON 05:30:30 CmdDone: Slew finished ('tcc track' command finished) 05:30:56 CmdDone: jaeger configuration reverse 05:30:56 CmdStarted: jaeger configuration execute 05:31:40 CmdDone: jaeger configuration execute 05:31:41 CmdStarted: FVC loop 05:32:41 CmdDone: FVC loop FVC loop results: fvc_rms= 5.9 fvc_deltarms= 36.27 fvc_filename=/data/fcam/61281/fimg-fvc1n-0018.fits fvc_perc_90= 8.40 fvc_percent_reached= 67.50 fvc iterations= 2 05:32:41 CmdStarted: tcc track (slew begins) 05:32:42 Ne lamps turned ON 05:32:49 CmdDone: Slew finished ('tcc track' command finished) 05:33:23 BOSS expNum=403953, expTime= 4.0s 05:33:31 HgCd lamps turned OFF 05:33:31 Ne lamps turned OFF 05:33:41 CmdStarted: cherno acquire 05:34:44 CmdDone: cherno acquire 05:34:45 CmdStarted: 'cherno guide' 05:34:46 hal expose: BOSS science, APOGEE science 05:35:15 BOSS expNum=403954, expTime= 730.0s 05:35:17 APOGEE expNum=57190032, reads=35, dithPos=A, Object 05:41:33 APOGEE expNum=57190033, reads=35, dithPos=B, Object 05:47:44 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 05:47:46 Field-Design-Configuration: 100478-752445-27456 **************************************************************** 05:47:52 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 05:47:53 CmdStarted: jaeger configuration reverse 05:47:58 CmdStarted: tcc track (slew begins) 05:48:03 HgCd lamps turned ON 05:48:07 Jaeger collision: 1339 05:48:09 CmdDone: jaeger configuration reverse 05:48:09 HgCd lamps turned OFF 05:48:09 Ne lamps turned OFF 05:48:13 CmdDone: Slew finished ('tcc track' command finished) 05:48:07 .jaeger 0 jaeger e text="A collision was detected in positioner 1339. Sending SEND_TRAJECTORY_ABORT and locking the FPS." 05:48:07 .jaeger 0 jaeger w text="Locking the FPS. (JaegerUserWarning)" 05:48:07 .jaeger 0 jaeger i fps_status=0x14 05:48:08 .jaeger 0 jaeger e locked=T; locked_by=1339; locked_axes=beta; locked_alpha=133.84994387626648; locked_beta=20.872591137886047 Positioner 1339 was eventually re-disabled, but it took moving the ROT about 180 deg from its initial location and then the 'explode' finally worked without a collision complaint from 1339, after which the 'unwind' succeeded and I re-disabled 1339 (and 885). **************************************************************** 05:57:03 Field-Design-Configuration: 100955-753239-27457 **************************************************************** 05:57:08 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 05:57:08 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup 05:57:10 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 05:57:12 CmdStarted: jaeger configuration execute 05:57:16 CmdStarted: tcc track (slew begins) 05:57:21 HgCd lamps turned ON 05:58:00 CmdDone: jaeger configuration execute 05:58:48 CmdDone: Slew finished ('tcc track' command finished) 05:58:49 CmdStarted: FVC loop 05:59:51 CmdDone: FVC loop FVC loop results: fvc_rms= 10.07 fvc_deltarms= 27.46 fvc_filename=/data/fcam/61281/fimg-fvc1n-0020.fits fvc_perc_90= 9.20 fvc_percent_reached= 65.30 fvc iterations= 2 05:59:51 CmdStarted: tcc track (slew begins) 05:59:51 Ne lamps turned ON 05:59:57 CmdDone: Slew finished ('tcc track' command finished) 06:00:32 BOSS expNum=403955, expTime= 4.0s 06:00:40 HgCd lamps turned OFF 06:00:40 Ne lamps turned OFF 06:00:50 CmdStarted: cherno acquire 06:02:34 CmdDone: cherno acquire 06:02:35 CmdStarted: 'cherno guide' 06:02:36 hal expose: BOSS science, APOGEE science 06:03:05 BOSS expNum=403956, expTime= 730.0s 06:03:07 APOGEE expNum=57190034, reads=35, dithPos=A, Object 06:09:23 APOGEE expNum=57190035, reads=35, dithPos=B, Object 06:15:34 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 06:15:36 Field-Design-Configuration: 100450-752224-27458 **************************************************************** 06:15:42 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 06:15:43 CmdStarted: jaeger configuration reverse 06:15:48 CmdStarted: tcc track (slew begins) 06:15:52 HgCd lamps turned ON 06:16:07 CmdDone: Slew finished ('tcc track' command finished) 06:16:32 CmdDone: jaeger configuration reverse 06:16:32 CmdStarted: jaeger configuration execute 06:17:22 CmdDone: jaeger configuration execute 06:17:22 CmdStarted: FVC loop 06:18:24 CmdDone: FVC loop FVC loop results: fvc_rms= 7.12 fvc_deltarms= 34.92 fvc_filename=/data/fcam/61281/fimg-fvc1n-0022.fits fvc_perc_90= 10.00 fvc_percent_reached= 57.80 fvc iterations= 2 06:18:24 CmdStarted: tcc track (slew begins) 06:18:25 Ne lamps turned ON 06:18:30 CmdDone: Slew finished ('tcc track' command finished) 06:19:07 BOSS expNum=403957, expTime= 4.0s 06:19:15 HgCd lamps turned OFF 06:19:15 Ne lamps turned OFF 06:19:24 CmdStarted: cherno acquire 06:20:37 CmdDone: cherno acquire 06:20:38 CmdStarted: 'cherno guide' 06:20:38 hal expose: BOSS science, APOGEE science 06:21:07 BOSS expNum=403958, expTime= 730.0s 06:21:10 APOGEE expNum=57190036, reads=35, dithPos=A, Object 06:27:25 APOGEE expNum=57190037, reads=35, dithPos=B, Object 06:33:36 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 06:33:38 Field-Design-Configuration: 100990-753363-27459 **************************************************************** 06:33:44 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 06:33:45 CmdStarted: jaeger configuration reverse 06:33:50 CmdStarted: tcc track (slew begins) 06:33:54 HgCd lamps turned ON 06:34:21 CmdDone: Slew finished ('tcc track' command finished) 06:34:35 CmdDone: jaeger configuration reverse 06:34:35 CmdStarted: jaeger configuration execute 06:35:24 CmdDone: jaeger configuration execute 06:35:24 CmdStarted: FVC loop 06:36:25 CmdDone: FVC loop FVC loop results: fvc_rms= 8.47 fvc_deltarms= 30.19 fvc_filename=/data/fcam/61281/fimg-fvc1n-0024.fits fvc_perc_90= 10.10 fvc_percent_reached= 60.90 fvc iterations= 2 06:36:25 CmdStarted: tcc track (slew begins) 06:36:25 Ne lamps turned ON 06:36:32 CmdDone: Slew finished ('tcc track' command finished) 06:37:09 BOSS expNum=403959, expTime= 4.0s 06:37:17 HgCd lamps turned OFF 06:37:17 Ne lamps turned OFF 06:37:27 CmdStarted: cherno acquire 06:38:48 CmdDone: cherno acquire 06:38:49 CmdStarted: 'cherno guide' 06:38:49 hal expose: BOSS science, APOGEE science 06:39:18 BOSS expNum=403960, expTime= 730.0s 06:39:20 APOGEE expNum=57190038, reads=35, dithPos=A, Object 06:45:36 APOGEE expNum=57190039, reads=35, dithPos=B, Object 06:51:47 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 06:51:50 Field-Design-Configuration: 102403-754407-27460 **************************************************************** 06:51:55 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 06:51:57 CmdStarted: jaeger configuration reverse 06:52:01 CmdStarted: tcc track (slew begins) 06:52:06 HgCd lamps turned ON 06:52:37 CmdDone: Slew finished ('tcc track' command finished) 06:52:46 CmdDone: jaeger configuration reverse 06:52:46 CmdStarted: jaeger configuration execute 06:53:33 CmdDone: jaeger configuration execute 06:53:33 CmdStarted: FVC loop 06:54:34 CmdDone: FVC loop FVC loop results: fvc_rms= 34.49 fvc_deltarms= 6.55 fvc_filename=/data/fcam/61281/fimg-fvc1n-0026.fits fvc_perc_90= 9.70 fvc_percent_reached= 60.00 fvc iterations= 2 06:54:34 CmdStarted: tcc track (slew begins) 06:54:34 Ne lamps turned ON 06:54:40 CmdDone: Slew finished ('tcc track' command finished) 06:55:17 BOSS expNum=403961, expTime= 4.0s 06:55:26 HgCd lamps turned OFF 06:55:26 Ne lamps turned OFF 06:55:35 CmdStarted: cherno acquire 06:56:41 CmdDone: cherno acquire 06:56:42 CmdStarted: 'cherno guide' 06:56:42 hal expose: BOSS science, APOGEE science 06:57:11 BOSS expNum=403962, expTime= 730.0s 06:57:14 APOGEE expNum=57190040, reads=35, dithPos=A, Object 07:03:29 APOGEE expNum=57190041, reads=35, dithPos=B, Object 07:09:40 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 07:09:43 Field-Design-Configuration: 100959-753304-27461 **************************************************************** 07:09:48 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 07:09:50 CmdStarted: jaeger configuration reverse 07:09:54 CmdStarted: tcc track (slew begins) 07:09:59 HgCd lamps turned ON 07:10:19 CmdDone: Slew finished ('tcc track' command finished) 07:10:37 CmdDone: jaeger configuration reverse 07:10:37 CmdStarted: jaeger configuration execute 07:11:24 CmdDone: jaeger configuration execute 07:11:24 CmdStarted: FVC loop 07:12:27 CmdDone: FVC loop FVC loop results: fvc_rms= 8.69 fvc_deltarms= 31.91 fvc_filename=/data/fcam/61281/fimg-fvc1n-0028.fits fvc_perc_90= 9.60 fvc_percent_reached= 57.80 fvc iterations= 2 07:12:27 CmdStarted: tcc track (slew begins) 07:12:27 Ne lamps turned ON 07:12:33 CmdDone: Slew finished ('tcc track' command finished) 07:13:09 BOSS expNum=403963, expTime= 4.0s 07:13:18 HgCd lamps turned OFF 07:13:18 Ne lamps turned OFF 07:13:27 CmdStarted: cherno acquire 07:14:30 CmdDone: cherno acquire 07:14:31 CmdStarted: 'cherno guide' 07:14:32 hal expose: BOSS science, APOGEE science 07:15:01 BOSS expNum=403964, expTime= 730.0s 07:15:03 APOGEE expNum=57190042, reads=35, dithPos=A, Object 07:21:18 APOGEE expNum=57190043, reads=35, dithPos=B, Object 07:27:30 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 07:27:32 Field-Design-Configuration: 102403-754417-27462 **************************************************************** 07:27:37 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 07:27:39 CmdStarted: jaeger configuration reverse 07:27:43 CmdStarted: tcc track (slew begins) 07:27:48 HgCd lamps turned ON 07:28:08 CmdDone: Slew finished ('tcc track' command finished) 07:28:26 CmdDone: jaeger configuration reverse 07:28:26 CmdStarted: jaeger configuration execute 07:29:26 CmdDone: jaeger configuration execute 07:29:26 CmdStarted: FVC loop 07:30:29 CmdDone: FVC loop FVC loop results: fvc_rms= 30.57 fvc_deltarms= 9.49 fvc_filename=/data/fcam/61281/fimg-fvc1n-0030.fits fvc_perc_90= 9.70 fvc_percent_reached= 61.00 fvc iterations= 2 07:30:30 CmdStarted: tcc track (slew begins) 07:30:30 Ne lamps turned ON 07:30:34 CmdDone: Slew finished ('tcc track' command finished) 07:31:12 BOSS expNum=403965, expTime= 4.0s 07:31:20 HgCd lamps turned OFF 07:31:20 Ne lamps turned OFF 07:31:29 CmdStarted: cherno acquire 07:32:08 CmdDone: cherno acquire 07:32:09 CmdStarted: 'cherno guide' 07:32:24 hal expose: BOSS science, APOGEE science 07:32:54 BOSS expNum=403966, expTime= 730.0s 07:32:55 APOGEE expNum=57190044, reads=35, dithPos=A, Object 07:39:11 APOGEE expNum=57190045, reads=35, dithPos=B, Object 07:45:22 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 07:45:25 Field-Design-Configuration: 104435-756519-27463 **************************************************************** 07:45:30 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 07:45:32 CmdStarted: jaeger configuration reverse 07:45:36 CmdStarted: tcc track (slew begins) 07:45:40 HgCd lamps turned ON 07:46:03 CmdDone: Slew finished ('tcc track' command finished) 07:46:31 CmdDone: jaeger configuration reverse 07:46:31 CmdStarted: jaeger configuration execute 07:47:16 CmdDone: jaeger configuration execute 07:47:16 CmdStarted: FVC loop 07:48:16 CmdDone: FVC loop FVC loop results: fvc_rms= 7.29 fvc_deltarms= 32.18 fvc_filename=/data/fcam/61281/fimg-fvc1n-0032.fits fvc_perc_90= 9.60 fvc_percent_reached= 55.80 fvc iterations= 2 07:48:17 CmdStarted: tcc track (slew begins) 07:48:17 Ne lamps turned ON 07:48:24 CmdDone: Slew finished ('tcc track' command finished) 07:49:00 BOSS expNum=403967, expTime= 4.0s 07:49:08 HgCd lamps turned OFF 07:49:08 Ne lamps turned OFF 07:49:17 CmdStarted: cherno acquire 07:50:48 CmdDone: cherno acquire 07:50:49 CmdStarted: 'cherno guide' 07:50:50 hal expose: BOSS science, APOGEE science 07:51:19 BOSS expNum=403968, expTime= 730.0s 07:51:21 APOGEE expNum=57190046, reads=35, dithPos=A, Object 07:57:37 APOGEE expNum=57190047, reads=35, dithPos=B, Object 08:03:49 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 08:03:51 Field-Design-Configuration: 100986-753359-27464 **************************************************************** 08:03:56 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 08:03:58 CmdStarted: jaeger configuration reverse 08:04:02 CmdStarted: tcc track (slew begins) 08:04:07 HgCd lamps turned ON 08:04:38 CmdDone: Slew finished ('tcc track' command finished) 08:04:43 CmdDone: jaeger configuration reverse 08:04:43 CmdStarted: jaeger configuration execute 08:05:32 CmdDone: jaeger configuration execute 08:05:32 CmdStarted: FVC loop 08:06:33 CmdDone: FVC loop FVC loop results: fvc_rms= 7.53 fvc_deltarms= 33.82 fvc_filename=/data/fcam/61281/fimg-fvc1n-0034.fits fvc_perc_90= 10.00 fvc_percent_reached= 62.00 fvc iterations= 2 08:06:33 CmdStarted: tcc track (slew begins) 08:06:34 Ne lamps turned ON 08:06:40 CmdDone: Slew finished ('tcc track' command finished) 08:07:15 BOSS expNum=403969, expTime= 4.0s 08:07:23 HgCd lamps turned OFF 08:07:23 Ne lamps turned OFF 08:07:32 CmdStarted: cherno acquire 08:08:53 CmdDone: cherno acquire 08:08:54 CmdStarted: 'cherno guide' 08:08:55 hal expose: BOSS science, APOGEE science 08:09:24 BOSS expNum=403970, expTime= 730.0s 08:09:26 APOGEE expNum=57190048, reads=35, dithPos=A, Object 08:15:42 APOGEE expNum=57190049, reads=35, dithPos=B, Object 08:21:53 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 08:21:56 Field-Design-Configuration: 100977-753346-27465 **************************************************************** 08:22:01 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 08:22:03 CmdStarted: jaeger configuration reverse 08:22:07 CmdStarted: tcc track (slew begins) 08:22:12 HgCd lamps turned ON 08:22:19 CmdDone: Slew finished ('tcc track' command finished) 08:22:51 CmdDone: jaeger configuration reverse 08:22:52 CmdStarted: jaeger configuration execute 08:23:40 CmdDone: jaeger configuration execute 08:23:40 CmdStarted: FVC loop 08:24:41 CmdDone: FVC loop FVC loop results: fvc_rms= 7.77 fvc_deltarms= 38.49 fvc_filename=/data/fcam/61281/fimg-fvc1n-0036.fits fvc_perc_90= 10.30 fvc_percent_reached= 55.40 fvc iterations= 2 08:24:41 CmdStarted: tcc track (slew begins) 08:24:41 Ne lamps turned ON 08:24:47 CmdDone: Slew finished ('tcc track' command finished) 08:25:25 BOSS expNum=403971, expTime= 4.0s 08:25:33 HgCd lamps turned OFF 08:25:33 Ne lamps turned OFF 08:25:42 CmdStarted: cherno acquire 08:26:11 CmdDone: cherno acquire 08:26:12 CmdStarted: 'cherno guide' 08:26:37 hal expose: BOSS science, APOGEE science 08:27:06 BOSS expNum=403972, expTime= 730.0s 08:27:09 APOGEE expNum=57190050, reads=35, dithPos=A, Object 08:33:25 APOGEE expNum=57190051, reads=35, dithPos=B, Object 08:39:36 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 08:39:38 Field-Design-Configuration: 104455-756588-27466 **************************************************************** 08:39:44 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 08:39:46 CmdStarted: jaeger configuration reverse 08:39:50 CmdStarted: tcc track (slew begins) 08:39:55 HgCd lamps turned ON 08:40:24 CmdDone: Slew finished ('tcc track' command finished) 08:40:34 CmdDone: jaeger configuration reverse 08:40:34 CmdStarted: jaeger configuration execute 08:41:31 CmdDone: jaeger configuration execute 08:41:31 CmdStarted: FVC loop 08:42:31 CmdDone: FVC loop FVC loop results: fvc_rms= 14.13 fvc_deltarms= 34.07 fvc_filename=/data/fcam/61281/fimg-fvc1n-0038.fits fvc_perc_90= 10.30 fvc_percent_reached= 59.40 fvc iterations= 2 08:42:31 CmdStarted: tcc track (slew begins) 08:42:31 Ne lamps turned ON 08:42:39 CmdDone: Slew finished ('tcc track' command finished) 08:43:13 BOSS expNum=403973, expTime= 4.0s 08:43:21 HgCd lamps turned OFF 08:43:21 Ne lamps turned OFF 08:43:30 CmdStarted: cherno acquire 08:45:03 CmdDone: cherno acquire 08:45:04 CmdStarted: 'cherno guide' 08:45:04 hal expose: BOSS science, APOGEE science 08:45:34 BOSS expNum=403974, expTime= 730.0s 08:45:35 APOGEE expNum=57190052, reads=35, dithPos=A, Object 08:51:52 APOGEE expNum=57190053, reads=35, dithPos=B, Object 08:58:02 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 08:58:05 Field-Design-Configuration: 104481-756625-27467 **************************************************************** 08:58:12 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 08:58:13 CmdStarted: jaeger configuration reverse 08:58:17 CmdStarted: tcc track (slew begins) 08:58:22 HgCd lamps turned ON 08:58:34 CmdDone: Slew finished ('tcc track' command finished) 08:59:09 CmdDone: jaeger configuration reverse 08:59:09 CmdStarted: jaeger configuration execute 08:59:58 CmdDone: jaeger configuration execute 08:59:58 CmdStarted: FVC loop 09:00:59 CmdDone: FVC loop FVC loop results: fvc_rms= 7.91 fvc_deltarms= 30.45 fvc_filename=/data/fcam/61281/fimg-fvc1n-0040.fits fvc_perc_90= 9.80 fvc_percent_reached= 59.00 fvc iterations= 2 09:00:59 CmdStarted: tcc track (slew begins) 09:00:59 Ne lamps turned ON 09:01:06 CmdDone: Slew finished ('tcc track' command finished) 09:01:41 BOSS expNum=403975, expTime= 4.0s 09:01:50 HgCd lamps turned OFF 09:01:50 Ne lamps turned OFF 09:01:59 CmdStarted: cherno acquire 09:03:33 CmdDone: cherno acquire 09:03:34 CmdStarted: 'cherno guide' 09:03:34 hal expose: BOSS science, APOGEE science 09:04:03 BOSS expNum=403976, expTime= 730.0s 09:04:05 APOGEE expNum=57190054, reads=35, dithPos=A, Object 09:10:21 APOGEE expNum=57190055, reads=35, dithPos=B, Object 09:16:33 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 09:16:35 Field-Design-Configuration: 102418-754523-27468 **************************************************************** 09:16:41 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 09:16:43 CmdStarted: jaeger configuration reverse 09:16:47 CmdStarted: tcc track (slew begins) 09:16:51 HgCd lamps turned ON 09:17:09 CmdDone: Slew finished ('tcc track' command finished) 09:17:31 CmdDone: jaeger configuration reverse 09:17:31 CmdStarted: jaeger configuration execute 09:18:22 CmdDone: jaeger configuration execute 09:18:22 CmdStarted: FVC loop 09:19:24 CmdDone: FVC loop FVC loop results: fvc_rms= 8.52 fvc_deltarms= 31.79 fvc_filename=/data/fcam/61281/fimg-fvc1n-0042.fits fvc_perc_90= 9.70 fvc_percent_reached= 56.90 fvc iterations= 2 09:19:24 CmdStarted: tcc track (slew begins) 09:19:25 Ne lamps turned ON 09:19:32 CmdDone: Slew finished ('tcc track' command finished) 09:20:10 BOSS expNum=403977, expTime= 4.0s 09:20:18 HgCd lamps turned OFF 09:20:18 Ne lamps turned OFF 09:20:27 CmdStarted: cherno acquire 09:21:54 CmdDone: cherno acquire 09:21:55 CmdStarted: 'cherno guide' 09:21:55 hal expose: BOSS science, APOGEE science 09:22:24 BOSS expNum=403978, expTime= 730.0s 09:22:26 APOGEE expNum=57190056, reads=35, dithPos=A, Object 09:28:42 APOGEE expNum=57190057, reads=35, dithPos=B, Object 09:34:54 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 09:34:56 Field-Design-Configuration: 102458-754641-27469 **************************************************************** 09:35:02 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 09:35:03 CmdStarted: jaeger configuration reverse 09:35:08 CmdStarted: tcc track (slew begins) 09:35:12 HgCd lamps turned ON 09:35:36 CmdDone: Slew finished ('tcc track' command finished) 09:35:52 CmdDone: jaeger configuration reverse 09:35:52 CmdStarted: jaeger configuration execute 09:36:47 CmdDone: jaeger configuration execute 09:36:48 CmdStarted: FVC loop 09:37:48 CmdDone: FVC loop FVC loop results: fvc_rms= 33.9 fvc_deltarms= 5.86 fvc_filename=/data/fcam/61281/fimg-fvc1n-0044.fits fvc_perc_90= 9.60 fvc_percent_reached= 59.30 fvc iterations= 2 09:37:49 CmdStarted: tcc track (slew begins) 09:37:49 Ne lamps turned ON 09:37:56 CmdDone: Slew finished ('tcc track' command finished) 09:38:34 BOSS expNum=403979, expTime= 4.0s 09:38:42 HgCd lamps turned OFF 09:38:42 Ne lamps turned OFF 09:38:51 CmdStarted: cherno acquire 09:40:25 CmdDone: cherno acquire 09:40:26 CmdStarted: 'cherno guide' 09:40:26 hal expose: BOSS science, APOGEE science 09:40:56 BOSS expNum=403980, expTime= 730.0s 09:40:58 APOGEE expNum=57190058, reads=35, dithPos=A, Object 09:47:14 APOGEE expNum=57190059, reads=35, dithPos=B, Object 09:53:25 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 09:53:27 Field-Design-Configuration: 100631-753084-27470 **************************************************************** 09:53:32 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 09:53:34 CmdStarted: jaeger configuration reverse 09:53:38 CmdStarted: tcc track (slew begins) 09:53:43 HgCd lamps turned ON 09:53:54 CmdDone: Slew finished ('tcc track' command finished) 09:54:29 CmdDone: jaeger configuration reverse 09:54:29 CmdStarted: jaeger configuration execute 09:55:18 CmdDone: jaeger configuration execute 09:55:18 CmdStarted: FVC loop 09:56:18 CmdDone: FVC loop FVC loop results: fvc_rms= 24.92 fvc_deltarms= 23.18 fvc_filename=/data/fcam/61281/fimg-fvc1n-0046.fits fvc_perc_90= 10.10 fvc_percent_reached= 55.70 fvc iterations= 2 09:56:19 CmdStarted: tcc track (slew begins) 09:56:19 Ne lamps turned ON 09:56:26 CmdDone: Slew finished ('tcc track' command finished) 09:57:00 BOSS expNum=403981, expTime= 4.0s 09:57:09 HgCd lamps turned OFF 09:57:09 Ne lamps turned OFF 09:57:18 CmdStarted: cherno acquire 09:58:21 CmdDone: cherno acquire 09:58:22 CmdStarted: 'cherno guide' 09:58:22 hal expose: BOSS science, APOGEE science 09:58:53 APOGEE expNum=57190060, reads=35, dithPos=A, Object 09:58:54 BOSS expNum=403982, expTime= 730.0s 10:05:10 APOGEE expNum=57190061, reads=35, dithPos=B, Object 10:11:20 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 10:11:22 Field-Design-Configuration: 102473-754669-27471 **************************************************************** 10:11:28 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:11:30 CmdStarted: jaeger configuration reverse 10:11:34 CmdStarted: tcc track (slew begins) 10:11:38 HgCd lamps turned ON 10:11:46 CmdDone: Slew finished ('tcc track' command finished) 10:12:17 CmdDone: jaeger configuration reverse 10:12:17 CmdStarted: jaeger configuration execute 10:13:05 CmdDone: jaeger configuration execute 10:13:05 CmdStarted: FVC loop 10:14:08 CmdDone: FVC loop FVC loop results: fvc_rms= 7.56 fvc_deltarms= 33.33 fvc_filename=/data/fcam/61281/fimg-fvc1n-0048.fits fvc_perc_90= 10.20 fvc_percent_reached= 54.90 fvc iterations= 2 10:14:09 CmdStarted: tcc track (slew begins) 10:14:09 Ne lamps turned ON 10:14:16 CmdDone: Slew finished ('tcc track' command finished) 10:14:54 BOSS expNum=403983, expTime= 4.0s 10:15:02 HgCd lamps turned OFF 10:15:02 Ne lamps turned OFF 10:15:11 CmdStarted: cherno acquire 10:16:38 CmdDone: cherno acquire 10:16:40 CmdStarted: 'cherno guide' 10:16:40 hal expose: BOSS science, APOGEE science 10:17:09 BOSS expNum=403984, expTime= 730.0s 10:17:11 APOGEE expNum=57190062, reads=35, dithPos=A, Object 10:23:27 APOGEE expNum=57190063, reads=35, dithPos=B, Object 10:29:38 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 10:29:40 Field-Design-Configuration: 102502-754730-27472 **************************************************************** 10:29:46 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:29:47 CmdStarted: jaeger configuration reverse 10:29:51 CmdStarted: tcc track (slew begins) 10:29:56 HgCd lamps turned ON 10:30:07 CmdDone: Slew finished ('tcc track' command finished) 10:30:36 CmdDone: jaeger configuration reverse 10:30:36 CmdStarted: jaeger configuration execute 10:31:20 CmdDone: jaeger configuration execute 10:31:20 CmdStarted: FVC loop 10:32:23 CmdDone: FVC loop FVC loop results: fvc_rms= 8.49 fvc_deltarms= 31.24 fvc_filename=/data/fcam/61281/fimg-fvc1n-0050.fits fvc_perc_90= 9.70 fvc_percent_reached= 53.90 fvc iterations= 2 10:32:23 CmdStarted: tcc track (slew begins) 10:32:23 Ne lamps turned ON 10:32:30 CmdDone: Slew finished ('tcc track' command finished) 10:33:06 BOSS expNum=403985, expTime= 4.0s 10:33:14 HgCd lamps turned OFF 10:33:14 Ne lamps turned OFF 10:33:23 CmdStarted: cherno acquire 10:34:15 CmdDone: cherno acquire 10:34:16 CmdStarted: 'cherno guide' 10:34:18 hal expose: BOSS science, APOGEE science 10:34:47 BOSS expNum=403986, expTime= 730.0s 10:34:49 APOGEE expNum=57190064, reads=35, dithPos=A, Object 10:41:05 APOGEE expNum=57190065, reads=35, dithPos=B, Object 10:47:16 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 10:47:18 Field-Design-Configuration: 100681-753138-27473 **************************************************************** 10:47:24 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 10:47:25 CmdStarted: jaeger configuration reverse 10:47:30 CmdStarted: tcc track (slew begins) 10:47:34 HgCd lamps turned ON 10:48:04 CmdDone: Slew finished ('tcc track' command finished) 10:48:10 CmdDone: jaeger configuration reverse 10:48:10 CmdStarted: jaeger configuration execute 10:49:02 CmdDone: jaeger configuration execute 10:49:02 CmdStarted: FVC loop 10:50:03 CmdDone: FVC loop FVC loop results: fvc_rms= 7.7 fvc_deltarms= 30.85 fvc_filename=/data/fcam/61281/fimg-fvc1n-0052.fits fvc_perc_90= 9.70 fvc_percent_reached= 62.70 fvc iterations= 2 10:50:03 CmdStarted: tcc track (slew begins) 10:50:03 Ne lamps turned ON 10:50:10 CmdDone: Slew finished ('tcc track' command finished) 10:50:45 BOSS expNum=403987, expTime= 4.0s 10:50:53 HgCd lamps turned OFF 10:50:53 Ne lamps turned OFF 10:51:02 CmdStarted: cherno acquire 10:52:29 CmdDone: cherno acquire 10:52:30 CmdStarted: 'cherno guide' 10:52:30 hal expose: BOSS science, APOGEE science 10:52:59 BOSS expNum=403988, expTime= 730.0s 10:53:01 APOGEE expNum=57190066, reads=35, dithPos=A, Object 10:59:18 APOGEE expNum=57190067, reads=35, dithPos=B, Object 11:05:29 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup **************************************************************** 11:05:31 Field-Design-Configuration: 100679-753135-27474 **************************************************************** 11:05:36 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide 11:05:38 CmdStarted: jaeger configuration reverse 11:05:42 CmdStarted: tcc track (slew begins) 11:05:47 HgCd lamps turned ON 11:06:03 CmdDone: Slew finished ('tcc track' command finished) 11:06:29 CmdDone: jaeger configuration reverse 11:06:29 CmdStarted: jaeger configuration execute 11:07:11 CmdDone: jaeger configuration execute 11:07:11 CmdStarted: FVC loop 11:08:12 CmdDone: FVC loop FVC loop results: fvc_rms= 6.43 fvc_deltarms= 32.58 fvc_filename=/data/fcam/61281/fimg-fvc1n-0054.fits fvc_perc_90= 10.30 fvc_percent_reached= 53.70 fvc iterations= 2 11:08:13 CmdStarted: tcc track (slew begins) 11:08:13 Ne lamps turned ON 11:08:20 CmdDone: Slew finished ('tcc track' command finished) 11:08:54 BOSS expNum=403989, expTime= 4.0s 11:09:02 HgCd lamps turned OFF 11:09:02 Ne lamps turned OFF 11:09:12 CmdStarted: cherno acquire 11:10:33 CmdDone: cherno acquire 11:10:35 CmdStarted: 'cherno guide' 11:10:35 hal expose: BOSS science, APOGEE science 11:11:04 BOSS expNum=403990, expTime= 730.0s 11:11:06 APOGEE expNum=57190068, reads=35, dithPos=A, Object 11:17:23 APOGEE expNum=57190069, reads=35, dithPos=B, Object APOGEE dome flats: 11:30:37 APOGEE expNum=57190070, reads=5, dithPos=B, DomeFlat 11:31:52 APOGEE expNum=57190071, reads=5, dithPos=B, DomeFlat 11:33:08 APOGEE expNum=57190072, reads=5, dithPos=B, DomeFlat MJD DomeFlat MTPs:1 2 3 4 5 6 7 8 9 10 AZ ALT ROT 61281 57190070 0.98 1.00 0.99 1.01 1.02 1.03 0.98 1.01 1.00 0.94 121.0 30.0 0.0 61281 57190071 0.99 1.00 0.99 1.01 1.02 1.03 0.98 1.00 1.00 0.95 121.0 30.0 0.0 61281 57190072 0.99 1.00 0.99 1.00 1.01 1.03 0.98 1.01 1.00 0.94 121.0 30.0 0.0 BOSS flat, arc, dark: 11:34:59 BOSS expNum=403991, expTime= 25.0s 11:36:56 BOSS expNum=403992, expTime= 4.0s 11:50:42 BOSS expNum=403993, expTime= 900.0s 11:48:13 2.5m Enclosure: CLOSED. FPS powered-off. Positioners are home/folded. APOGEE morning cals started. ---- Mirror Numbers ---- PRIMARY: -------- Scale: 1.000000 MIGS TONIGHT NOMINAL Axial A: -5.4280 -5.4280 Axial B: -5.0480 -5.0490 Axial C: -5.2770 -5.2770 Trans D: -0.8700 -0.8730 Lateral E: 0.3240 0.3240 Lateral F: 0.5330 0.5330 GALILS Commanded: 11450 -1050 5900 26550 -14400 -13850 Actual: 11450 -1050 5900 26550 -14400 -13850 SETMIR VALUES PrimDesOrient: 0.0 -43.05 -8.77 -751.82 -556.2 PrimOrient: -1.06 -43.05 -8.77 -751.82 -556.2 SECONDARY: ---------- Focus: 0.0000 Air temp: 0.00 Altitude: 30.0 MIGS TONIGHT NOMINAL Axial A: -7.9580 -7.9570 Axial B: -6.1080 -6.1070 Axial C: -6.0780 -6.0780 Trans D: 0.1930 0.2040 GALILS Commanded: 354650 372600 335400 -1500 1800 Actual: 321958 350002 299640 -1500 1800 SETMIR VALUES SecDesOrient: 257.0 14.2 5.9 -72.0 -12.76 SecOrient: 257.01 14.21 5.89 -71.8 -12.63 ---- Focus Log ---- /home/sdss5/opt/pyenv/versions/observers39/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3432: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /home/sdss5/opt/pyenv/versions/observers39/lib/python3.9/site-packages/numpy/core/_methods.py:190: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) Time Field-Design AZ ALT ROT Focus rms ArcOffX OffY Seeing errRA[prc10,prc90] errDec[prc10,prc90] 02:30 100537-752741 16.66 56.33 220.22 14 0.253 12 1 1.36 0.021[-0.106, 0.118] 0.000[-0.111, 0.141] 02:49 100497-752544 21.94 62.14 223.31 25 0.254 14 5 1.24 -0.020[-0.085, 0.129] -0.019[-0.066, 0.084] 03:07 100567-752915 6.38 46.66 212.11 25 0.235 9 1 1.28 0.003[-0.137, 0.125] 0.008[-0.113, 0.073] 03:25 100548-752791 7.53 46.94 212.15 25 0.311 8 1 1.37 0.040[-0.203, 0.428] -0.021[-0.093, 0.254] 03:39 100548-752791 2.60 47.19 207.94 37 0.401 1 10 2.02 0.082[-0.343, 1.452] -0.006[-1.772, 0.507] 03:53 100548-752791 -2.33 47.20 203.73 49 0.241 9 0 1.64 -0.002[-0.133, 0.143] 0.046[-0.149, 0.157] 04:10 100454-752267 -16.70 73.39 194.73 80 nan 11 -4 1.33 nan[-9.990,-9.990] nan[-9.990,-9.990] 04:59 102435-754592 156.08 75.90 -18.96 138 0.286 -20 15 1.58 0.014[-0.272, 0.139] 0.034[-0.133, 0.157] 05:17 100467-752381 307.22 60.46 162.71 159 0.293 7 -10 1.54 0.004[-0.092, 0.092] 0.040[-0.111, 0.113] 05:35 100488-752514 310.71 56.48 169.44 170 0.226 9 -7 1.29 -0.006[-0.082, 0.075] 0.015[-0.166, 0.164] 06:03 100955-753239 294.41 62.42 158.17 170 0.237 4 -12 1.67 0.018[-0.106, 0.117] -0.014[-0.112, 0.170] 06:21 100450-752224 284.15 54.60 154.24 170 0.240 4 -12 1.85 -0.002[-0.151, 0.191] -0.038[-0.153, 0.183] 06:39 100990-753363 313.06 45.73 165.39 181 0.231 6 -7 1.94 0.025[-0.179, 0.201] -0.025[-0.177, 0.222] 06:57 102403-754407 275.66 55.00 117.11 181 0.243 3 -13 1.78 -0.013[-0.196, 0.135] -0.004[-0.185, 0.146] 07:15 100959-753304 279.38 50.40 139.60 181 0.232 0 -13 1.82 0.026[-0.112, 0.099] 0.001[-0.125, 0.088] 07:32 102403-754417 270.31 47.49 116.48 181 0.263 0 -13 1.75 0.010[-0.154, 0.155] -0.057[-0.149, 0.165] 07:51 104435-756519 244.38 53.84 95.01 181 0.293 0 -14 2.20 0.050[-0.263, 0.157] -0.017[-0.146, 0.171] 08:09 100986-753359 280.67 42.86 143.09 181 0.272 0 -9 2.18 -0.000[-0.227, 0.172] -0.005[-0.097, 0.167] 08:27 100977-753346 274.87 38.80 139.58 181 0.251 0 -10 3.30 0.042[-0.154, 0.187] -0.027[-0.299, 0.269] 08:45 104455-756588 235.15 46.49 97.05 155 0.295 -7 -12 2.12 0.066[-0.162, 0.192] 0.036[-0.148, 0.197] 09:04 104481-756625 228.27 43.70 95.67 137 0.321 -5 -11 3.24 0.029[-0.318, 0.251] -0.009[-0.261, 0.185] 09:22 102418-754523 243.38 38.54 111.10 137 0.311 -5 -9 2.80 -0.013[-0.204, 0.184] -0.056[-0.189, 0.267] 09:40 102458-754641 234.76 45.97 107.69 114 0.327 -7 -11 2.19 0.043[-0.250, 0.289] 0.009[-0.209, 0.139] 09:58 100631-753084 234.13 45.28 110.32 136 0.322 -4 -11 2.63 -0.022[-0.166, 0.196] 0.012[-0.375, 0.313] 10:17 102473-754669 236.82 41.31 115.59 161 0.329 -6 -10 2.37 -0.005[-0.209, 0.220] -0.067[-0.350, 0.234] 10:34 102502-754730 241.83 38.42 122.34 161 0.314 -6 -9 2.42 -0.063[-0.274, 0.306] -0.026[-0.249, 0.203] 10:53 100681-753138 243.58 52.55 126.42 161 0.264 -8 -11 2.21 0.014[-0.214, 0.205] -0.021[-0.118, 0.119] ---- Telescope Status ---- Status at: 11:58Z Telescope stowed at: 121.0, 30.0, 0.0 mount Instrument mounted: FPS Counterweights at: manually adjusted for telescope balance w/ FPS mounted LN2 autofill systems: Connected and turned on 180L LN2 dewar scales: SP1 270 lbs,21 psi FPS Status ========== CP10 Loop = 7.49 L/min GFA Loop = 2.37 L/min Supply pressure = 43.0 psi Return pressure = 29.5 psi Chiller Status ============== Set temperarature = 13.1 C Flow = 5.1 Gpm ---- Software Used ---- STUI : 1.7.21 alerts : 2.3.0 boss : 6.0.5a1 -boss.specMec : 3.0.4 apo : 4.1.0 fliswarm : 0.7.0 jaeger : 1.11.2a1 kaiju : 1.4.1 coordio : 1.18.1 fps_calibrati : 2025.12.04 cherno : 1.2.2 astrometrynet : 0.97 hal : 1.4.1 hartmann : 2.0.2 hub : 5.0.1a0 mcp : 2.3.1 -mcp.plc : 35 -mcp.azFiduci : v1_154 -mcp.altFiduc : v1_154 -mcp.rotFiduc : v1_154 tcc : 1.4.0 apogee : not available apogeecal : not available apogeeql : 2.0.0a0 sdss-obstools : 3.9.3 TPM : 1.2-31-gd7eb MCP : 2.3.1 PLC : V35 Fiducials : v1_154 idlspec2d : 6.3.0dev Kronos : 2.1.0a roboscheduler : 2.1.0 ---- APOGEE Data Summary ---- 61280 morning cals: APOGEE data list, mjd=61280, all raw_data: /data/apogee/utr_cdr/61280/apRaw-*.fits archive: /data/apogee/archive/61280/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/61280/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 61280 10 10:11 57180010 Dark 60 B 13.5 ---------- a-b-c - - 61280 11 10:21 57180011 Dark 60 B 13.5 ---------- a-b-c - - 61280 12 10:32 57180012 Dark 60 B 13.5 ---------- a-b-c - - 61280 13 10:34 57180013 QuarFlat 10 B 13.5 ---------- a-b-c - 104 61280 14 10:36 57180014 QuarFlat 10 B 13.5 ---------- a-b-c - 104 61280 15 10:38 57180015 QuarFlat 10 B 13.5 ---------- a-b-c - 104 61280 16 10:40 57180016 Arc-Thar 12 B 13.5 ---------- a-b-c 3.14 104 61280 17 10:47 57180017 Arc-Une 40 B 13.5 ---------- a-b-c 3.19 95 61280 18 10:49 57180018 Arc-Thar 12 A 13.0 ---------- a-b-c 2.66 101 61280 19 10:56 57180019 Arc-Une 40 A 13.0 ---------- a-b-c 2.70 97 61280 20 11:02 57180020 Dark 30 A 13.0 ---------- a-b-c - - 61280 21 11:07 57180021 InteFlat 30 A 13.0 ---------- a-b-c - - 61280 22 11:13 57180022 InteFlat 30 A 13.0 ---------- a-b-c - - 61280 23 11:18 57180023 InteFlat 30 A 13.0 ---------- a-b-c - - 61280 24 11:23 57180024 Dark 30 A 13.0 ---------- a-b-c - - 61280 25 11:29 57180025 Arc---- 30 A 13.0 ---------- a-b-c - 4 FPI A 61280 26 11:34 57180026 Arc---- 30 B 13.5 ---------- a-b-c - 7 FPI B ------------------------------------------------------------------------------ The FPI flux is very low. Looks like the shutter didn't open for the FPI. (observers39) [observer@sdss5-hub: ~]$ list_ap APOGEE data list, mjd=61281, all raw_data: /data/apogee/utr_cdr/61281/apRaw-*.fits archive: /data/apogee/archive/61281/apR-[a,b,c]-*.apz quickred: /data/apogee/quickred/61281/apq-*.fits ------------------------------------------------------------------------------ mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset %Flux/FPI Comm ------------------------------------------------------------------------------ 61281 1 22:12 57190001 Dark 3 B 13.5 ---------- a-b-c - - 61281 2 23:13 57190002 Dark 60 B 13.5 ---------- a-b-c - - 61281 3 23:15 57190003 QuarFlat 10 B 13.5 ---------- a-b-c - 104 61281 4 23:17 57190004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.14 103 61281 5 23:25 57190005 Arc-Une 40 B 13.5 ---------- a-b-c 3.18 96 61281 6 23:27 57190006 Arc-Thar 12 A 13.0 ---------- a-b-c 2.67 102 61281 7 23:34 57190007 Arc-Une 40 A 13.0 ---------- a-b-c 2.69 96 61281 8 23:45 57190008 Dark 60 A 13.0 ---------- a-b-c - - 61281 9 23:51 57190009 Arc---- 30 A 13.0 ---------- a-b-c - 96 FPI A 61281 10 23:56 57190010 Arc---- 30 B 13.5 ---------- a-b-c - 96 FPI B 61281 11 02:07 57190011 DomeFlat 5 B 13.5 -999-27447 a-b-c - 135 61281 12 02:08 57190012 DomeFlat 5 B 13.5 -999-27447 a-b-c - 132 61281 13 02:09 57190013 DomeFlat 5 B 13.5 -999-27447 a-b-c - 131 61281 14 02:36 57190014 Object 35 A 13.0 752741-27448 a-b-c 2.69 1,1 61281 15 02:42 57190015 Object 35 B 13.5 752741-27448 a-b-c 3.18 1,1 61281 16 02:55 57190016 Object 35 A 13.0 752544-27449 a-b-c 2.69 1,1 61281 17 03:01 57190017 Object 35 B 13.5 752544-27449 a-b-c 3.18 1,1 61281 18 03:13 57190018 Object 35 A 13.0 752915-27450 a-b-c 2.70 1,1 61281 19 03:19 57190019 Object 35 B 13.5 752915-27450 a-b-c 3.18 1,1 61281 20 03:31 57190020 Object 35 A 13.0 752791-27451 a-b-c 2.69 1,1 61281 21 03:37 57190021 Object 35 B 13.5 752791-27451 a-b-c 3.17 1,1 61281 22 03:44 57190022 Object 35 B 13.5 752791-27451 a-b-c 3.18 1,1 61281 23 03:50 57190023 Object 35 A 13.0 752791-27451 a-b-c 2.69 1,1 61281 24 03:56 57190024 Object 35 A 13.0 752791-27451 a-b-c 2.70 1,1 61281 25 04:02 57190025 Object 35 B 13.5 752791-27451 a-b-c 3.18 1,1 61281 26 04:16 57190026 Object 35 A 13.0 752267-27452 a-b-c 2.68 1,1 61281 27 04:22 57190027 Object 35 B 13.5 752267-27452 a-b-c 3.17 1,1 61281 28 05:04 57190028 Object 35 A 13.0 754592-27453 a-b-c 2.71 1,1 61281 29 05:11 57190029 Object 35 B 13.5 754592-27453 a-b-c 3.19 1,1 61281 30 05:23 57190030 Object 35 A 13.0 752381-27454 a-b-c 2.69 1,1 61281 31 05:29 57190031 Object 35 B 13.5 752381-27454 a-b-c 3.18 1,1 61281 32 05:40 57190032 Object 35 A 13.0 752514-27455 a-b-c 2.70 1,1 61281 33 05:47 57190033 Object 35 B 13.5 752514-27455 a-b-c 3.18 1,1 61281 34 06:08 57190034 Object 35 A 13.0 753239-27457 a-b-c 2.69 1,1 61281 35 06:15 57190035 Object 35 B 13.5 753239-27457 a-b-c 3.18 1,1 61281 36 06:26 57190036 Object 35 A 13.0 752224-27458 a-b-c 2.43 1,1 61281 37 06:33 57190037 Object 35 B 13.5 752224-27458 a-b-c 2.91 1,1 61281 38 06:44 57190038 Object 35 A 13.0 753363-27459 a-b-c 2.69 1,1 61281 39 06:51 57190039 Object 35 B 13.5 753363-27459 a-b-c 3.18 1,1 61281 40 07:02 57190040 Object 35 A 13.0 754407-27460 a-b-c 2.71 1,1 61281 41 07:09 57190041 Object 35 B 13.5 754407-27460 a-b-c 3.19 1,1 61281 42 07:20 57190042 Object 35 A 13.0 753304-27461 a-b-c 2.75 1,1 61281 43 07:26 57190043 Object 35 B 13.5 753304-27461 a-b-c 3.26 1,1 61281 44 07:38 57190044 Object 35 A 13.0 754417-27462 a-b-c 2.71 1,1 61281 45 07:44 57190045 Object 35 B 13.5 754417-27462 a-b-c 3.19 1,1 61281 46 07:56 57190046 Object 35 A 13.0 756519-27463 a-b-c 2.74 1,1 61281 47 08:03 57190047 Object 35 B 13.5 756519-27463 a-b-c 3.23 1,1 61281 48 08:15 57190048 Object 35 A 13.0 753359-27464 a-b-c 2.90 1,1 61281 49 08:21 57190049 Object 35 B 13.5 753359-27464 a-b-c 3.40 1,1 61281 50 08:32 57190050 Object 35 A 13.0 753346-27465 a-b-c 2.73 1,1 61281 51 08:39 57190051 Object 35 B 13.5 753346-27465 a-b-c 3.20 1,1 61281 52 08:51 57190052 Object 35 A 13.0 756588-27466 a-b-c 2.70 1,1 61281 53 08:57 57190053 Object 35 B 13.5 756588-27466 a-b-c 3.19 1,1 61281 54 09:09 57190054 Object 35 A 13.0 756625-27467 a-b-c 2.70 1,1 61281 55 09:15 57190055 Object 35 B 13.5 756625-27467 a-b-c 3.18 1,1 61281 56 09:28 57190056 Object 35 A 13.0 754523-27468 a-b-c 2.70 1,1 61281 57 09:34 57190057 Object 35 B 13.5 754523-27468 a-b-c 3.19 1,1 61281 58 09:46 57190058 Object 35 A 13.0 754641-27469 a-b-c 2.69 1,1 61281 59 09:52 57190059 Object 35 B 13.5 754641-27469 a-b-c 3.18 1,1 61281 60 10:04 57190060 Object 35 A 13.0 753084-27470 a-b-c 2.70 1,1 61281 61 10:10 57190061 Object 35 B 13.5 753084-27470 a-b-c 3.19 1,1 61281 62 10:22 57190062 Object 35 A 13.0 754669-27471 a-b-c 2.71 1,1 61281 63 10:29 57190063 Object 35 B 13.5 754669-27471 a-b-c 3.20 1,1 61281 64 10:40 57190064 Object 35 A 13.0 754730-27472 a-b-c 2.72 1,1 61281 65 10:46 57190065 Object 35 B 13.5 754730-27472 a-b-c 3.19 1,1 61281 66 10:58 57190066 Object 35 A 13.0 753138-27473 a-b-c 2.71 1,1 61281 67 11:04 57190067 Object 35 B 13.5 753138-27473 a-b-c 3.20 1,1 61281 68 11:16 57190068 Object 35 A 13.0 753135-27474 a-b-c 2.71 1,1 61281 69 11:23 57190069 Object 35 B 13.5 753135-27474 a-b-c 3.20 1,1 61281 70 11:31 57190070 DomeFlat 5 B 13.5 753135-27474 a-b-c - 133 61281 71 11:32 57190071 DomeFlat 5 B 13.5 753135-27474 a-b-c - 131 61281 72 11:33 57190072 DomeFlat 5 B 13.5 753135-27474 a-b-c - 133 ------------------------------------------------------------------------------ ---- BOSS Data Summary ---- Reading FITS from /data/spectro//61281 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - n Time Filename design-field cfg flav shu Exp colA colB colC Hrt - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 22:12 b1-00403923 ----------- ---- bia Clo 0.0 27263 32092 29679 2 22:12 r1-00403923 ----------- ---- bia Clo 0.0 27263 32092 29679 3 22:45 r1-00403924 -999--999 27447 arc Lef 4.1 27263 32092 29679 HART 4 22:45 b1-00403924 -999--999 27447 arc Lef 4.1 27263 32092 29679 HART 5 22:46 b1-00403925 -999--999 27447 arc Rig 4.1 27263 32092 29679 HART 6 22:46 r1-00403925 -999--999 27447 arc Rig 4.1 27263 32092 29679 HART 7 22:47 b1-00403926 -999--999 27447 arc Lef 4.1 25989 30818 28405 HART 8 22:47 r1-00403926 -999--999 27447 arc Lef 4.1 25989 30818 28405 HART 9 22:48 b1-00403927 -999--999 27447 arc Rig 4.1 25989 30818 28405 HART 10 22:48 r1-00403927 -999--999 27447 arc Rig 4.1 25989 30818 28405 HART 11 22:49 b1-00403928 -999--999 27447 arc Out 4.1 26013 30841 28429 12 22:49 r1-00403928 -999--999 27447 arc Out 4.1 26013 30841 28429 13 22:50 b1-00403929 -999--999 27447 fla Out 25.1 26013 30841 28429 14 22:50 r1-00403929 -999--999 27447 fla Out 25.1 26013 30841 28429 15 02:13 b1-00403930 -999--999 27447 arc Lef 4.1 26013 30841 28429 HART 16 02:13 r1-00403930 -999--999 27447 arc Lef 4.1 26013 30841 28429 HART 17 02:13 b1-00403931 -999--999 27447 arc Rig 4.1 26013 30841 28429 HART 18 02:13 r1-00403931 -999--999 27447 arc Rig 4.1 26013 30841 28429 HART 19 02:14 r1-00403932 -999--999 27447 arc Lef 4.1 26847 31677 29265 HART 20 02:14 b1-00403932 -999--999 27447 arc Lef 4.1 26847 31677 29265 HART 21 02:15 r1-00403933 -999--999 27447 arc Rig 4.1 26847 31677 29265 HART 22 02:15 b1-00403933 -999--999 27447 arc Rig 4.1 26847 31677 29265 HART 23 02:16 r1-00403934 -999--999 27447 fla Out 25.1 27031 31861 29448 24 02:16 b1-00403934 -999--999 27447 fla Out 25.1 27031 31861 29448 25 02:18 b1-00403935 -999--999 27447 fla Out 25.1 27031 31861 29448 26 02:18 r1-00403935 -999--999 27447 fla Out 25.1 27031 31861 29448 27 02:20 r1-00403936 -999--999 27447 fla Out 25.1 27031 31861 29448 28 02:20 b1-00403936 -999--999 27447 fla Out 25.1 27031 31861 29448 29 02:27 b1-00403937 752741-100537 27448 arc Out 4.1 27031 31861 29448 30 02:27 r1-00403937 752741-100537 27448 arc Out 4.1 27031 31861 29448 31 02:30 b1-00403938 752741-100537 27448 sci Out 730.1 27031 31861 29448 32 02:30 r1-00403938 752741-100537 27448 sci Out 730.1 27031 31861 29448 33 02:47 b1-00403939 752544-100497 27449 arc Out 4.1 27031 31861 29448 34 02:47 r1-00403939 752544-100497 27449 arc Out 4.1 27031 31861 29448 35 02:49 r1-00403940 752544-100497 27449 sci Out 730.1 27031 31861 29448 36 02:49 b1-00403940 752544-100497 27449 sci Out 730.1 27031 31861 29448 37 03:06 b1-00403941 752915-100567 27450 arc Out 4.1 27031 31861 29448 38 03:06 r1-00403941 752915-100567 27450 arc Out 4.1 27031 31861 29448 39 03:07 b1-00403942 752915-100567 27450 sci Out 730.1 27031 31861 29448 40 03:07 r1-00403942 752915-100567 27450 sci Out 730.1 27031 31861 29448 41 03:24 b1-00403943 752791-100548 27451 arc Out 4.1 27031 31861 29448 42 03:24 r1-00403943 752791-100548 27451 arc Out 4.1 27031 31861 29448 43 03:25 r1-00403944 752791-100548 27451 sci Out 730.1 27031 31861 29448 44 03:25 b1-00403944 752791-100548 27451 sci Out 730.1 27031 31861 29448 45 03:39 b1-00403945 752791-100548 27451 sci Out 730.1 27031 31861 29448 46 03:39 r1-00403945 752791-100548 27451 sci Out 730.1 27031 31861 29448 47 03:53 r1-00403946 752791-100548 27451 sci Out 730.1 27031 31861 29448 48 03:53 b1-00403946 752791-100548 27451 sci Out 730.1 27031 31861 29448 49 04:09 r1-00403947 752267-100454 27452 arc Out 4.1 27031 31861 29448 50 04:09 b1-00403947 752267-100454 27452 arc Out 4.1 27031 31861 29448 51 04:10 b1-00403948 752267-100454 27452 sci Out 730.1 27031 31861 29448 52 04:10 r1-00403948 752267-100454 27452 sci Out 730.1 27031 31861 29448 53 04:56 r1-00403949 754592-102435 27453 arc Out 4.1 27031 31861 29448 54 04:56 b1-00403949 754592-102435 27453 arc Out 4.1 27031 31861 29448 55 04:59 b1-00403950 754592-102435 27453 sci Out 730.1 27031 31861 29448 56 04:59 r1-00403950 754592-102435 27453 sci Out 730.1 27031 31861 29448 57 05:15 r1-00403951 752381-100467 27454 arc Out 4.1 27031 31861 29448 58 05:15 b1-00403951 752381-100467 27454 arc Out 4.1 27031 31861 29448 59 05:17 r1-00403952 752381-100467 27454 sci Out 730.1 27031 31861 29448 60 05:17 b1-00403952 752381-100467 27454 sci Out 730.1 27031 31861 29448 61 05:33 b1-00403953 752514-100488 27455 arc Out 4.1 27031 31861 29448 62 05:33 r1-00403953 752514-100488 27455 arc Out 4.1 27031 31861 29448 63 05:35 b1-00403954 752514-100488 27455 sci Out 730.1 27031 31861 29448 64 05:35 r1-00403954 752514-100488 27455 sci Out 730.1 27031 31861 29448 65 06:00 b1-00403955 753239-100955 27457 arc Out 4.1 27031 31861 29448 66 06:00 r1-00403955 753239-100955 27457 arc Out 4.1 27031 31861 29448 67 06:03 r1-00403956 753239-100955 27457 sci Out 730.1 27031 31861 29448 68 06:03 b1-00403956 753239-100955 27457 sci Out 730.1 27031 31861 29448 69 06:19 b1-00403957 752224-100450 27458 arc Out 4.1 27031 31861 29448 70 06:19 r1-00403957 752224-100450 27458 arc Out 4.1 27031 31861 29448 71 06:21 r1-00403958 752224-100450 27458 sci Out 730.1 27031 31861 29448 72 06:21 b1-00403958 752224-100450 27458 sci Out 730.1 27031 31861 29448 73 06:37 b1-00403959 753363-100990 27459 arc Out 4.1 27031 31861 29448 74 06:37 r1-00403959 753363-100990 27459 arc Out 4.1 27031 31861 29448 75 06:39 r1-00403960 753363-100990 27459 sci Out 730.1 27031 31861 29448 76 06:39 b1-00403960 753363-100990 27459 sci Out 730.1 27031 31861 29448 77 06:55 b1-00403961 754407-102403 27460 arc Out 4.1 27031 31861 29448 78 06:55 r1-00403961 754407-102403 27460 arc Out 4.1 27031 31861 29448 79 06:57 b1-00403962 754407-102403 27460 sci Out 730.1 27031 31861 29448 80 06:57 r1-00403962 754407-102403 27460 sci Out 730.1 27031 31861 29448 81 07:13 b1-00403963 753304-100959 27461 arc Out 4.1 27031 31861 29448 82 07:13 r1-00403963 753304-100959 27461 arc Out 4.1 27031 31861 29448 83 07:15 b1-00403964 753304-100959 27461 sci Out 730.1 27031 31861 29448 84 07:15 r1-00403964 753304-100959 27461 sci Out 730.1 27031 31861 29448 85 07:31 b1-00403965 754417-102403 27462 arc Out 4.1 27031 31861 29448 86 07:31 r1-00403965 754417-102403 27462 arc Out 4.1 27031 31861 29448 87 07:32 r1-00403966 754417-102403 27462 sci Out 730.1 27031 31861 29448 88 07:32 b1-00403966 754417-102403 27462 sci Out 730.1 27031 31861 29448 89 07:49 b1-00403967 756519-104435 27463 arc Out 4.1 27031 31861 29448 90 07:49 r1-00403967 756519-104435 27463 arc Out 4.1 27031 31861 29448 91 07:51 r1-00403968 756519-104435 27463 sci Out 730.1 27031 31861 29448 92 07:51 b1-00403968 756519-104435 27463 sci Out 730.1 27031 31861 29448 93 08:07 r1-00403969 753359-100986 27464 arc Out 4.1 27031 31861 29448 94 08:07 b1-00403969 753359-100986 27464 arc Out 4.1 27031 31861 29448 95 08:09 r1-00403970 753359-100986 27464 sci Out 730.1 27031 31861 29448 96 08:09 b1-00403970 753359-100986 27464 sci Out 730.1 27031 31861 29448 97 08:25 r1-00403971 753346-100977 27465 arc Out 4.1 27031 31861 29448 98 08:25 b1-00403971 753346-100977 27465 arc Out 4.1 27031 31861 29448 99 08:27 b1-00403972 753346-100977 27465 sci Out 730.1 27031 31861 29448 100 08:27 r1-00403972 753346-100977 27465 sci Out 730.1 27031 31861 29448 101 08:43 r1-00403973 756588-104455 27466 arc Out 4.1 27031 31861 29448 102 08:43 b1-00403973 756588-104455 27466 arc Out 4.1 27031 31861 29448 103 08:45 r1-00403974 756588-104455 27466 sci Out 730.1 27031 31861 29448 104 08:45 b1-00403974 756588-104455 27466 sci Out 730.1 27031 31861 29448 105 09:01 r1-00403975 756625-104481 27467 arc Out 4.1 27031 31861 29448 106 09:01 b1-00403975 756625-104481 27467 arc Out 4.1 27031 31861 29448 107 09:04 r1-00403976 756625-104481 27467 sci Out 730.1 27031 31861 29448 108 09:04 b1-00403976 756625-104481 27467 sci Out 730.1 27031 31861 29448 109 09:20 b1-00403977 754523-102418 27468 arc Out 4.1 27031 31861 29448 110 09:20 r1-00403977 754523-102418 27468 arc Out 4.1 27031 31861 29448 111 09:22 r1-00403978 754523-102418 27468 sci Out 730.1 27031 31861 29448 112 09:22 b1-00403978 754523-102418 27468 sci Out 730.1 27031 31861 29448 113 09:38 r1-00403979 754641-102458 27469 arc Out 4.1 27031 31861 29448 114 09:38 b1-00403979 754641-102458 27469 arc Out 4.1 27031 31861 29448 115 09:40 r1-00403980 754641-102458 27469 sci Out 730.1 27031 31861 29448 116 09:40 b1-00403980 754641-102458 27469 sci Out 730.1 27031 31861 29448 117 09:57 r1-00403981 753084-100631 27470 arc Out 4.1 27031 31861 29448 118 09:57 b1-00403981 753084-100631 27470 arc Out 4.1 27031 31861 29448 119 09:58 r1-00403982 753084-100631 27470 sci Out 730.1 27031 31861 29448 120 09:58 b1-00403982 753084-100631 27470 sci Out 730.1 27031 31861 29448 121 10:14 b1-00403983 754669-102473 27471 arc Out 4.1 27031 31861 29448 122 10:14 r1-00403983 754669-102473 27471 arc Out 4.1 27031 31861 29448 123 10:17 r1-00403984 754669-102473 27471 sci Out 730.1 27031 31861 29448 124 10:17 b1-00403984 754669-102473 27471 sci Out 730.1 27031 31861 29448 125 10:33 b1-00403985 754730-102502 27472 arc Out 4.1 27031 31861 29448 126 10:33 r1-00403985 754730-102502 27472 arc Out 4.1 27031 31861 29448 127 10:34 r1-00403986 754730-102502 27472 sci Out 730.1 27031 31861 29448 128 10:34 b1-00403986 754730-102502 27472 sci Out 730.1 27031 31861 29448 129 10:50 b1-00403987 753138-100681 27473 arc Out 4.1 27031 31861 29448 130 10:50 r1-00403987 753138-100681 27473 arc Out 4.1 27031 31861 29448 131 10:53 r1-00403988 753138-100681 27473 sci Out 730.1 27031 31861 29448 132 10:53 b1-00403988 753138-100681 27473 sci Out 730.1 27031 31861 29448 133 11:08 r1-00403989 753135-100679 27474 arc Out 4.1 27031 31861 29448 134 11:08 b1-00403989 753135-100679 27474 arc Out 4.1 27031 31861 29448 135 11:11 r1-00403990 753135-100679 27474 sci Out 730.1 27031 31861 29448 136 11:11 b1-00403990 753135-100679 27474 sci Out 730.1 27031 31861 29448 137 11:34 r1-00403991 753135-100679 27474 fla Out 25.1 27031 31861 29448 138 11:34 b1-00403991 753135-100679 27474 fla Out 25.1 27031 31861 29448 139 11:36 b1-00403992 753135-100679 27474 arc Out 4.1 27031 31861 29448 140 11:36 r1-00403992 753135-100679 27474 arc Out 4.1 27031 31861 29448 ---- BOSS Spectrographs Collimation ---- Time TSP1 R1 B1 Move1 B1Resid ============================================= 22:46:50 10.4 -1649 2.8 -1274 -1.2 22:48:30 10.4 -659 -2.2 24 -2.1 02:14:30 10.8 0 -5.2 835 -2.6 02:16:10 10.8 -659 -3.2 184 -2.6 ---- Weather Log ---- Wind 1 um DIMM IRSC Time Temp C Dewp C MPH Dir Dust fwhm mean sigma 18:00 17.4 10.7 5.6 59.6 dpDep 18:15 17.3 10.3 7.1 54.0 2433 18:30 15.3 11.5 2.0 27.1 3142 18:45 15.2 7.5 5.7 87.1 dpDep 19:00 15.1 -1.2 8.4 46.1 dpDep 19:15 13.9 -2.1 5.6 82.6 3608 19:30 15.5 -2.3 1.5 66.0 1823 19:45 14.6 -2.2 5.6 39.4 2900 20:00 15.6 -0.7 4.2 87.3 2459 20:15 14.7 -0.1 4.7 47.0 3543 20:30 16.3 2.1 7.1 79.8 3537 20:45 18.6 1.4 2.3 98.1 2222 21:00 19.8 2.7 9.2 86.0 2468 21:15 19.1 1.6 9.1 36.4 2360 21:30 19.7 1.9 2.5 96.1 2061 21:45 19.6 2.3 3.6 92.2 2113 22:00 19.1 0.7 6.1 97.2 1963 22:15 19.1 0.8 6.6 86.8 1868 22:30 19.1 1.6 6.2 18.7 2276 22:45 19.3 1.9 3.4 16.3 2398 23:00 20.1 1.9 5.7 28.5 2255 23:15 21.1 2.0 7.2 62.0 2209 23:30 20.7 2.6 5.6 91.0 2186 23:45 21.6 2.3 8.3 32.0 2079 00:00 20.9 2.0 5.4 63.8 2205 00:15 20.8 2.4 4.8 57.1 2121 00:30 20.8 2.1 4.1 89.1 2190 00:45 20.5 2.0 2.1 49.2 2169 01:00 19.7 1.2 1.4 68.5 2167 01:15 19.3 0.1 9.1 56.3 1914 01:30 18.8 0.1 4.8 02.8 2027 01:45 18.8 -0.9 5.2 80.0 1732 02:00 18.6 -0.9 5.1 85.3 1646 02:15 18.3 -1.4 2.9 88.1 1730 02:30 18.5 -1.0 5.1 83.1 1728 02:45 18.3 -1.8 6.4 84.3 1630 347 40.31 03:00 18.0 -1.8 6.6 85.3 1552 404 73.60 03:15 18.1 -2.2 12.9 49.6 1565 589 62.11 03:30 18.0 -2.1 17.1 48.3 1539 392 104.81 03:45 17.7 -1.8 16.6 35.3 1752 353 86.91 04:00 17.4 -2.4 18.5 41.0 1945 399 106.10 04:15 17.2 -2.1 22.6 44.0 1947 340 33.23 04:30 17.2 -2.3 17.7 38.1 2044 341 25.26 04:45 16.8 -3.9 22.7 36.6 2084 342 31.26 05:00 16.6 -3.1 17.9 31.3 2012 342 19.86 05:15 16.2 -3.8 20.5 26.1 2043 341 16.86 05:30 16.4 -3.9 18.1 31.3 2008 341 17.04 05:45 16.3 -3.8 20.3 34.0 2207 341 10.87 06:00 16.1 -3.8 17.4 27.3 2141 340 2.19 06:15 16.1 -4.4 20.0 28.5 2004 340 1.94 06:30 15.9 0.1 17.9 28.5 2064 341 2.45 06:45 16.1 0.1 18.1 38.1 1936 340 1.86 07:00 15.7 0.0 17.6 35.1 1976 340 2.05 07:15 15.9 -0.1 13.3 31.3 2109 341 2.20 07:30 15.6 -0.2 17.1 35.7 1953 340 1.94 07:45 15.7 -0.1 14.3 42.7 1936 340 1.93 08:00 15.5 -0.2 16.3 44.3 1992 340 1.68 08:15 15.0 -0.2 15.8 48.8 2011 340 1.87 08:30 15.2 -0.3 9.0 36.7 1985 341 1.88 08:45 15.1 -0.3 10.3 34.9 2064 340 2.40 09:00 14.8 -0.3 7.3 23.8 2048 340 1.70 09:15 14.9 -0.4 9.1 48.5 2188 341 1.86 09:30 15.0 -0.4 11.7 37.1 2012 340 2.53 09:45 14.8 -0.4 6.4 26.0 2130 340 1.91 10:00 14.6 -0.8 11.5 41.3 2143 341 2.06 10:15 14.4 -0.7 9.4 28.8 2043 341 2.05 10:30 14.2 -0.8 8.0 16.2 2014 341 2.08 10:45 14.8 -0.8 7.4 51.1 1995 341 1.85 11:00 14.2 -0.8 7.0 83.9 1990 340 1.89 11:15 14.1 -0.7 5.3 83.3 1898 340 2.51 ---- Time Tracking ---- *=*=*= Observatory: APO *=*=*= MJD: 61281 Fill night time in Hours below *=*=*= Total bright time : 9.9 *=*=*= Bright time, queued observations : 9.0 *=*=*= Bright time, commissioning/engineering: 0 *=*=*= Bright time lost to weather : 0.4 *=*=*= Bright time lost to technical issues : 0.5 *=*=*= Total dark time : 0 *=*=*= Dark time, queued observations : 0 *=*=*= Dark time, commissioning/engineering : 0 *=*=*= Dark time lost to weather : 0 *=*=*= Dark time lost to technical issues : 0 ---- Problems in Detail ---- The last timestamp from cherno was this guider_status state (0x4 corresponds to bitmask (1<<2) which is a PROCESSING status): 04:10:54 .cherno 0 cherno i guider_status=0x4 04:10:54 hal.hal 6436 cherno i text="Extracting sources." I checked the cherno processes and saw about two dozen(!) processes that looked like attempts at starting cherno, all with the same 04:10Z timestamp: (3.12.9:2.7.18) [sdss5@sdss5-hub ~]$ ps aux | grep cherno sdss5 3469292 0.0 0.0 222860 4032 ? Ss Aug28 0:00 /bin/bash /home/sdss5/config/services/execScripts/cherno.sh sdss5 3469627 1.4 1.1 2509112 364756 ? Sl Aug28 5:38 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3490763 0.0 0.0 234272 13840 ? S 02:28 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python -c from multiprocessing.resource_tracker import main;main(15) sdss5 3507962 0.0 0.9 2467844 296564 ? S 04:10 0:01 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507963 0.0 0.8 2451620 264640 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507964 0.0 0.8 2451620 264640 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507965 0.0 0.8 2451620 264640 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507966 0.0 0.8 2451620 264640 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507967 0.0 0.8 2451620 264640 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507970 0.0 0.9 2467896 296576 ? S 04:10 0:01 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507971 0.0 0.8 2451672 264648 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507972 0.0 0.8 2451672 264648 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507973 0.0 0.8 2451672 264648 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507974 0.0 0.8 2451672 264648 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507975 0.0 0.8 2451672 264648 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507978 0.0 0.8 2459904 265952 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507979 0.0 0.8 2459888 265944 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507981 0.0 0.0 0 0 ? Z 04:10 0:00 [cherno] sdss5 3507982 0.0 0.0 0 0 ? Z 04:10 0:00 [cherno] sdss5 3507983 0.0 0.0 0 0 ? Z 04:10 0:00 [cherno] sdss5 3507986 0.0 0.9 2492556 296140 ? S 04:10 0:01 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507987 0.0 0.8 2476332 265844 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507988 0.0 0.8 2476348 265852 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507989 0.0 0.8 2476348 265852 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507990 0.0 0.8 2476348 265852 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507991 0.0 0.8 2476348 265852 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507994 0.0 0.9 2509000 296172 ? S 04:10 0:01 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507995 0.0 0.8 2492792 265880 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507996 0.0 0.8 2492792 265880 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507997 0.0 0.8 2492792 265880 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507998 0.0 0.8 2492792 265880 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug sdss5 3507999 0.0 0.8 2492792 265880 ? S 04:10 0:00 /home/sdss5/opt/pyenv/versions/3.12.9/envs/cherno-1.2.2/bin/python /home/sdss5/opt/pyenv/versions/cherno-1.2.2/bin/cherno start --debug I stopped cherno and confirmed ALL the relevant PIDs were gone and then restarted cherno. After the cherno restart a preload seemed to work (because it got past the 'get-scale' command it had hung at when cherno was unresponsive), but eventually the 'preload' failed because: 04:42:36 APO.doravetz_2 15 jaeger f error="Another command jaeger-command-parser_configuration_load is already running." 04:42:36 CmdDone: APO.doravetz_2 15 jaeger configuration load --from-preloaded 04:47:27 CmdStarted: APO.doravetz_2 16 jaeger configuration load --from-preloaded 04:47:28 APO.doravetz_2 16 jaeger > 04:47:28 APO.doravetz_2 16 jaeger f error="Another command jaeger-command-parser_configuration_load is already running." 04:47:28 CmdDone: APO.doravetz_2 16 jaeger configuration load --from-preloaded Had to restart jaeger then reload another new design and from that point things were fine.