Date: Mon, 17 Mar 2025 15:52:54 -0600 Subject: Re: [sloan-25mlog 7691] 2.5m obslog 20250316 (SJD 60751) From: "Daniel Oravetz" Hi José, Thanks for the good info. A couple more things about last night's issues related to your email: - I ran a 'top' while I was logged-in to sdss5-fps before I eventually restarted jaeger and during that time when there were timeouts and problems, jaeger was consuming 17.1% for %MEM and %CPU was 0.3. Just checking it right now for a quick comparison I see %MEM is 4.5 for jaeger (but it probably makes more sense to check it after things are powered-up and in an operationally-busy state?). - Before power-cycling the IEB and after manually ensuring there were no reported collisions in the snapshots, we restarted jaeger as another attempt at peeling-back the layers of what might restore things. The systemctl 'restart' command took several minutes to actually complete and once it was complete I verified the PID running on sdss5-fps. Unfortunately, jaeger could not connect to hub after that restart, so I don't know that we could have tried those jaeger 'power on' and 'power off' commands at that point, but it's certainly worth noting those commands for future reference, so thank you! - It seems interesting that both the 'explode' and the 'unwind' commands DID NOT work, but the 'goto' commands DID work. Wouldn't the expectation be that NONE of the commands to move positioners would work if there were a general problem of the robot positions not being able to get updated? - And just an FYI, the problems with jaeger and the timeouts happened about six hours earlier than the STUI/MCPTPM/boson drop-out, so it was not as if there had been one isolated event with the network/communication; there were at least two separate "events." Thanks again for the useful info, Dan > Hi Dan, all, > > Regarding the jaeger issue, and looking at the other problems during the > night, I suspect that there were some transitory network problems that > caused the timeouts getting information from the positioners. I can see > how > that could cause a "Kaiju-collided" error if the positions of the robots > could not be updated. This could also be a server/software issue if jaeger > were running out of resources, but I don't see anything obvious in the > sdss5-fps machine. If this happens again it may be worth doing an htop > before restarting and seeing where the resources stand. But the issue with > the MCPGUI and Boson point to a disconnection from the hub, which is a > separate machine, so networking seems somewhat more likely. > > In these cases I'm not sure that power cycling the IEB as a whole would > help much. I think it's better to power cycle things from jaeger if > necessary, for example > > jaeger power off cm1 cm2 cm3 cm4 cm5 cm6 > jaeger power on cm1 cm2 cm3 cm4 cm5 cm6 > > To power cycle the CAN devices which are associated with the communication > with the robots (followed most likely by a jaeger reboot). > > Thanks, > > José > > > On Mar 17, 2025 at 06:47:31, Daniel Oravetz via sloan-25mlog sloan-25mlog@sdss.org> wrote: > >> >> APACHE POINT OBSERVATORY SDSS 2.5M >> >> OBSERVING LOG >> Sunday March 16, 2025 >> >> (SJD 60751) >> >> ---- Observing Team ---- >> >> Dan Oravetz >> Conor Sayres (support) >> >> ---- Observing Plan ---- >> >> Sun 16 Mar 2025 60751 dark until 03:38 ( 1.3 h), then bright ( 8.7 h, >> w/ >> 0.3 h for 12 deg design) >> >> ---- Observing Summary ---- >> >> A much better night weather-wise tonight with only intermittent cirrus >> clouds, though the cloud cover increased the last ~1.5 hours of the >> night. >> Completed all of the following designs. >> >> Time Field-Design-Conf DesignType BOSS/APOGEE Seeing Overhead >> Complete? Comment >> 01:50:03 104002-715583-19920 bright_time 1x730/1xAB 1.4" 0:00 >> 02:09:14 101951-713816-19921 dark_monit_no_apogee_skies 1x900/1xAB 1.5" >> 9:09 >> 02:33:24 101951-713817-19922 dark_monit_no_apogee_skies 1x900/1xAB 1.5" >> 5:35 >> 02:54:00 101951-713818-19923 dark_monit_no_apogee_skies 1x900/1xAB 1.2" >> 4:18 >> 03:13:19 100862-712758-19924 bright_time 1x730/1xAB 1.2" 6:21 >> 03:31:52 100875-712774-19925 bright_time 1x730/1xAB 1.2" 6:04 >> 03:50:07 100864-712761-19926 bright_time 1x730/1xAB 1.2" 6:05 >> 06:03:37 101322-713039-19928 bright_time_no_apogee_skies 1x730/1xAB >> 2.0" >> 122:11 >> 06:22:45 101344-713056-19929 bright_time_no_apogee_skies 1x730/1xAB >> 2.1" >> 6:00 >> 06:52:16 101612-713564-19931 bright_time_no_apogee_skies 1x730/1xAB >> 1.5" >> 19:34 >> 07:12:41 101624-713576-19932 bright_time_no_apogee_skies 1x730/1xAB >> 1.4" >> 18:47 >> 07:43:39 101655-713598-19933 bright_time_no_apogee_skies 1x730/1xAB >> 1.6" >> 5:27 >> 08:01:17 101373-713083-19934 bright_time_no_apogee_skies 1x730/1xAB >> 1.9" >> 7:29 >> 08:35:18 102330-714019-19936 bright_time_no_apogee_skies 1x730/1xAB >> 1.1" >> 19:31 >> 08:52:38 102562-714388-19937 bright_time_no_apogee_skies 1x730/1xAB >> 1.3" >> 5:29 >> 09:10:18 103203-714815-19938 bright_time 1x730/1xAB 1.4" 6:02 >> 09:28:32 103212-714824-19939 bright_time 1x730/1xAB 1.5" 7:06 >> 09:47:49 103199-714810-19940 bright_time 1x730/1xAB 2.2" 5:36 >> 10:05:35 100023-711320-19941 bright_time 1x730/1xAB 1.6" 23:35; boson >> appeared to be running, but was actually hung >> 10:52:38 100537-712280-19942 bright_time 2x730/2xAB 1.8" 8:46 >> 11:25:52 103228-714866-19943 bright_time 2x730/2xAB 1.3" 5:34 >> 11:57:17 103239-715007-19944 bright_time 1x730/1xAB 1.3" 5:31 >> >> Notes >> ===== >> (1) FPI1 was not showing any flux tonight. I tried a re-plug, but still >> nothing. >> >> (2) At 04:08Z the gotoField failed when the 'reverse' and subsequent >> 'unwind' attempt failed. Jaeger was reporting the grid was >> Kaiju-collided, >> although there were no specific positioners complaining of a collision. >> At >> the same time there were "timed-out" warning messages showing up in >> jaeger. >> I contacted Conor in case it was worth looking at these in real time >> before >> I restarted anything. The details are in the Log body, but ultimately >> the >> problem only resolved once we power-cycled the IEB. >> >> (3) There were some instances of 1187 complaining of collisions, but >> that's because I did not immediately re-disabled 1187 after restarting >> jaeger. Now, however, 1187 is back to being manually-disabled. >> >> (4) At 10:08Z my STUI session disconnected and the MCPTPM GUI also >> reported an 'unavailable' state. I found out shortly later on that boson >> had also disconnected. >> >> ---- Observing Log ---- >> >> >> ============================================================================ >> AFTERNOON CHECKOUT >> >> ============================================================================ >> >> All systems alive and responsive. >> >> Disk space: >> >> Filesystem Size Used Avail Use% Mounted on >> /dev/sdb1 2.0T 462G 1.6T 23% /home >> /dev/sdd1 22T 13T 8.6T 59% /data >> >> FPS powered on. Positioners are home/folded: >> >> 00:21:01 APO.doravetz 31 jaeger i folded=T >> 00:21:01 APO.doravetz 31 jaeger : text="All positioners are folded." >> >> Loaded a manual folded configuration: >> >> 00:21:37 CmdStarted: APO.doravetz 32 jaeger configuration load >> --from-positions >> 00:21:39 APO.doravetz 32 jaeger d text="Resetting cherno offsets." >> 00:21:40 APO.doravetz 32 jaeger d text="Focal plane scale: 1.0." >> 00:21:40 APO.doravetz 32 jaeger d >> configuration_loaded=19919,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/019XXX/0199XX/confSummary-19919.par,F >> 00:21:44 .jaeger 0 jaeger d alive_at=1742170904.671852 >> 00:21:46 APO.doravetz 32 jaeger i >> configuration_snapshot=/data/logs/jaeger/configuration_snapshots/60751/configuration_snapshot_60751_19919.pdf >> 00:21:46 APO.doravetz 32 jaeger d design_preloaded=-999 >> 00:21:46 APO.doravetz 32 jaeger : text="Configuration 19919 loaded." >> 00:21:47 CmdDone: APO.doravetz 32 jaeger configuration load >> --from-positions >> >> Took a '--no-apply' FVC image: >> >> FVC loop results: >> fvc_rms= 73.67 >> fvc_deltarms= 38.87 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0001.fits >> fvc_perc_90= 96.20 >> fvc_percent_reached= 1.40 >> fvc iterations= 1 >> >> Scattered-light check: >> >> 00:26:14 CmdStarted: APO.doravetz 2 fliswarm talk -n fvc expose 50 >> >> fimg-fvc1n-0002.fits shows no evidence of light artifacts. >> >> Exercised positioners: >> >> >> permanently_disabled=201,275,460,478,503,535,639,716,802,846,935,984,985,995,997,1026,1043,1049,1202,1231,1246,1290 >> manually_disabled=630,1187 >> >> 00:27:33 CmdStarted: APO.doravetz 3 jaeger configuration random --danger >> 00:28:58 APO.doravetz 3 jaeger d >> trajectory_dump_file=/data/logs/jaeger/trajectories/60751/trajectory-60751-0001.json >> 00:28:58 CmdDone: APO.doravetz 3 jaeger configuration random --danger >> 00:29:02 .jaeger 0 jaeger i >> snapshot=/data/logs/jaeger/snapshots/60751/fps_snapshot_60751_0001.pdf >> >> 00:29:55 CmdStarted: APO.doravetz 4 jaeger configuration reverse >> >> GFA biases/darks: >> >> 00:31:47 CmdStarted: APO.doravetz 6 fliswarm talk -c gfa expose --bias >> --count 10 >> 00:32:55 CmdStarted: APO.doravetz 7 fliswarm talk -c gfa expose --dark >> --count 5 5 >> >> BOSS arc and flat: >> >> 00:43:12 BOSS expNum=383034, expTime= 4.0s >> 00:44:41 BOSS expNum=383035, expTime= 25.0s >> >> 00:45Z - Ventilation started. >> >> >> ============================================================================ >> NIGHT >> >> ============================================================================ >> >> 01:18:26 2.5m Enclosure: OPEN. Partly cloudy. Force-filled BOSS LN2 >> dewars. >> >> APOGEE dome flats: >> >> 01:31:50 APOGEE expNum=51890011, reads=5, dithPos=B, DomeFlat >> 01:33:06 APOGEE expNum=51890012, reads=5, dithPos=B, DomeFlat >> 01:34:23 APOGEE expNum=51890013, reads=5, dithPos=B, DomeFlat >> >> MJD DomeFlat MTPs:1 2 3 4 5 6 7 8 9 10 AZ >> ALT >> ROT >> 60751 51890011 1.01 1.02 1.04 1.02 1.03 0.99 0.97 0.99 0.99 0.94 121.0 >> 30.0 0.0 >> 60751 51890012 1.02 1.03 1.04 1.01 1.03 1.00 0.97 0.99 0.98 0.94 121.0 >> 30.0 0.0 >> 60751 51890013 1.02 1.03 1.04 1.01 1.03 0.99 0.97 0.99 0.98 0.94 121.0 >> 30.0 0.0 >> >> BOSS hartmanns: >> >> 01:38:43 BOSS expNum=383036, expTime= 4.0s >> 01:39:24 BOSS expNum=383037, expTime= 4.0s >> 01:40:07 sp1.motor.move= 1326, 1325, 1326 >> 01:40:08 Hartmann collimate output >> sp1: offset: r= 659 (In Focus); b= -6.2 (Out of focus) >> sp1: pred. move: spAverageMove= 1325 >> sp1: pred. resid: r= -666, b= -2.1, txt= OK, spTemp = 3.0 >> >> BOSS flats: >> >> 01:42:11 BOSS expNum=383038, expTime= 25.0s >> 01:44:00 BOSS expNum=383039, expTime= 25.0s >> 01:45:49 BOSS expNum=383040, expTime= 25.0s >> >> **************************************************************** >> 01:50:03 Field-Design-Configuration: 104002-715583-19920 >> **************************************************************** >> 01:50:30 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 01:50:30 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 01:50:32 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 01:50:34 CmdStarted: jaeger configuration execute >> 01:50:38 CmdStarted: tcc track (slew begins) >> 01:50:42 HgCd lamps turned ON >> 01:51:21 CmdDone: jaeger configuration execute >> 01:52:17 CmdDone: Slew finished ('tcc track' command finished) >> 01:52:17 CmdStarted: FVC loop >> 01:53:23 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 11.23 >> fvc_deltarms= 55.5 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0005.fits >> fvc_perc_90= 12.00 >> fvc_percent_reached= 52.80 >> fvc iterations= 2 >> 01:53:23 CmdStarted: tcc track (slew begins) >> 01:53:23 Ne lamps turned ON >> 01:53:32 CmdDone: Slew finished ('tcc track' command finished) >> 01:54:06 BOSS expNum=383041, expTime= 4.0s >> 01:54:14 HgCd lamps turned OFF >> 01:54:14 Ne lamps turned OFF >> 01:54:23 CmdStarted: cherno acquire >> 01:56:12 CmdDone: cherno acquire >> 01:56:13 CmdStarted: 'cherno guide' >> 01:56:14 hal expose: BOSS science, APOGEE science >> 01:56:43 BOSS expNum=383042, expTime= 730.0s >> 01:56:45 APOGEE expNum=51890014, reads=35, dithPos=A, Object >> 02:03:01 APOGEE expNum=51890015, reads=35, dithPos=B, Object >> 02:09:12 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 02:09:14 Field-Design-Configuration: 101951-713816-19921 >> **************************************************************** >> 02:09:20 hal goto-field: slew, reconfigure, fvc, reslew, flat, arc, >> acquire, guide >> 02:09:21 CmdStarted: jaeger configuration reverse >> 02:09:26 CmdStarted: tcc track (slew begins) >> 02:10:09 CmdDone: jaeger configuration reverse >> 02:10:09 CmdStarted: jaeger configuration execute >> 02:11:29 CmdDone: Slew finished ('tcc track' command finished) >> 02:11:47 CmdDone: jaeger configuration execute >> 02:11:47 CmdStarted: FVC loop >> 02:12:53 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 16.32 >> fvc_deltarms= 52.43 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0007.fits >> fvc_perc_90= 12.00 >> fvc_percent_reached= 38.70 >> fvc iterations= 2 >> 02:12:53 CmdStarted: tcc track (slew begins) >> 02:12:59 CmdDone: Slew finished ('tcc track' command finished) >> 02:13:17 BOSS expNum=383043, expTime= 25.0s >> 02:13:47 HgCd lamps turned ON >> 02:13:47 Ne lamps turned ON >> 02:15:54 BOSS expNum=383044, expTime= 4.0s >> 02:16:02 HgCd lamps turned OFF >> 02:16:02 Ne lamps turned OFF >> 02:16:11 CmdStarted: cherno acquire >> 02:17:32 CmdDone: cherno acquire >> 02:17:33 CmdStarted: 'cherno guide' >> 02:17:34 hal expose: BOSS science, APOGEE science >> 02:18:03 BOSS expNum=383045, expTime= 900.0s >> 02:18:05 APOGEE expNum=51890016, reads=43, dithPos=A, Object >> 02:25:46 APOGEE expNum=51890017, reads=43, dithPos=B, Object >> 02:33:22 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 02:33:24 Field-Design-Configuration: 101951-713817-19922 >> **************************************************************** >> 02:33:32 hal goto-field: slew, reconfigure, fvc, reslew, acquire, guide >> 02:33:34 CmdStarted: jaeger configuration reverse >> 02:33:38 CmdStarted: tcc track (slew begins) >> 02:33:44 CmdDone: Slew finished ('tcc track' command finished) >> 02:35:12 CmdDone: jaeger configuration reverse >> 02:35:12 CmdStarted: jaeger configuration execute >> 02:36:00 CmdDone: jaeger configuration execute >> 02:36:00 CmdStarted: FVC loop >> 02:37:07 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 9.41 >> fvc_deltarms= 58.65 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0009.fits >> fvc_perc_90= 10.80 >> fvc_percent_reached= 51.90 >> fvc iterations= 2 >> 02:37:07 CmdStarted: tcc track (slew begins) >> 02:37:13 CmdDone: Slew finished ('tcc track' command finished) >> 02:37:14 CmdStarted: cherno acquire >> 02:38:08 CmdDone: cherno acquire >> 02:38:09 CmdStarted: 'cherno guide' >> 02:38:10 hal expose: BOSS science, APOGEE science >> 02:38:39 BOSS expNum=383046, expTime= 900.0s >> 02:38:41 APOGEE expNum=51890018, reads=43, dithPos=A, Object >> 02:46:22 APOGEE expNum=51890019, reads=43, dithPos=B, Object >> 02:53:58 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 02:54:00 Field-Design-Configuration: 101951-713818-19923 >> **************************************************************** >> 02:54:06 hal goto-field: slew, reconfigure, fvc, reslew, acquire, guide >> 02:54:07 CmdStarted: jaeger configuration reverse >> 02:54:12 CmdStarted: tcc track (slew begins) >> 02:54:17 CmdDone: Slew finished ('tcc track' command finished) >> 02:54:56 CmdDone: jaeger configuration reverse >> 02:54:56 CmdStarted: jaeger configuration execute >> 02:55:43 CmdDone: jaeger configuration execute >> 02:55:43 CmdStarted: FVC loop >> 02:56:51 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 13.86 >> fvc_deltarms= 55.28 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0011.fits >> fvc_perc_90= 11.40 >> fvc_percent_reached= 44.60 >> fvc iterations= 2 >> 02:56:51 CmdStarted: tcc track (slew begins) >> 02:56:58 CmdDone: Slew finished ('tcc track' command finished) >> 02:56:59 CmdStarted: cherno acquire >> 02:57:26 CmdDone: cherno acquire >> 02:57:28 CmdStarted: 'cherno guide' >> 02:57:29 hal expose: BOSS science, APOGEE science >> 02:57:58 BOSS expNum=383047, expTime= 900.0s >> 02:58:00 APOGEE expNum=51890020, reads=43, dithPos=A, Object >> 03:05:41 APOGEE expNum=51890021, reads=43, dithPos=B, Object >> 03:13:17 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 03:13:19 Field-Design-Configuration: 100862-712758-19924 >> **************************************************************** >> 03:13:24 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 03:13:26 CmdStarted: jaeger configuration reverse >> 03:13:30 CmdStarted: tcc track (slew begins) >> 03:13:35 HgCd lamps turned ON >> 03:14:14 CmdDone: jaeger configuration reverse >> 03:14:14 CmdStarted: jaeger configuration execute >> 03:15:00 CmdDone: jaeger configuration execute >> 03:15:21 CmdDone: Slew finished ('tcc track' command finished) >> 03:15:21 CmdStarted: FVC loop >> 03:16:29 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 23.41 >> fvc_deltarms= 48.05 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0013.fits >> fvc_perc_90= 10.40 >> fvc_percent_reached= 53.90 >> fvc iterations= 2 >> 03:16:29 CmdStarted: tcc track (slew begins) >> 03:16:30 Ne lamps turned ON >> 03:16:34 CmdDone: Slew finished ('tcc track' command finished) >> 03:17:11 BOSS expNum=383048, expTime= 4.0s >> 03:17:20 HgCd lamps turned OFF >> 03:17:20 Ne lamps turned OFF >> 03:17:29 CmdStarted: cherno acquire >> 03:18:50 CmdDone: cherno acquire >> 03:18:51 CmdStarted: 'cherno guide' >> 03:18:51 hal expose: BOSS science, APOGEE science >> 03:19:20 BOSS expNum=383049, expTime= 730.0s >> 03:19:23 APOGEE expNum=51890022, reads=35, dithPos=A, Object >> 03:25:39 APOGEE expNum=51890023, reads=35, dithPos=B, Object >> 03:31:50 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 03:31:52 Field-Design-Configuration: 100875-712774-19925 >> **************************************************************** >> 03:31:57 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 03:31:59 CmdStarted: jaeger configuration reverse >> 03:32:04 CmdStarted: tcc track (slew begins) >> 03:32:08 HgCd lamps turned ON >> 03:32:22 CmdDone: Slew finished ('tcc track' command finished) >> 03:32:46 CmdDone: jaeger configuration reverse >> 03:32:46 CmdStarted: jaeger configuration execute >> 03:33:58 CmdDone: jaeger configuration execute >> 03:33:58 CmdStarted: FVC loop >> 03:35:05 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 10.08 >> fvc_deltarms= 55.47 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0015.fits >> fvc_perc_90= 11.70 >> fvc_percent_reached= 52.50 >> fvc iterations= 2 >> 03:35:06 CmdStarted: tcc track (slew begins) >> 03:35:06 Ne lamps turned ON >> 03:35:12 CmdDone: Slew finished ('tcc track' command finished) >> 03:35:49 BOSS expNum=383050, expTime= 4.0s >> 03:36:00 HgCd lamps turned OFF >> 03:36:00 Ne lamps turned OFF >> 03:36:09 CmdStarted: cherno acquire >> 03:37:04 CmdDone: cherno acquire >> 03:37:05 CmdStarted: 'cherno guide' >> 03:37:06 hal expose: BOSS science, APOGEE science >> 03:37:35 BOSS expNum=383051, expTime= 730.0s >> 03:37:37 APOGEE expNum=51890024, reads=35, dithPos=A, Object >> 03:43:53 APOGEE expNum=51890025, reads=35, dithPos=B, Object >> 03:50:04 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 03:50:07 Field-Design-Configuration: 100864-712761-19926 >> **************************************************************** >> 03:50:12 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 03:50:14 CmdStarted: jaeger configuration reverse >> 03:50:18 CmdStarted: tcc track (slew begins) >> 03:50:23 HgCd lamps turned ON >> 03:50:32 CmdDone: Slew finished ('tcc track' command finished) >> 03:51:25 CmdDone: jaeger configuration reverse >> 03:51:25 CmdStarted: jaeger configuration execute >> 03:52:16 CmdDone: jaeger configuration execute >> 03:52:16 CmdStarted: FVC loop >> 03:53:21 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 21.78 >> fvc_deltarms= 49.04 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0017.fits >> fvc_perc_90= 11.80 >> fvc_percent_reached= 50.90 >> fvc iterations= 2 >> 03:53:22 CmdStarted: tcc track (slew begins) >> 03:53:22 Ne lamps turned ON >> 03:53:26 CmdDone: Slew finished ('tcc track' command finished) >> 03:54:06 BOSS expNum=383052, expTime= 4.0s >> 03:54:14 HgCd lamps turned OFF >> 03:54:14 Ne lamps turned OFF >> 03:54:23 CmdStarted: cherno acquire >> 03:55:20 CmdDone: cherno acquire >> 03:55:21 CmdStarted: 'cherno guide' >> 03:55:22 hal expose: BOSS science, APOGEE science >> 03:55:51 BOSS expNum=383053, expTime= 730.0s >> 03:55:53 APOGEE expNum=51890026, reads=35, dithPos=A, Object >> 04:02:09 APOGEE expNum=51890027, reads=35, dithPos=B, Object >> 04:08:20 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 04:08:22 Field-Design-Configuration: 102853-714519-19927 >> **************************************************************** >> 04:08:28 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 04:08:30 CmdStarted: jaeger configuration reverse >> 04:08:34 CmdStarted: tcc track (slew begins) >> 04:08:37 CmdDone: jaeger configuration reverse >> 04:08:38 HgCd lamps turned ON >> 04:08:47 HgCd lamps turned OFF >> 04:08:47 Ne lamps turned OFF >> 04:08:58 CmdDone: Slew finished ('tcc track' command finished) >> >> The gotoField failed after the 'jaeger configuration reverse' and >> 'jaeger >> unwind' attempts by hal did not complete and instead produced "Something >> went wrong sending the trajectory: Failed sending SEND_NEW_TRAJECTORY" >> error messages. Suspiciously, there were also some time-out error >> messages >> that showed up at the same time. E.g., here's the failed 'reverse', the >> failed' unwind', and some 'GET_ACTUAL_POSITION' time outs: >> >> 04:08:37 hal.hal 15190 jaeger f error="Trajectory failed with error: >> Something went wrong sending the trajectory: Failed sending >> SEND_NEW_TRAJECTORY" >> 04:08:37 CmdDone: hal.hal 15190 jaeger configuration reverse >> 04:08:37 CmdStarted: hal.hal 15192 jaeger unwind >> 04:08:37 hal.hal 15192 jaeger > >> 04:08:39 .jaeger 0 jaeger w text="GET_ACTUAL_POSITION timed out. >> Retrying." >> 04:08:43 .jaeger 0 jaeger w text="GET_ACTUAL_POSITION timed out. >> Retrying." >> 04:08:45 hal.hal 15192 jaeger i folded=F >> 04:08:45 hal.hal 15192 jaeger d text="Calculating unwind trajectory." >> 04:08:47 hal.hal 15192 jaeger f error="Command 'unwind' failed because >> of >> an uncaught error 'RuntimeError: Grid is Kaiju-collided, cannot generate >> paths'. See traceback in the log for more information." >> 04:08:47 CmdDone: hal.hal 15192 jaeger unwind >> >> When I first saw the "Grid is Kaiju-collided" error message, I thought >> there would be a corresponding message about a positioner complaining of >> a >> collision, but there was no such positioner-specific collision error >> message. >> >> After a little more poking, there were additional "timed-out" messages >> produced in jaeger: >> >> 04:20:58 .jaeger 0 jaeger w text="GET_FIRMWARE_VERSION timed out. >> Retrying." >> 04:21:03 .jaeger 0 jaeger w text="GET_STATUS timed out. Retrying." >> 04:21:05 .jaeger 0 jaeger w text="GET_FIRMWARE_VERSION timed out. >> Retrying." >> 04:21:31 .jaeger 0 jaeger w text="GET_ACTUAL_POSITION timed out. >> Retrying." >> 04:21:35 .jaeger 0 jaeger w text="GET_ACTUAL_POSITION timed out. >> Retrying." >> >> At this point, I contacted Conor in case there was some detail worth >> investigating before I attempted to restart anything. >> >> Conor logged-in to STUI and the first thing we did was to make sure >> there >> were no positioners in a collision-reporting state according to the >> snapshots. In fact there was a group of three positioners that was in a >> software-collided state (red-colored in the snapshot; not physically >> collided, but within the acceptable tolerance defined in the software). >> We >> proceeded to manually de-clutter that group of positioners by carefully >> moving individual positioners and once completed a 'jaeger snapshot' >> indicated that there were NO positioners in any sort of >> collision-reporting >> state. >> >> At that point, we once again tried to unwind the FPS, but we still saw >> the >> "Something went wrong sending the trajectory: Failed sending >> SEND_NEW_TRAJECTORY" error. >> >> Next we tried to restart jaeger, which took quite a long time to >> actually >> complete (of order a couple minutes). After the jaeger restart, the >> connection to the hub did not establish. >> >> Now was time to power-cycle the IEB. After power-cycling the IEB and >> again >> restarting jaeger (to be safe we're starting from a known "fresh" >> state), >> the positioners moved - the 'SEND_NEW_TRAJECTORY' actually worked! >> >> The next "problem" was that after the IEB power-cycle, there were the >> Docker daemon and "no containers" warning messages from fliswarm (which >> usually aren't a problem because they clear with soft reboots). When we >> tried the soft reboot, GFA3 and GFA4 still warned of no client. >> Eventually, >> memory triggered and I remembered the email from August from Jose >> discussing how the spare NUC3 and NUC4 units do not boot on IEB >> power-up, >> so they need to be manually switched on. Manually-turned on NUC3 and >> NUC4 >> inside the IEB, then soft-rebooted those two, issued a couple of >> 'fliswarm >> reconnect' commands and finally we were back to having all GFAs and the >> FVC >> connected. >> >> **************************************************************** >> 06:03:37 Field-Design-Configuration: 101322-713039-19928 >> **************************************************************** >> 06:03:46 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 06:03:46 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 06:03:47 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 06:03:49 CmdStarted: jaeger configuration execute >> 06:03:54 CmdStarted: tcc track (slew begins) >> 06:03:58 HgCd lamps turned ON >> 06:04:40 CmdDone: jaeger configuration execute >> 06:06:19 CmdDone: Slew finished ('tcc track' command finished) >> 06:06:19 CmdStarted: FVC loop >> 06:07:24 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 16.57 >> fvc_deltarms= 51.31 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0019.fits >> fvc_perc_90= 11.20 >> fvc_percent_reached= 51.10 >> fvc iterations= 2 >> 06:07:25 CmdStarted: tcc track (slew begins) >> 06:07:25 Ne lamps turned ON >> 06:07:31 CmdDone: Slew finished ('tcc track' command finished) >> 06:08:07 BOSS expNum=383054, expTime= 4.0s >> 06:08:15 HgCd lamps turned OFF >> 06:08:15 Ne lamps turned OFF >> 06:08:24 CmdStarted: cherno acquire >> 06:09:42 CmdDone: cherno acquire >> 06:09:43 CmdStarted: 'cherno guide' >> 06:09:44 hal expose: BOSS science, APOGEE science >> 06:10:13 BOSS expNum=383055, expTime= 730.0s >> 06:10:15 APOGEE expNum=51890028, reads=35, dithPos=A, Object >> 06:16:31 APOGEE expNum=51890029, reads=35, dithPos=B, Object >> 06:22:42 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 06:22:45 Field-Design-Configuration: 101344-713056-19929 >> **************************************************************** >> 06:22:51 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 06:22:52 CmdStarted: jaeger configuration reverse >> 06:22:57 CmdStarted: tcc track (slew begins) >> 06:23:01 HgCd lamps turned ON >> 06:23:16 CmdDone: Slew finished ('tcc track' command finished) >> 06:23:44 CmdDone: jaeger configuration reverse >> 06:23:44 CmdStarted: jaeger configuration execute >> 06:24:36 CmdDone: jaeger configuration execute >> 06:24:36 CmdStarted: FVC loop >> 06:25:40 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 14.43 >> fvc_deltarms= 53.56 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0021.fits >> fvc_perc_90= 10.90 >> fvc_percent_reached= 47.80 >> fvc iterations= 2 >> 06:25:41 CmdStarted: tcc track (slew begins) >> 06:25:41 Ne lamps turned ON >> 06:25:48 CmdDone: Slew finished ('tcc track' command finished) >> 06:26:23 BOSS expNum=383056, expTime= 4.0s >> 06:26:32 HgCd lamps turned OFF >> 06:26:32 Ne lamps turned OFF >> 06:26:41 CmdStarted: cherno acquire >> 06:27:54 CmdDone: cherno acquire >> 06:27:55 CmdStarted: 'cherno guide' >> 06:27:55 hal expose: BOSS science, APOGEE science >> 06:28:24 BOSS expNum=383057, expTime= 730.0s >> 06:28:27 APOGEE expNum=51890030, reads=35, dithPos=A, Object >> 06:34:43 APOGEE expNum=51890031, reads=35, dithPos=B, Object >> 06:40:54 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 06:40:56 Field-Design-Configuration: 101591-713536-19930 >> **************************************************************** >> 06:41:02 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 06:41:04 CmdStarted: jaeger configuration reverse >> 06:41:08 CmdStarted: tcc track (slew begins) >> 06:41:12 HgCd lamps turned ON >> 06:41:34 CmdDone: Slew finished ('tcc track' command finished) >> 06:41:56 CmdDone: jaeger configuration reverse >> 06:41:56 CmdStarted: jaeger configuration execute >> 06:42:39 Jaeger collision: 1187 >> 06:42:40 CmdDone: jaeger configuration execute >> 06:42:40 HgCd lamps turned OFF >> 06:42:40 Ne lamps turned OFF >> 06:48:33 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 06:48:33 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 06:48:35 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 06:48:37 CmdStarted: jaeger configuration execute >> 06:48:41 CmdStarted: tcc track (slew begins) >> 06:48:45 HgCd lamps turned ON >> 06:48:47 CmdDone: Slew finished ('tcc track' command finished) >> 06:49:19 Jaeger collision: 1187 >> 06:49:20 CmdDone: jaeger configuration execute >> 06:49:20 HgCd lamps turned OFF >> 06:49:20 Ne lamps turned OFF >> >> **************************************************************** >> 06:52:16 Field-Design-Configuration: 101612-713564-19931 >> **************************************************************** >> 06:53:04 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 06:53:04 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 06:53:06 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 06:53:08 CmdStarted: jaeger configuration execute >> 06:53:12 CmdStarted: tcc track (slew begins) >> 06:53:16 HgCd lamps turned ON >> 06:54:00 CmdDone: jaeger configuration execute >> 06:56:20 CmdDone: Slew finished ('tcc track' command finished) >> 06:56:20 CmdStarted: FVC loop >> 06:57:25 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 18.4 >> fvc_deltarms= 43.63 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0023.fits >> fvc_perc_90= 10.40 >> fvc_percent_reached= 51.30 >> fvc iterations= 2 >> 06:57:25 CmdStarted: tcc track (slew begins) >> 06:57:25 Ne lamps turned ON >> 06:57:31 CmdDone: Slew finished ('tcc track' command finished) >> 06:58:07 BOSS expNum=383058, expTime= 4.0s >> 06:58:15 HgCd lamps turned OFF >> 06:58:15 Ne lamps turned OFF >> 06:58:24 CmdStarted: cherno acquire >> 06:59:38 CmdDone: cherno acquire >> 06:59:39 CmdStarted: 'cherno guide' >> 06:59:40 hal expose: BOSS science, APOGEE science >> 07:00:09 BOSS expNum=383059, expTime= 730.0s >> 07:00:12 APOGEE expNum=51890032, reads=35, dithPos=A, Object >> 07:06:27 APOGEE expNum=51890033, reads=35, dithPos=B, Object >> 07:12:39 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 07:12:41 Field-Design-Configuration: 101624-713576-19932 >> **************************************************************** >> 07:12:47 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 07:24:22 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 07:24:22 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 07:24:24 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 07:24:26 CmdStarted: jaeger configuration execute >> 07:24:30 CmdStarted: tcc track (slew begins) >> 07:24:35 HgCd lamps turned ON >> 07:25:08 CmdDone: jaeger configuration execute >> 07:27:37 CmdDone: Slew finished ('tcc track' command finished) >> 07:27:37 CmdStarted: FVC loop >> 07:28:41 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 44.45 >> fvc_deltarms= 19.21 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0025.fits >> fvc_perc_90= 10.20 >> fvc_percent_reached= 55.60 >> fvc iterations= 2 >> 07:28:41 CmdStarted: tcc track (slew begins) >> 07:28:41 Ne lamps turned ON >> 07:28:47 CmdDone: Slew finished ('tcc track' command finished) >> 07:29:23 BOSS expNum=383060, expTime= 4.0s >> 07:29:33 HgCd lamps turned OFF >> 07:29:33 Ne lamps turned OFF >> 07:29:42 CmdStarted: cherno acquire >> 07:30:33 CmdDone: cherno acquire >> 07:30:35 CmdStarted: 'cherno guide' >> 07:30:38 hal expose: BOSS science, APOGEE science >> 07:31:07 BOSS expNum=383061, expTime= 730.0s >> 07:31:09 APOGEE expNum=51890034, reads=35, dithPos=A, Object >> 07:37:25 APOGEE expNum=51890035, reads=35, dithPos=B, Object >> 07:43:37 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 07:43:39 Field-Design-Configuration: 101655-713598-19933 >> **************************************************************** >> 07:43:44 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 07:43:46 CmdStarted: jaeger configuration reverse >> 07:43:50 CmdStarted: tcc track (slew begins) >> 07:43:55 HgCd lamps turned ON >> 07:44:09 CmdDone: Slew finished ('tcc track' command finished) >> 07:44:29 CmdDone: jaeger configuration reverse >> 07:44:29 CmdStarted: jaeger configuration execute >> 07:45:16 CmdDone: jaeger configuration execute >> 07:45:16 CmdStarted: FVC loop >> 07:46:19 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 17.37 >> fvc_deltarms= 52.65 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0027.fits >> fvc_perc_90= 11.80 >> fvc_percent_reached= 45.80 >> fvc iterations= 2 >> 07:46:19 CmdStarted: tcc track (slew begins) >> 07:46:19 Ne lamps turned ON >> 07:46:24 CmdDone: Slew finished ('tcc track' command finished) >> 07:47:01 BOSS expNum=383062, expTime= 4.0s >> 07:47:10 HgCd lamps turned OFF >> 07:47:10 Ne lamps turned OFF >> 07:47:19 CmdStarted: cherno acquire >> 07:48:11 CmdDone: cherno acquire >> 07:48:12 CmdStarted: 'cherno guide' >> 07:48:16 hal expose: BOSS science, APOGEE science >> 07:48:45 BOSS expNum=383063, expTime= 730.0s >> 07:48:47 APOGEE expNum=51890036, reads=35, dithPos=A, Object >> 07:55:04 APOGEE expNum=51890037, reads=35, dithPos=B, Object >> 08:01:15 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 08:01:17 Field-Design-Configuration: 101373-713083-19934 >> **************************************************************** >> 08:01:23 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 08:01:25 CmdStarted: jaeger configuration reverse >> 08:01:29 CmdStarted: tcc track (slew begins) >> 08:01:34 HgCd lamps turned ON >> 08:02:12 CmdDone: jaeger configuration reverse >> 08:02:12 CmdStarted: jaeger configuration execute >> 08:02:58 CmdDone: jaeger configuration execute >> 08:04:33 CmdDone: Slew finished ('tcc track' command finished) >> 08:04:33 CmdStarted: FVC loop >> 08:05:37 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 57.48 >> fvc_deltarms= 8.36 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0029.fits >> fvc_perc_90= 11.00 >> fvc_percent_reached= 50.90 >> fvc iterations= 2 >> 08:05:38 CmdStarted: tcc track (slew begins) >> 08:05:38 Ne lamps turned ON >> 08:05:44 CmdDone: Slew finished ('tcc track' command finished) >> 08:06:19 BOSS expNum=383064, expTime= 4.0s >> 08:06:28 HgCd lamps turned OFF >> 08:06:28 Ne lamps turned OFF >> 08:06:37 CmdStarted: cherno acquire >> 08:07:54 CmdDone: cherno acquire >> 08:07:56 CmdStarted: 'cherno guide' >> 08:07:57 hal expose: BOSS science, APOGEE science >> 08:08:26 BOSS expNum=383065, expTime= 730.0s >> 08:08:28 APOGEE expNum=51890038, reads=35, dithPos=A, Object >> 08:14:44 APOGEE expNum=51890039, reads=35, dithPos=B, Object >> 08:20:56 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 08:20:58 Field-Design-Configuration: 102593-714402-19935 >> **************************************************************** >> 08:21:03 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 08:21:05 CmdStarted: jaeger configuration reverse >> 08:21:09 CmdStarted: tcc track (slew begins) >> 08:21:14 HgCd lamps turned ON >> 08:21:51 CmdDone: jaeger configuration reverse >> 08:21:51 CmdStarted: jaeger configuration execute >> 08:22:34 Jaeger collision: 1187 >> 08:22:36 CmdDone: jaeger configuration execute >> 08:22:36 HgCd lamps turned OFF >> 08:22:36 Ne lamps turned OFF >> 08:24:09 CmdDone: Slew finished ('tcc track' command finished) >> >> **************************************************************** >> 08:35:18 Field-Design-Configuration: 102330-714019-19936 >> **************************************************************** >> 08:35:24 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 08:35:24 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 08:35:25 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 08:35:27 CmdStarted: jaeger configuration execute >> 08:35:31 CmdStarted: tcc track (slew begins) >> 08:35:36 HgCd lamps turned ON >> 08:36:05 CmdDone: Slew finished ('tcc track' command finished) >> 08:36:13 CmdDone: jaeger configuration execute >> 08:36:13 CmdStarted: FVC loop >> 08:37:17 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 9.55 >> fvc_deltarms= 59.73 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0031.fits >> fvc_perc_90= 10.90 >> fvc_percent_reached= 52.80 >> fvc iterations= 2 >> 08:37:18 CmdStarted: tcc track (slew begins) >> 08:37:18 Ne lamps turned ON >> 08:37:24 CmdDone: Slew finished ('tcc track' command finished) >> 08:38:01 BOSS expNum=383066, expTime= 4.0s >> 08:38:12 HgCd lamps turned OFF >> 08:38:12 Ne lamps turned OFF >> 08:38:21 CmdStarted: cherno acquire >> 08:39:36 CmdDone: cherno acquire >> 08:39:37 CmdStarted: 'cherno guide' >> 08:39:37 hal expose: BOSS science, APOGEE science >> 08:40:07 BOSS expNum=383067, expTime= 730.0s >> 08:40:09 APOGEE expNum=51890040, reads=35, dithPos=A, Object >> 08:46:25 APOGEE expNum=51890041, reads=35, dithPos=B, Object >> 08:52:36 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 08:52:39 Field-Design-Configuration: 102562-714388-19937 >> **************************************************************** >> 08:52:44 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 08:52:46 CmdStarted: jaeger configuration reverse >> 08:52:51 CmdStarted: tcc track (slew begins) >> 08:52:55 HgCd lamps turned ON >> 08:53:06 CmdDone: Slew finished ('tcc track' command finished) >> 08:53:31 CmdDone: jaeger configuration reverse >> 08:53:31 CmdStarted: jaeger configuration execute >> 08:54:18 CmdDone: jaeger configuration execute >> 08:54:18 CmdStarted: FVC loop >> 08:55:22 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 15.72 >> fvc_deltarms= 53.13 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0033.fits >> fvc_perc_90= 11.00 >> fvc_percent_reached= 50.50 >> fvc iterations= 2 >> 08:55:23 CmdStarted: tcc track (slew begins) >> 08:55:23 Ne lamps turned ON >> 08:55:30 CmdDone: Slew finished ('tcc track' command finished) >> 08:56:05 BOSS expNum=383068, expTime= 4.0s >> 08:56:13 HgCd lamps turned OFF >> 08:56:13 Ne lamps turned OFF >> 08:56:22 CmdStarted: cherno acquire >> 08:57:15 CmdDone: cherno acquire >> 08:57:16 CmdStarted: 'cherno guide' >> 08:57:18 hal expose: BOSS science, APOGEE science >> 08:57:47 BOSS expNum=383069, expTime= 730.0s >> 08:57:49 APOGEE expNum=51890042, reads=35, dithPos=A, Object >> 09:04:05 APOGEE expNum=51890043, reads=35, dithPos=B, Object >> 09:10:16 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 09:10:19 Field-Design-Configuration: 103203-714815-19938 >> **************************************************************** >> 09:10:24 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 09:10:27 CmdStarted: jaeger configuration reverse >> 09:10:32 CmdStarted: tcc track (slew begins) >> 09:10:36 HgCd lamps turned ON >> 09:11:14 CmdDone: jaeger configuration reverse >> 09:11:14 CmdStarted: jaeger configuration execute >> 09:11:16 CmdDone: Slew finished ('tcc track' command finished) >> 09:12:03 CmdDone: jaeger configuration execute >> 09:12:03 CmdStarted: FVC loop >> 09:13:07 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 15.61 >> fvc_deltarms= 50.53 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0035.fits >> fvc_perc_90= 11.60 >> fvc_percent_reached= 48.60 >> fvc iterations= 2 >> 09:13:08 CmdStarted: tcc track (slew begins) >> 09:13:08 Ne lamps turned ON >> 09:13:14 CmdDone: Slew finished ('tcc track' command finished) >> 09:13:52 BOSS expNum=383070, expTime= 4.0s >> 09:14:00 HgCd lamps turned OFF >> 09:14:00 Ne lamps turned OFF >> 09:14:10 CmdStarted: cherno acquire >> 09:15:29 CmdDone: cherno acquire >> 09:15:30 CmdStarted: 'cherno guide' >> 09:15:31 hal expose: BOSS science, APOGEE science >> 09:16:00 BOSS expNum=383071, expTime= 730.0s >> 09:16:02 APOGEE expNum=51890044, reads=35, dithPos=A, Object >> 09:22:19 APOGEE expNum=51890045, reads=35, dithPos=B, Object >> 09:28:30 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 09:28:32 Field-Design-Configuration: 103212-714824-19939 >> **************************************************************** >> 09:28:38 hal goto-field: slew, reconfigure, fvc, reslew, hartmanns, >> arc, >> acquire, guide >> 09:28:40 CmdStarted: jaeger configuration reverse >> 09:28:44 CmdStarted: tcc track (slew begins) >> 09:28:48 HgCd lamps turned ON >> 09:29:01 CmdDone: Slew finished ('tcc track' command finished) >> 09:29:28 CmdDone: jaeger configuration reverse >> 09:29:28 CmdStarted: jaeger configuration execute >> 09:30:15 CmdDone: jaeger configuration execute >> 09:30:15 CmdStarted: FVC loop >> 09:31:19 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 13.47 >> fvc_deltarms= 52.06 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0037.fits >> fvc_perc_90= 10.70 >> fvc_percent_reached= 52.10 >> fvc iterations= 2 >> 09:31:19 CmdStarted: tcc track (slew begins) >> 09:31:20 Ne lamps turned ON >> 09:31:27 CmdDone: Slew finished ('tcc track' command finished) >> 09:31:45 BOSS expNum=383072, expTime= 4.0s >> 09:32:26 BOSS expNum=383073, expTime= 4.0s >> 09:33:13 sp1.motor.move= 1186, 1186, 1186 >> 09:33:13 Hartmann collimate output >> sp1: offset: r= 2308 (Out of focus); b= -0.2 (In Focus) >> sp1: pred. move: spAverageMove= 1186 >> sp1: pred. resid: r= 1122, b= 3.5, txt= OK, spTemp = 3.2 >> 09:33:31 BOSS expNum=383074, expTime= 4.0s >> 09:33:40 HgCd lamps turned OFF >> 09:33:40 Ne lamps turned OFF >> 09:33:50 CmdStarted: cherno acquire >> 09:34:46 CmdDone: cherno acquire >> 09:34:48 CmdStarted: 'cherno guide' >> 09:34:48 hal expose: BOSS science, APOGEE science >> 09:35:17 BOSS expNum=383075, expTime= 730.0s >> 09:35:19 APOGEE expNum=51890046, reads=35, dithPos=A, Object >> 09:41:35 APOGEE expNum=51890047, reads=35, dithPos=B, Object >> 09:47:47 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 09:47:49 Field-Design-Configuration: 103199-714810-19940 >> **************************************************************** >> 09:47:55 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 09:47:56 CmdStarted: jaeger configuration reverse >> 09:48:01 CmdStarted: tcc track (slew begins) >> 09:48:05 HgCd lamps turned ON >> 09:48:13 CmdDone: Slew finished ('tcc track' command finished) >> 09:48:45 CmdDone: jaeger configuration reverse >> 09:48:45 CmdStarted: jaeger configuration execute >> 09:49:32 CmdDone: jaeger configuration execute >> 09:49:32 CmdStarted: FVC loop >> 09:50:36 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 14.8 >> fvc_deltarms= 56.32 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0039.fits >> fvc_perc_90= 11.00 >> fvc_percent_reached= 49.80 >> fvc iterations= 2 >> 09:50:37 CmdStarted: tcc track (slew begins) >> 09:50:37 Ne lamps turned ON >> 09:50:44 CmdDone: Slew finished ('tcc track' command finished) >> 09:51:21 BOSS expNum=383076, expTime= 4.0s >> 09:51:30 HgCd lamps turned OFF >> 09:51:30 Ne lamps turned OFF >> 09:51:39 CmdStarted: cherno acquire >> 09:52:06 CmdDone: cherno acquire >> 09:52:07 CmdStarted: 'cherno guide' >> 09:52:34 hal expose: BOSS science, APOGEE science >> 09:53:04 BOSS expNum=383077, expTime= 730.0s >> 09:53:06 APOGEE expNum=51890048, reads=35, dithPos=A, Object >> 09:59:22 APOGEE expNum=51890049, reads=35, dithPos=B, Object >> 10:06:14 hal expose: BOSS science, APOGEE science >> >> **************************************************************** >> 10:06:14 Field-Design-Configuration: 103199-714810-19940 >> **************************************************************** >> 10:06:40 hal expose: BOSS science, APOGEE science >> >> **************************************************************** >> 10:06:41 Field-Design-Configuration: 103199-714810-19940 >> **************************************************************** >> 10:08:49 CmdStarted: FVC loop >> >> 10:08Z - STUI disconnected. MCPTPM GUI connection also became >> unavailable. >> >> 10:09:53 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 30.09 >> fvc_deltarms= 39.79 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0041.fits >> fvc_perc_90= 10.90 >> fvc_percent_reached= 53.80 >> fvc iterations= 2 >> 10:09:54 CmdStarted: tcc track (slew begins) >> 10:09:54 Ne lamps turned ON >> 10:09:59 CmdDone: Slew finished ('tcc track' command finished) >> 10:10:36 BOSS expNum=383078, expTime= 4.0s >> 10:10:46 HgCd lamps turned OFF >> 10:10:46 Ne lamps turned OFF >> 10:21:36 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 10:21:36 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 10:22:17 hal goto-field: slew, acquire, guide >> 10:22:22 CmdStarted: tcc track (slew begins) >> 10:22:31 CmdDone: Slew finished ('tcc track' command finished) >> 10:22:31 CmdStarted: tcc track (slew begins) >> 10:22:36 CmdDone: Slew finished ('tcc track' command finished) >> 10:22:37 CmdStarted: cherno acquire >> 10:23:53 CmdDone: cherno acquire >> 10:23:55 CmdStarted: 'cherno guide' >> >> Apparently when STUI and MCPTPM reported being disconnected, boson was >> also not responsive, but it was not obvious because it looked like the >> progress bar was active. I restarted boson, then it showed that the >> gotoField had stopped because ROT was not tracking for the 'acquire' >> stage. >> >> 10:38:21 hal expose: BOSS science, APOGEE science >> 10:38:44 APOGEE expNum=51890050, reads=35, dithPos=B, Object >> 10:38:50 BOSS expNum=383079, expTime= 730.0s >> 10:45:04 APOGEE expNum=51890051, reads=35, dithPos=A, Object >> >> **************************************************************** >> 10:52:38 Field-Design-Configuration: 100537-712280-19942 >> **************************************************************** >> 10:52:44 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 10:52:44 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> 10:52:46 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 10:52:48 CmdStarted: jaeger configuration reverse >> 10:52:52 CmdStarted: tcc track (slew begins) >> 10:52:57 HgCd lamps turned ON >> 10:53:46 CmdDone: jaeger configuration reverse >> 10:53:46 CmdStarted: jaeger configuration execute >> 10:54:39 CmdDone: jaeger configuration execute >> 10:55:55 CmdDone: Slew finished ('tcc track' command finished) >> 10:55:55 CmdStarted: FVC loop >> 10:56:59 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 27.13 >> fvc_deltarms= 45.41 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0043.fits >> fvc_perc_90= 11.20 >> fvc_percent_reached= 51.30 >> fvc iterations= 2 >> 10:56:59 CmdStarted: tcc track (slew begins) >> 10:57:00 Ne lamps turned ON >> 10:57:05 CmdDone: Slew finished ('tcc track' command finished) >> 10:57:43 BOSS expNum=383080, expTime= 4.0s >> 10:57:51 HgCd lamps turned OFF >> 10:57:51 Ne lamps turned OFF >> 10:58:00 CmdStarted: cherno acquire >> 10:59:16 CmdDone: cherno acquire >> 10:59:17 CmdStarted: 'cherno guide' >> 10:59:18 hal expose: BOSS science, APOGEE science >> 10:59:41 APOGEE expNum=51890052, reads=35, dithPos=A, Object >> 10:59:47 BOSS expNum=383081, expTime= 730.0s >> 11:05:57 APOGEE expNum=51890053, reads=35, dithPos=B, Object >> 11:12:10 APOGEE expNum=51890054, reads=35, dithPos=B, Object >> 11:13:25 BOSS expNum=383082, expTime= 730.0s >> 11:18:31 APOGEE expNum=51890055, reads=35, dithPos=A, Object >> 11:25:50 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 11:25:52 Field-Design-Configuration: 103228-714866-19943 >> **************************************************************** >> 11:25:58 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 11:26:00 CmdStarted: jaeger configuration reverse >> 11:26:04 CmdStarted: tcc track (slew begins) >> 11:26:08 HgCd lamps turned ON >> 11:26:27 CmdDone: Slew finished ('tcc track' command finished) >> 11:26:52 CmdDone: jaeger configuration reverse >> 11:26:52 CmdStarted: jaeger configuration execute >> 11:27:37 CmdDone: jaeger configuration execute >> 11:27:37 CmdStarted: FVC loop >> 11:28:42 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 29.25 >> fvc_deltarms= 39.17 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0045.fits >> fvc_perc_90= 11.80 >> fvc_percent_reached= 50.70 >> fvc iterations= 2 >> 11:28:42 CmdStarted: tcc track (slew begins) >> 11:28:42 Ne lamps turned ON >> 11:28:49 CmdDone: Slew finished ('tcc track' command finished) >> 11:29:27 BOSS expNum=383083, expTime= 4.0s >> 11:29:35 HgCd lamps turned OFF >> 11:29:36 Ne lamps turned OFF >> 11:29:45 CmdStarted: cherno acquire >> 11:30:39 CmdDone: cherno acquire >> 11:30:40 CmdStarted: 'cherno guide' >> 11:30:41 hal expose: BOSS science, APOGEE science >> 11:31:04 APOGEE expNum=51890056, reads=38, dithPos=A, Object >> 11:31:10 BOSS expNum=383084, expTime= 730.0s >> 11:37:52 APOGEE expNum=51890057, reads=38, dithPos=B, Object >> 11:44:37 APOGEE expNum=51890058, reads=35, dithPos=B, Object >> 11:44:51 BOSS expNum=383085, expTime= 730.0s >> 11:50:57 APOGEE expNum=51890059, reads=35, dithPos=A, Object >> 11:57:15 hal auto-pilot: auto_pilot, prepare, load, goto_field, expose, >> cleanup >> >> **************************************************************** >> 11:57:17 Field-Design-Configuration: 103239-715007-19944 >> **************************************************************** >> 11:57:23 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, >> guide >> 11:57:25 CmdStarted: jaeger configuration reverse >> 11:57:29 CmdStarted: tcc track (slew begins) >> 11:57:34 HgCd lamps turned ON >> 11:57:46 CmdDone: Slew finished ('tcc track' command finished) >> 11:58:09 CmdDone: jaeger configuration reverse >> 11:58:09 CmdStarted: jaeger configuration execute >> 11:59:01 CmdDone: jaeger configuration execute >> 11:59:01 CmdStarted: FVC loop >> 12:00:05 CmdDone: FVC loop >> FVC loop results: >> fvc_rms= 15.34 >> fvc_deltarms= 54.67 >> fvc_filename=/data/fcam/60751/fimg-fvc1n-0047.fits >> fvc_perc_90= 10.40 >> fvc_percent_reached= 51.00 >> fvc iterations= 2 >> 12:00:06 CmdStarted: tcc track (slew begins) >> 12:00:06 Ne lamps turned ON >> 12:00:13 CmdDone: Slew finished ('tcc track' command finished) >> 12:00:50 BOSS expNum=383086, expTime= 4.0s >> 12:00:58 HgCd lamps turned OFF >> 12:00:58 Ne lamps turned OFF >> 12:01:08 CmdStarted: cherno acquire >> 12:01:36 CmdDone: cherno acquire >> 12:01:37 CmdStarted: 'cherno guide' >> 12:02:03 hal expose: BOSS science, APOGEE science >> 12:02:26 APOGEE expNum=51890060, reads=35, dithPos=A, Object >> 12:02:32 BOSS expNum=383087, expTime= 730.0s >> 12:08:42 APOGEE expNum=51890061, reads=35, dithPos=B, Object >> >> APOGEE dome flats: >> >> 12:20:40 APOGEE expNum=51890062, reads=5, dithPos=B, DomeFlat >> 12:21:56 APOGEE expNum=51890063, reads=5, dithPos=B, DomeFlat >> 12:23:11 APOGEE expNum=51890064, reads=5, dithPos=B, DomeFlat >> >> MJD DomeFlat MTPs:1 2 3 4 5 6 7 8 9 10 AZ >> ALT >> ROT >> 60751 51890062 0.99 1.01 1.02 1.01 1.02 1.01 0.98 1.00 0.98 0.95 121.0 >> 30.0 0.0 >> 60751 51890063 1.00 1.02 1.02 1.01 1.02 1.01 0.98 1.00 0.98 0.95 121.0 >> 30.0 0.0 >> 60751 51890064 1.01 1.02 1.02 1.01 1.02 1.01 0.98 0.99 0.98 0.94 121.0 >> 30.0 0.0 >> >> BOSS flat and arc: >> >> 12:24:23 BOSS expNum=383088, expTime= 25.0s >> 12:26:22 BOSS expNum=383089, expTime= 4.0s >> >> 12:33:52 2.5m Enclosure: CLOSED. FPS powered-off. Positioners are >> home/folded. APOGEE morning cals started. >> >> ---- Mirror Numbers ---- >> >> PRIMARY: >> -------- >> Scale: 1.000000 >> >> MIGS TONIGHT NOMINAL >> Axial A: -0.0010 0.0040 >> Axial B: -0.0009 -0.0010 >> Axial C: -0.0120 -0.0060 >> Trans D: -0.0070 -0.0100 >> Lateral E: 0.3380 0.3380 >> Lateral F: 0.5455 0.5455 >> >> GALILS >> Commanded: 9800 -300 6750 19150 -3550 -3250 >> Actual: 9800 -300 6750 19150 -3550 -3250 >> >> SETMIR VALUES >> PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 >> PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 >> >> SECONDARY: >> ---------- >> Focus: 0.0000 >> Air temp: 0.00 >> Altitude: 30.0 >> >> MIGS TONIGHT NOMINAL >> Axial A: -0.1040 -0.0970 >> Axial B: -0.0670 -0.0650 >> Axial C: -0.0390 -0.0420 >> Trans D: 0.0020 -0.0030 >> >> GALILS >> Commanded: 5902050 5849950 5976550 -1500 -7250 >> Actual: 5883744 5811599 5905999 -1500 -7250 >> >> SETMIR VALUES >> SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 >> SecOrient: 4656.95 -24.46 -17.57 17.64 125.23 >> >> ---- Telescope Offsets and Scale ---- >> >> Time Field-Design Az Alt Rot RA Off Dec Off Rot Off Guide >> RMS >> (um) >> >> ================================================================================ >> 02:15:00 101951-713816 98.0 63.5 -64.4 -0.005 0.002 -0.371 >> 0.336 >> 02:33:11 101951-713816 96.6 67.4 -63.1 0.003 0.005 -0.321 >> 0.243 >> 02:53:47 101951-713817 95.0 71.6 -61.9 0.003 0.007 -0.319 >> 0.182 >> 03:13:06 101951-713818 93.3 75.8 -60.8 0.003 0.008 -0.320 >> 0.166 >> 03:30:00 100862-712758 267.5 66.5 134.2 -0.002 0.004 -0.378 >> 0.172 >> 03:49:54 100875-712774 273.2 65.2 140.1 -0.002 0.004 -0.377 >> 0.251 >> 04:08:10 100864-712761 266.9 62.9 137.2 -0.001 0.003 -0.381 >> 0.285 >> 06:10:03 101322-713039 115.2 34.6 266.2 0.004 -0.002 -0.051 >> 0.447 >> 06:28:15 101344-713056 120.8 33.6 265.7 0.004 -0.002 -0.031 >> 0.358 >> 06:45:00 101591-713536 98.1 36.7 261.5 0.004 -0.001 -0.029 >> 0.248 >> 07:00:00 101612-713564 108.3 48.8 -88.0 0.004 -0.001 -0.076 >> 8.094 >> 07:30:57 101624-713576 102.5 55.3 260.5 0.003 0.002 -0.076 >> 2.782 >> 07:48:35 101655-713598 90.5 56.9 265.0 0.003 0.003 -0.082 >> 3.581 >> 08:08:15 101373-713083 121.7 36.5 -76.6 0.002 -0.001 0.005 >> 0.509 >> 08:30:00 102593-714402 72.8 39.4 258.7 0.002 -0.000 0.005 >> 0.426 >> 08:46:10 102330-714019 70.2 59.2 252.2 0.004 0.002 -0.088 >> 0.249 >> 09:03:49 102562-714388 65.0 60.1 248.7 0.004 0.002 -0.086 >> 0.271 >> 09:22:03 103203-714815 67.1 32.0 258.8 0.001 -0.002 -0.062 >> 0.349 >> 09:41:20 103212-714824 59.6 32.3 256.9 0.001 -0.002 -0.065 >> 0.361 >> 09:59:06 103199-714810 52.9 34.6 254.1 0.001 -0.001 -0.068 >> 0.239 >> 10:15:00 100023-711320 92.9 33.1 -85.3 0.001 -0.001 -0.069 >> 0.446 >> 10:38:32 100023-711320 89.7 38.1 -85.1 0.002 -0.001 -0.048 >> 0.264 >> 10:59:29 100537-712280 61.0 37.3 253.5 0.002 -0.001 -0.062 >> 3.761 >> 11:15:00 100537-712280 57.7 40.0 251.5 0.002 -0.001 -0.064 >> 0.207 >> >> ---- Focus Log ---- >> >> Time Field-Design M1 M2 Focus Az Alt Temp Wind Dir >> >> ================================================================================ >> 02:15:00 101951-713816 0.00 4466.83 -77 98.0 63.5 0.0 5 347 >> 02:33:11 101951-713816 0.00 4427.61 -109 96.6 67.4 0.0 7 351 >> 02:53:47 101951-713817 0.00 4420.18 -109 95.0 71.6 0.0 7 350 >> 03:13:06 101951-713818 0.00 4345.59 -177 93.3 75.8 0.0 7 322 >> 03:30:00 100862-712758 0.00 4360.42 -177 267.5 66.5 0.0 8 324 >> 03:49:54 100875-712774 0.00 4395.00 -145 273.2 65.2 0.0 6 339 >> 04:08:10 100864-712761 0.00 4411.69 -134 266.9 62.9 0.0 9 312 >> 06:10:03 101322-713039 0.00 4528.47 -109 115.2 34.6 0.0 7 213 >> 06:28:15 101344-713056 0.00 4532.52 -109 120.8 33.6 0.0 6 194 >> 06:45:00 101591-713536 0.00 4517.16 -112 98.1 36.7 0.0 6 189 >> 07:00:00 101612-713564 0.00 4473.03 -112 108.3 48.8 0.0 7 223 >> 07:30:57 101624-713576 0.00 4453.09 -112 102.5 55.3 0.0 8 194 >> 07:48:35 101655-713598 0.00 4448.59 -112 90.5 56.9 0.0 8 192 >> 08:08:15 101373-713083 0.00 4518.06 -112 121.7 36.5 0.0 11 223 >> 08:30:00 102593-714402 0.00 4506.63 -112 72.8 39.4 0.0 10 199 >> 08:46:10 102330-714019 0.00 4428.88 -126 70.2 59.2 0.0 10 204 >> 09:03:49 102562-714388 0.00 4415.89 -136 65.0 60.1 0.0 11 209 >> 09:22:03 103203-714815 0.00 4501.47 -147 67.1 32.0 0.0 9 190 >> 09:41:20 103212-714824 0.00 4499.99 -147 59.6 32.3 0.0 11 217 >> 09:59:06 103199-714810 0.00 4490.41 -147 52.9 34.6 0.0 9 203 >> 10:15:00 100023-711320 0.00 4496.80 -147 92.9 33.1 0.0 10 220 >> 10:38:32 100023-711320 0.00 4476.27 -147 89.7 38.1 0.0 12 235 >> 10:59:29 100537-712280 0.00 4479.60 -147 61.0 37.3 0.0 11 231 >> 11:15:00 100537-712280 0.00 4469.02 -147 57.7 40.0 0.0 7 225 >> >> ---- Telescope Status ---- >> >> Status at: 12:47Z >> >> Telescope stowed at: 121.0, 30.0, 0.0 mount >> Instrument mounted: FPS >> Counterweights at: 647 655 653 651 >> LN2 autofill systems: Connected and turned on >> 180L LN2 dewar scales: SP1 196 lbs, 20 psi >> >> FPS Status >> ========== >> CP10 Loop = 7.42 L/min >> GFA Loop = 2.53 L/min >> Supply pressure = 44.5 psi >> Return pressure = 29.2 psi >> >> Chiller Status >> ============== >> Set temperarature = 3.6 C >> Flow = 4.9 Gpm >> >> ---- Software Used ---- >> >> STUI : 1.7.19 >> alerts : 2.3.0 >> boss : 6.0.2 >> -boss.specMec : 3.0.4 >> apo : 4.1.0 >> fliswarm : 0.5.1a0 >> jaeger : 1.9.1 >> kaiju : 1.4.0 >> coordio : 1.14.0 >> fps_calibrati : 2025.02.24 >> cherno : 1.1.0 >> astrometrynet : 0.93 >> hal : 1.4.1 >> hartmann : 2.0.2 >> hub : 5.0.1a0 >> mcp : 2.1 >> -mcp.plc : 35 >> -mcp.azFiduci : v1_150 >> -mcp.altFiduc : v1_150 >> -mcp.rotFiduc : v1_150 >> tcc : 1.4.0 >> apogee : not available >> apogeecal : not available >> apogeeql : v1_7_0dev >> STUIAdditions : 3.1.18 >> boson : 0.4.9 >> sdss-obstools : 3.9.3 >> TPM : 1.2-31-gd7eb >> MCP : 2.1-1-ge461 >> PLC : V35 >> Fiducials : v1_150 >> idlspec2d : v6_2_0 >> Kronos : 2.0.0 >> roboscheduler : 2.1.0 >> >> ---- APOGEE Data Summary ---- >> >> 60750 morning cals: >> >> num field design config exptype dithpix nread relflux okay >> dx >> dy sigx sigy fpi1 fpi2 >> -------- ----- ------ ------ ------------ ------- ----- ------- ----- >> ------- ------- ------ ------ ---- ---- >> 51880011 -999 -999 19918 DARK 13.5 60 -30 True >> -- -- -- -- -- -- >> 51880012 -999 -999 19918 DARK 13.5 60 -32 True >> -- -- -- -- -- -- >> 51880013 -999 -999 19918 DARK 13.5 60 -5 True >> -- -- -- -- -- -- >> 51880014 -999 -999 19918 QUARTZFLAT 13.5 10 104 True >> -- -0.17 -- 0.86 -- -- >> 51880015 -999 -999 19918 QUARTZFLAT 13.5 10 105 True >> -- -0.19 -- 0.86 -- -- >> 51880016 -999 -999 19918 QUARTZFLAT 13.5 10 105 True >> -- -0.19 -- 0.86 -- -- >> 51880017 -999 -999 19918 ARCLAMP-THAR 13.5 12 102 True >> 1.96 0.63 1.08 0.90 -- -- >> 51880018 -999 -999 19918 ARCLAMP-UNE 13.5 40 95 True >> 1.55 0.50 1.07 0.88 -- -- >> 51880019 -999 -999 19918 ARCLAMP-THAR 13.0 12 93 True >> 1.46 0.59 1.07 0.89 -- -- >> 51880020 -999 -999 19918 ARCLAMP-UNE 13.0 40 95 True >> 1.04 0.43 1.10 0.90 -- -- >> 51880021 -999 -999 19918 DARK 13.0 30 -20 False >> -- -- -- -- -- -- >> 51880022 -999 -999 19918 INTERNALFLAT 13.0 30 99 True >> -- -- -- -- -- -- >> 51880023 -999 -999 19918 INTERNALFLAT 13.0 30 100 True >> -- -- -- -- -- -- >> 51880024 -999 -999 19918 INTERNALFLAT 13.0 30 100 True >> -- -- -- -- -- -- >> 51880025 -999 -999 19918 DARK 13.0 30 240 True >> -- -- -- -- -- -- >> 51880026 -999 -999 19918 FPI 13.0 30 188 True >> 12.88 0.40 1.14 0.88 -- -- >> 51880027 -999 -999 19918 FPI 13.5 30 187 True >> -11.23 0.42 1.16 0.88 -- -- >> num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 >> mtp10 >> -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- >> ----- >> 51880014 QUARTZFLAT 10 65 69 80 89 90 84 87 88 88 >> 81 >> 51880015 QUARTZFLAT 10 68 73 81 90 90 84 87 89 88 >> 82 >> 51880016 QUARTZFLAT 10 69 74 81 90 91 85 88 89 89 >> 83 >> observers39) [observer@sdss5-hub: ~]$ check_ap 60751 >> Using nfowler=1 for DOMEFLAT >> Using nfowler=1 for DOMEFLAT >> Using nfowler=1 for DOMEFLAT >> Using nfowler=1 for DOMEFLAT >> Using nfowler=1 for DOMEFLAT >> Using nfowler=1 for DOMEFLAT >> num field design config exptype dithpix nread relflux okay >> dx >> dy sigx sigy fpi1 fpi2 >> -------- ------ ------ ------ ------------ ------- ----- ------- ---- >> ------- ------- ------ ------ ---- ---- >> 51890001 -999 -999 19918 DARK 13.5 3 100 True >> -- -- -- -- -- -- >> 51890002 -999 -999 19919 DARK 13.5 60 -18 True >> -- -- -- -- -- -- >> 51890003 -999 -999 19919 QUARTZFLAT 13.5 10 102 True >> -- -0.27 -- 0.86 -- -- >> 51890004 -999 -999 19919 ARCLAMP-THAR 13.5 12 99 True >> 1.98 0.55 1.07 0.89 -- -- >> 51890005 -999 -999 19919 ARCLAMP-UNE 13.5 40 96 True >> 1.56 0.44 1.08 0.88 -- -- >> 51890006 -999 -999 19919 ARCLAMP-THAR 13.0 12 90 True >> 1.47 0.52 1.05 0.88 -- -- >> 51890007 -999 -999 19919 ARCLAMP-UNE 13.0 40 96 True >> 1.07 0.37 1.11 0.91 -- -- >> 51890008 -999 -999 19919 DARK 13.0 60 -15 True >> -- -- -- -- -- -- >> 51890009 -999 -999 19919 FPI 13.0 30 196 True >> 12.91 0.33 1.18 0.89 -- -- >> 51890010 -999 -999 19919 FPI 13.5 30 199 True >> -11.21 0.33 1.17 0.88 -- -- >> 51890011 -999 -999 19919 DOMEFLAT 13.5 5 111 True >> -- -0.24 -- 0.86 -- -- >> 51890012 -999 -999 19919 DOMEFLAT 13.5 5 107 True >> -- -0.27 -- 0.87 -- -- >> 51890013 -999 -999 19919 DOMEFLAT 13.5 5 103 True >> -- -0.25 -- 0.85 -- -- >> 51890014 104002 715583 19920 OBJECT 13.0 35 86 True >> 1.47 -0.26 1.73 0.89 5 164 >> 51890015 104002 715583 19920 OBJECT 13.5 35 95 True >> 1.97 -0.24 1.76 0.84 5 163 >> 51890016 101951 713816 19921 OBJECT 13.0 43 90 True >> 1.45 -0.25 1.75 0.87 0 165 >> 51890017 101951 713816 19921 OBJECT 13.5 43 92 True >> 1.97 -0.25 1.75 0.85 0 166 >> 51890018 101951 713817 19922 OBJECT 13.0 43 91 True >> 1.47 -0.25 1.75 0.86 0 164 >> 51890019 101951 713817 19922 OBJECT 13.5 43 85 True >> 1.97 -0.25 1.75 0.84 0 162 >> 51890020 101951 713818 19923 OBJECT 13.0 43 78 True >> 1.45 -0.24 1.74 0.86 1 162 >> 51890021 101951 713818 19923 OBJECT 13.5 43 77 True >> 1.98 -0.24 1.75 0.85 1 164 >> 51890022 100862 712758 19924 OBJECT 13.0 35 66 True >> 1.46 -0.24 1.74 0.85 0 162 >> 51890023 100862 712758 19924 OBJECT 13.5 35 64 True >> 1.96 -0.25 1.75 0.84 0 162 >> 51890024 100875 712774 19925 OBJECT 13.0 35 63 True >> 1.48 -0.23 1.68 0.85 4 163 >> 51890025 100875 712774 19925 OBJECT 13.5 35 61 True >> 1.98 -0.24 1.68 0.85 4 161 >> 51890026 100864 712761 19926 OBJECT 13.0 35 51 True >> 1.46 -0.24 1.71 0.87 0 167 >> 51890027 100864 712761 19926 OBJECT 13.5 35 52 True >> 1.95 -0.24 1.60 0.86 0 162 >> 51890028 101322 713039 19928 OBJECT 13.0 35 53 True >> 1.45 -0.21 1.74 0.85 1 164 >> 51890029 101322 713039 19928 OBJECT 13.5 35 56 True >> 1.96 -0.22 1.74 0.84 0 163 >> 51890030 101344 713056 19929 OBJECT 13.0 35 60 True >> 1.48 -0.22 1.70 0.86 1 163 >> 51890031 101344 713056 19929 OBJECT 13.5 35 58 True >> 2.00 -0.22 1.66 0.85 1 162 >> 51890032 101612 713564 19931 OBJECT 13.0 35 52 True >> 1.47 -0.21 1.68 0.90 1 161 >> 51890033 101612 713564 19931 OBJECT 13.5 35 48 True >> 1.96 -0.21 1.67 0.86 0 161 >> 51890034 101624 713576 19932 OBJECT 13.0 35 40 True >> 1.45 -0.21 1.73 0.88 -- 161 >> 51890035 101624 713576 19932 OBJECT 13.5 35 40 True >> 1.96 -0.21 1.75 0.83 1 158 >> 51890036 101655 713598 19933 OBJECT 13.0 35 38 True >> 1.45 -0.21 1.75 0.86 0 163 >> 51890037 101655 713598 19933 OBJECT 13.5 35 37 True >> 1.98 -0.21 1.75 0.83 1 162 >> 51890038 101373 713083 19934 OBJECT 13.0 35 57 True >> 1.46 -0.20 1.70 0.86 1 154 >> 51890039 101373 713083 19934 OBJECT 13.5 35 57 True >> 1.97 -0.20 1.68 0.85 2 156 >> 51890040 102330 714019 19936 OBJECT 13.0 35 46 True >> 1.46 -0.20 1.74 0.85 2 155 >> 51890041 102330 714019 19936 OBJECT 13.5 35 47 True >> 1.95 -0.19 1.74 0.84 2 158 >> 51890042 102562 714388 19937 OBJECT 13.0 35 49 True >> 1.47 -0.20 1.69 0.89 2 158 >> 51890043 102562 714388 19937 OBJECT 13.5 35 46 True >> 1.97 -0.19 1.67 0.85 1 158 >> 51890044 103203 714815 19938 OBJECT 13.0 35 68 True >> 1.45 -0.19 1.75 0.86 2 155 >> 51890045 103203 714815 19938 OBJECT 13.5 35 66 True >> 1.96 -0.19 1.75 0.84 2 159 >> 51890046 103212 714824 19939 OBJECT 13.0 35 62 True >> 1.47 -0.19 1.70 0.87 1 156 >> 51890047 103212 714824 19939 OBJECT 13.5 35 61 True >> 1.97 -0.19 1.69 0.85 1 154 >> 51890048 103199 714810 19940 OBJECT 13.0 35 63 True >> 1.47 -0.19 1.71 0.85 4 158 >> 51890049 103199 714810 19940 OBJECT 13.5 35 61 True >> 1.98 -0.19 1.69 0.84 3 159 >> 51890050 100023 711320 19941 OBJECT 13.5 35 74 True >> 1.96 -0.18 1.74 0.84 1 151 >> 51890051 100023 711320 19941 OBJECT 13.0 35 65 True >> 1.45 -0.18 1.74 0.86 1 151 >> 51890052 100537 712280 19942 OBJECT 13.0 35 82 True >> 1.46 -0.19 1.71 0.89 0 153 >> 51890053 100537 712280 19942 OBJECT 13.5 35 78 True >> 1.96 -0.19 1.71 0.85 0 153 >> 51890054 100537 712280 19942 OBJECT 13.5 35 68 True >> 1.97 -0.18 1.74 0.84 1 151 >> 51890055 100537 712280 19942 OBJECT 13.0 35 70 True >> 1.46 -0.18 1.73 0.89 0 157 >> 51890056 103228 714866 19943 OBJECT 13.0 38 78 True >> 1.46 -0.18 1.75 0.89 0 155 >> 51890057 103228 714866 19943 OBJECT 13.5 38 77 True >> 1.98 -0.18 1.75 0.84 0 153 >> 51890058 103228 714866 19943 OBJECT 13.5 35 81 True >> 1.96 -0.18 1.75 0.84 1 150 >> 51890059 103228 714866 19943 OBJECT 13.0 35 80 True >> 1.45 -0.17 1.75 0.87 1 155 >> 51890060 103239 715007 19944 OBJECT 13.0 35 78 True >> 1.45 -0.17 1.73 0.88 2 153 >> 51890061 103239 715007 19944 OBJECT 13.5 35 80 True >> 1.96 -0.17 1.75 0.84 1 153 >> 51890062 103239 715007 19944 DOMEFLAT 13.5 5 110 True >> -- -0.19 -- 0.86 -- -- >> 51890063 103239 715007 19944 DOMEFLAT 13.5 5 110 True >> -- -0.15 -- 0.85 -- -- >> 51890064 103239 715007 19944 DOMEFLAT 13.5 5 109 True >> -- -0.17 -- 0.84 -- -- >> num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 >> mtp10 >> -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- >> ----- >> 51890003 QUARTZFLAT 10 65 70 81 90 90 85 87 89 88 >> 81 >> 51890011 DOMEFLAT 5 78 87 97 103 109 97 103 105 101 >> 97 >> 51890012 DOMEFLAT 5 78 86 95 100 106 94 100 102 98 >> 96 >> 51890013 DOMEFLAT 5 76 85 93 98 104 93 98 100 97 >> 95 >> 51890062 DOMEFLAT 5 77 85 94 101 108 97 103 107 101 >> 97 >> 51890063 DOMEFLAT 5 78 87 94 101 108 97 103 106 100 >> 97 >> 51890064 DOMEFLAT 5 79 87 96 102 109 98 103 107 101 >> 97 >> >> ---- BOSS Data Summary ---- >> >> Reading FITS from /data/spectro//60751 >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - >> - - >> n Time Filename design-field cfg flav shu Exp colA colB >> colC Hrt >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - >> - - >> 1 00:18 b1-00383029 ----------- ---- bia Clo 0.0 -4969 -4847 >> -4881 >> 2 00:18 r1-00383029 ----------- ---- bia Clo 0.0 -4969 -4847 >> -4881 >> 3 00:35 b1-00383030 -999--999 19919 arc Lef 4.1 -4969 -4847 -4881 >> HART >> 4 00:35 r1-00383030 -999--999 19919 arc Lef 4.1 -4969 -4847 -4881 >> HART >> 5 00:35 r1-00383031 -999--999 19919 arc Rig 4.1 -4969 -4847 -4881 >> HART >> 6 00:35 b1-00383031 -999--999 19919 arc Rig 4.1 -4969 -4847 -4881 >> HART >> 7 00:37 b1-00383032 -999--999 19919 arc Lef 4.1 -3153 -3032 -3065 >> HART >> 8 00:37 r1-00383032 -999--999 19919 arc Lef 4.1 -3153 -3032 -3065 >> HART >> 9 00:37 b1-00383033 -999--999 19919 arc Rig 4.1 -3153 -3032 -3065 >> HART >> 10 00:37 r1-00383033 -999--999 19919 arc Rig 4.1 -3153 -3032 -3065 >> HART >> 11 00:43 r1-00383034 -999--999 19919 arc Out 4.1 -3290 -3168 -3202 >> 12 00:43 b1-00383034 -999--999 19919 arc Out 4.1 -3290 -3168 -3202 >> 13 00:44 b1-00383035 -999--999 19919 fla Out 25.1 -3290 -3168 -3202 >> 14 00:44 r1-00383035 -999--999 19919 fla Out 25.1 -3290 -3168 -3202 >> 15 01:38 b1-00383036 -999--999 19919 arc Lef 4.1 -3290 -3168 -3202 >> HART >> 16 01:38 r1-00383036 -999--999 19919 arc Lef 4.1 -3290 -3168 -3202 >> HART >> 17 01:39 b1-00383037 -999--999 19919 arc Rig 4.1 -3290 -3168 -3202 >> HART >> 18 01:39 r1-00383037 -999--999 19919 arc Rig 4.1 -3290 -3168 -3202 >> HART >> 19 01:42 r1-00383038 -999--999 19919 fla Out 25.1 -1964 -1843 -1876 >> 20 01:42 b1-00383038 -999--999 19919 fla Out 25.1 -1964 -1843 -1876 >> 21 01:44 r1-00383039 -999--999 19919 fla Out 25.1 -1964 -1843 -1876 >> 22 01:44 b1-00383039 -999--999 19919 fla Out 25.1 -1964 -1843 -1876 >> 23 01:45 r1-00383040 -999--999 19919 fla Out 25.1 -1964 -1843 -1876 >> 24 01:45 b1-00383040 -999--999 19919 fla Out 25.1 -1964 -1843 -1876 >> 25 01:54 r1-00383041 715583-104002 19920 arc Out 4.1 -1964 -1843 >> -1876 >> 26 01:54 b1-00383041 715583-104002 19920 arc Out 4.1 -1964 -1843 >> -1876 >> 27 01:56 b1-00383042 715583-104002 19920 sci Out 730.1 -1964 -1843 >> -1876 >> 28 01:56 r1-00383042 715583-104002 19920 sci Out 730.1 -1964 -1843 >> -1876 >> 29 02:13 r1-00383043 713816-101951 19921 fla Out 25.1 -1964 -1843 >> -1876 >> 30 02:13 b1-00383043 713816-101951 19921 fla Out 25.1 -1964 -1843 >> -1876 >> 31 02:15 r1-00383044 713816-101951 19921 arc Out 4.1 -1964 -1843 >> -1876 >> 32 02:15 b1-00383044 713816-101951 19921 arc Out 4.1 -1964 -1843 >> -1876 >> 33 02:18 r1-00383045 713816-101951 19921 sci Out 900.1 -1964 -1843 >> -1876 >> 34 02:18 b1-00383045 713816-101951 19921 sci Out 900.1 -1964 -1843 >> -1876 >> 35 02:38 b1-00383046 713817-101951 19922 sci Out 900.1 -1964 -1843 >> -1876 >> 36 02:38 r1-00383046 713817-101951 19922 sci Out 900.1 -1964 -1843 >> -1876 >> 37 02:57 r1-00383047 713818-101951 19923 sci Out 900.1 -1964 -1843 >> -1876 >> 38 02:57 b1-00383047 713818-101951 19923 sci Out 900.1 -1964 -1843 >> -1876 >> 39 03:17 r1-00383048 712758-100862 19924 arc Out 4.1 -1964 -1843 >> -1876 >> 40 03:17 b1-00383048 712758-100862 19924 arc Out 4.1 -1964 -1843 >> -1876 >> 41 03:19 r1-00383049 712758-100862 19924 sci Out 730.1 -1964 -1843 >> -1876 >> 42 03:19 b1-00383049 712758-100862 19924 sci Out 730.1 -1964 -1843 >> -1876 >> 43 03:35 r1-00383050 712774-100875 19925 arc Out 4.1 -1964 -1843 >> -1876 >> 44 03:35 b1-00383050 712774-100875 19925 arc Out 4.1 -1964 -1843 >> -1876 >> 45 03:37 r1-00383051 712774-100875 19925 sci Out 730.1 -1964 -1843 >> -1876 >> 46 03:37 b1-00383051 712774-100875 19925 sci Out 730.1 -1964 -1843 >> -1876 >> 47 03:54 r1-00383052 712761-100864 19926 arc Out 4.1 -1964 -1843 >> -1876 >> 48 03:54 b1-00383052 712761-100864 19926 arc Out 4.1 -1964 -1843 >> -1876 >> 49 03:55 b1-00383053 712761-100864 19926 sci Out 730.1 -1964 -1843 >> -1876 >> 50 03:55 r1-00383053 712761-100864 19926 sci Out 730.1 -1964 -1843 >> -1876 >> 51 06:08 b1-00383054 713039-101322 19928 arc Out 4.1 -1964 -1843 >> -1876 >> 52 06:08 r1-00383054 713039-101322 19928 arc Out 4.1 -1964 -1843 >> -1876 >> 53 06:10 b1-00383055 713039-101322 19928 sci Out 730.1 -1964 -1843 >> -1876 >> 54 06:10 r1-00383055 713039-101322 19928 sci Out 730.1 -1964 -1843 >> -1876 >> 55 06:26 b1-00383056 713056-101344 19929 arc Out 4.1 -1964 -1843 >> -1876 >> 56 06:26 r1-00383056 713056-101344 19929 arc Out 4.1 -1964 -1843 >> -1876 >> 57 06:28 r1-00383057 713056-101344 19929 sci Out 730.1 -1964 -1843 >> -1876 >> 58 06:28 b1-00383057 713056-101344 19929 sci Out 730.1 -1964 -1843 >> -1876 >> 59 06:58 r1-00383058 713564-101612 19931 arc Out 4.1 -1964 -1843 >> -1876 >> 60 06:58 b1-00383058 713564-101612 19931 arc Out 4.1 -1964 -1843 >> -1876 >> 61 07:00 r1-00383059 713564-101612 19931 sci Out 730.1 -1964 -1843 >> -1876 >> 62 07:00 b1-00383059 713564-101612 19931 sci Out 730.1 -1964 -1843 >> -1876 >> 63 07:29 r1-00383060 713576-101624 19932 arc Out 4.1 -1964 -1843 >> -1876 >> 64 07:29 b1-00383060 713576-101624 19932 arc Out 4.1 -1964 -1843 >> -1876 >> 65 07:31 r1-00383061 713576-101624 19932 sci Out 730.1 -1964 -1843 >> -1876 >> 66 07:31 b1-00383061 713576-101624 19932 sci Out 730.1 -1964 -1843 >> -1876 >> 67 07:47 r1-00383062 713598-101655 19933 arc Out 4.1 -1964 -1843 >> -1876 >> 68 07:47 b1-00383062 713598-101655 19933 arc Out 4.1 -1964 -1843 >> -1876 >> 69 07:48 b1-00383063 713598-101655 19933 sci Out 730.1 -1964 -1843 >> -1876 >> 70 07:48 r1-00383063 713598-101655 19933 sci Out 730.1 -1964 -1843 >> -1876 >> 71 08:06 b1-00383064 713083-101373 19934 arc Out 4.1 -1964 -1843 >> -1876 >> 72 08:06 r1-00383064 713083-101373 19934 arc Out 4.1 -1964 -1843 >> -1876 >> 73 08:08 b1-00383065 713083-101373 19934 sci Out 730.1 -1964 -1843 >> -1876 >> 74 08:08 r1-00383065 713083-101373 19934 sci Out 730.1 -1964 -1843 >> -1876 >> 75 08:38 b1-00383066 714019-102330 19936 arc Out 4.1 -1964 -1843 >> -1876 >> 76 08:38 r1-00383066 714019-102330 19936 arc Out 4.1 -1964 -1843 >> -1876 >> 77 08:40 b1-00383067 714019-102330 19936 sci Out 730.1 -1964 -1843 >> -1876 >> 78 08:40 r1-00383067 714019-102330 19936 sci Out 730.1 -1964 -1843 >> -1876 >> 79 08:56 b1-00383068 714388-102562 19937 arc Out 4.1 -1964 -1843 >> -1876 >> 80 08:56 r1-00383068 714388-102562 19937 arc Out 4.1 -1964 -1843 >> -1876 >> 81 08:57 b1-00383069 714388-102562 19937 sci Out 730.1 -1964 -1843 >> -1876 >> 82 08:57 r1-00383069 714388-102562 19937 sci Out 730.1 -1964 -1843 >> -1876 >> 83 09:13 r1-00383070 714815-103203 19938 arc Out 4.1 -1964 -1843 >> -1876 >> 84 09:13 b1-00383070 714815-103203 19938 arc Out 4.1 -1964 -1843 >> -1876 >> 85 09:16 b1-00383071 714815-103203 19938 sci Out 730.1 -1964 -1843 >> -1876 >> 86 09:16 r1-00383071 714815-103203 19938 sci Out 730.1 -1964 -1843 >> -1876 >> 87 09:31 b1-00383072 714824-103212 19939 arc Lef 4.1 -1964 -1843 >> -1876 HART >> 88 09:31 r1-00383072 714824-103212 19939 arc Lef 4.1 -1964 -1843 >> -1876 HART >> 89 09:32 b1-00383073 714824-103212 19939 arc Rig 4.1 -1964 -1843 >> -1876 HART >> 90 09:32 r1-00383073 714824-103212 19939 arc Rig 4.1 -1964 -1843 >> -1876 HART >> 91 09:33 r1-00383074 714824-103212 19939 arc Out 4.1 -778 -657 >> -690 >> 92 09:33 b1-00383074 714824-103212 19939 arc Out 4.1 -778 -657 >> -690 >> 93 09:35 r1-00383075 714824-103212 19939 sci Out 730.1 -778 -657 >> -690 >> 94 09:35 b1-00383075 714824-103212 19939 sci Out 730.1 -778 -657 >> -690 >> 95 09:51 r1-00383076 714810-103199 19940 arc Out 4.1 -778 -657 >> -690 >> 96 09:51 b1-00383076 714810-103199 19940 arc Out 4.1 -778 -657 >> -690 >> 97 09:53 r1-00383077 714810-103199 19940 sci Out 730.1 -778 -657 >> -690 >> 98 09:53 b1-00383077 714810-103199 19940 sci Out 730.1 -778 -657 >> -690 >> 99 10:10 r1-00383078 711320-100023 19941 arc Out 4.1 -778 -657 >> -690 >> 100 10:10 b1-00383078 711320-100023 19941 arc Out 4.1 -778 -657 >> -690 >> 101 10:38 r1-00383079 711320-100023 19941 sci Out 730.1 -778 -657 >> -690 >> 102 10:38 b1-00383079 711320-100023 19941 sci Out 730.1 -778 -657 >> -690 >> 103 10:57 r1-00383080 712280-100537 19942 arc Out 4.1 -778 -657 >> -690 >> 104 10:57 b1-00383080 712280-100537 19942 arc Out 4.1 -778 -657 >> -690 >> 105 10:59 r1-00383081 712280-100537 19942 sci Out 730.1 -778 -657 >> -690 >> 106 10:59 b1-00383081 712280-100537 19942 sci Out 730.1 -778 -657 >> -690 >> 107 11:13 r1-00383082 712280-100537 19942 sci Out 730.1 -778 -657 >> -690 >> 108 11:13 b1-00383082 712280-100537 19942 sci Out 730.1 -778 -657 >> -690 >> 109 11:29 r1-00383083 714866-103228 19943 arc Out 4.1 -778 -657 >> -690 >> 110 11:29 b1-00383083 714866-103228 19943 arc Out 4.1 -778 -657 >> -690 >> 111 11:31 b1-00383084 714866-103228 19943 sci Out 730.1 -778 -657 >> -690 >> 112 11:31 r1-00383084 714866-103228 19943 sci Out 730.1 -778 -657 >> -690 >> 113 11:44 r1-00383085 714866-103228 19943 sci Out 730.1 -778 -657 >> -690 >> 114 11:44 b1-00383085 714866-103228 19943 sci Out 730.1 -778 -657 >> -690 >> 115 12:00 b1-00383086 715007-103239 19944 arc Out 4.1 -778 -657 >> -690 >> 116 12:00 r1-00383086 715007-103239 19944 arc Out 4.1 -778 -657 >> -690 >> 117 12:02 r1-00383087 715007-103239 19944 sci Out 730.1 -778 -657 >> -690 >> 118 12:02 b1-00383087 715007-103239 19944 sci Out 730.1 -778 -657 >> -690 >> 119 12:24 r1-00383088 715007-103239 19944 fla Out 25.1 -778 -657 >> -690 >> 120 12:24 b1-00383088 715007-103239 19944 fla Out 25.1 -778 -657 >> -690 >> 121 12:26 r1-00383089 715007-103239 19944 arc Out 4.1 -778 -657 >> -690 >> 122 12:26 b1-00383089 715007-103239 19944 arc Out 4.1 -778 -657 >> -690 >> >> ---- BOSS Spectrographs Collimation ---- >> >> Afternoon focus check >> ===================== >> 00:37:08 BOSS expNum=383032, expTime= 4.0s >> 00:37:49 BOSS expNum=383033, expTime= 4.0s >> 00:38:30 sp1.motor.move= -137, -136, -137 >> 00:38:30 Hartmann collimate output >> sp1: offset: r= -659 (In Focus); b= -1.2 (In Focus) >> sp1: pred. move: spAverageMove= -137 >> sp1: pred. resid: r= -522, b= -1.6, txt= OK, spTemp = 3.2 >> >> Night >> ===== >> Time TSP1 R1 B1 Move1 B1Resid >> ============================================= >> 01:40:00 3.0 659 -6.2 1325 -2.1 >> 09:33:00 3.1 2308 -0.2 1186 3.5 >> >> ---- Weather Log ---- >> >> Time Field-Design Temp DP Diff Humid Wind Dir 1μm Dust IRSC >> σ >> IRSC μ >> >> ================================================================================ >> 02:15:00 101951-713816 0.0 0.0 0.0 100.0 5.0 347 nan -999.0 >> -999 >> 02:33:11 101951-713816 0.0 0.0 0.0 100.0 6.6 351 nan -999.0 >> -999 >> 02:53:47 101951-713817 0.0 0.0 0.0 100.0 6.7 350 nan -999.0 >> -999 >> 03:13:06 101951-713818 0.0 0.0 0.0 100.0 7.4 322 nan -999.0 >> -999 >> 03:30:00 100862-712758 0.0 0.0 0.0 100.0 7.5 324 nan -999.0 >> -999 >> 03:49:54 100875-712774 0.0 0.0 0.0 100.0 6.1 339 nan -999.0 >> -999 >> 04:08:10 100864-712761 0.0 0.0 0.0 100.0 8.9 312 nan -999.0 >> -999 >> 06:10:03 101322-713039 0.0 0.0 0.0 100.0 6.6 213 nan -999.0 >> -999 >> 06:28:15 101344-713056 0.0 0.0 0.0 100.0 5.8 194 nan -999.0 >> -999 >> 06:45:00 101591-713536 0.0 0.0 0.0 100.0 5.6 189 nan -999.0 >> -999 >> 07:00:00 101612-713564 0.0 0.0 0.0 100.0 7.2 223 nan -999.0 >> -999 >> 07:30:57 101624-713576 0.0 0.0 0.0 100.0 8.4 194 nan -999.0 >> -999 >> 07:48:35 101655-713598 0.0 0.0 0.0 100.0 8.1 192 nan -999.0 >> -999 >> 08:08:15 101373-713083 0.0 0.0 0.0 100.0 11.2 223 nan -999.0 >> -999 >> 08:30:00 102593-714402 0.0 0.0 0.0 100.0 9.9 199 nan 14.8 >> 348 >> 08:46:10 102330-714019 0.0 0.0 0.0 100.0 10.3 204 nan 3.3 >> 347 >> 09:03:49 102562-714388 0.0 0.0 0.0 100.0 10.8 209 nan 1.4 >> 348 >> 09:22:03 103203-714815 0.0 0.0 0.0 100.0 9.1 190 nan 2.1 >> 347 >> 09:41:20 103212-714824 0.0 0.0 0.0 100.0 11.0 217 nan 9.1 >> 348 >> 09:59:06 103199-714810 0.0 0.0 0.0 100.0 8.7 203 nan 7.8 >> 347 >> 10:15:00 100023-711320 0.0 0.0 0.0 100.0 10.2 220 nan 5.7 >> 348 >> 10:38:32 100023-711320 0.0 0.0 0.0 100.0 12.4 235 nan 34.5 >> 352 >> 10:59:29 100537-712280 0.0 0.0 0.0 100.0 11.0 231 nan 33.6 >> 347 >> 11:15:00 100537-712280 0.0 0.0 0.0 100.0 7.4 225 nan 32.3 >> 351 >> >> ---- Time Tracking ---- >> >> *=*=*= Observatory: APO >> *=*=*= MJD: 60751 >> Fill night time in Hours below >> *=*=*= Total bright time : 8.7 >> *=*=*= Bright time, queued observations : 6.7 >> *=*=*= Bright time, commissioning/engineering: 0 >> *=*=*= Bright time lost to weather : 0 >> *=*=*= Bright time lost to technical issues : 2.0 >> >> *=*=*= Total dark time : 1.3 >> *=*=*= Dark time, queued observations : 1.3 >> *=*=*= Dark time, commissioning/engineering : 0 >> *=*=*= Dark time lost to weather : 0 >> *=*=*= Dark time lost to technical issues : 0 >> >> _______________________________________________ >> Logs of Nightly APO Activities >> >> https://urldefense.com/v3/__https://mailman.sdss.org/archives/list/sloan-25mlog@sdss.org/message/PPXRANKGN3C6ZPLE2ZGSTEAU4P7D5MFZ/__;!!K-Hz7m0Vt54!hiOkMgPPBQIIOWUKC3xITnJsz1gZfSAsvdHvgtpCub-fsNtV9mqpr4J_sCEtXJb0kIbHtySYNvFi7BOluzduZA$ >> >> >