Date: Tue, 25 Jun 2024 06:52:07 -0600 Subject: Re: [sloan-25mlog 7307] 2.5m obslog 20240624 (SJD 60486) From: John Downey Great work on closing up Dan, thanks for saving me a trip in on very little sleep! Thinking about this problem a bit more, I'm also concerned that it may be a failure deeper in the PLC rather than at the SLC500 card in the MCP.  I think the stop button lights should come on when the button is activated even without the MCP communications. First thing I'm going to confirm when I get in this morning, and fortunately it should be fairly easy to evaluate on a first pass. I'm also approaching this cautiously on the PLC side.  There were a lot of changes made in 2021 and I don't know the state of back ups.  I need to poke around in Ryan's computer and see if I can get his PLC interface system running before trying to copy the modified files over to the ancient engineering laptop. On 6/25/24 04:00, doravetz--- via sloan-25mlog wrote: > APACHE POINT OBSERVATORY SDSS 2.5M > > OBSERVING LOG > Monday June 24, 2024 > > (SJD 60486) > > ---- Observing Team ---- > > Dan Oravetz > John Downey > > ---- Observing Plan ---- > > Mon 24 Jun 2024 60486 dark until 04:58 (1.4 h), then bright (6.6 h, w/1.0 h for 8 deg design) > M107 BOSS Calibration Cluster designs > > ---- Observing Summary ---- > > The first few hours of the night were lost to high dust, clouds, and radar returns/virga. The dust settled down first, then the clouds and radar returns cleared out awhile later and we opened at 04:30Z to a mostly-clear sky in time to squeeze-in one RM design before bright time started. After observing bright_time for about 20 minutes, clouds started moving back in from the north around 05:30Z and grew progressively worse. > > When I made the decision it was time to close for clouds at ~06:10Z, I noticed the AZ, ALT, and ROT axes all reported an out-of-closed-loop state and I was unable to command the telescope to move with either the MCP or TCC. At the same time, several other MCP/PLC-related items weren't working: e.g., E-stops didn't engage or light-up, the FFS were closed, but reported an 'Open' state, arc/FF lamps wouldn't turn on when commanded, etc. And there was an error message about an error reading the SLC (PLC). > > After calling John, we tried an MCP restart before ultimately closing for the night using the hand pendant/windbaffle and the building-interlock bypass. As of now the problem appears to be the SLC. There are more play-by-play details in the Log body. > > Note below the same bright_time design was observed. This feels like a problem with Kronos because it never displayed 621638 as having been loaded, so I think the HAL auto-pilot just kept observing it. I caught this during the second clone and let it finish those exposures. Details in the Software Problems section and the Log body. > > Time Field-Design-Conf DesignType BOSS/APOGEE Seeing Overhead Complete? Comment > 04:45:04 112359-403333-15024 dark_rm 1x900/1xAB 1.5" 0:00 DONE > 05:09:48 103240-621638-15025 bright_time 1x730/1xAB 2.0" 6:23 DONE > 05:28:20 103240-621638-15026 bright_time 1x730/1xAB 1.6" 1:43 DONE > 05:42:29 103240-621638-15027 bright_time 1x730/1xAB 2.3" 1:39 DONE > > Notes > ===== > (1) Saved new NOMINAL MIG values this afternoon. > > (2) I didn't get a before-closing set of APOGEE dome flats because the FF lamps couldn't be commanded on given the state of the SLC error. > > (3) 1.0 um Integrated Dust Counts: ~4100 dust-hrs > > ---- Engineering Issues ---- > > SLC problem. Details in Log body. > > ---- Software Issues ---- > > Kronos problem: > > After the RM design completed, I loaded the first bright_time design (621638) and then engaged the HAL auto-pilot. Everything seemed to work fine and the design was exposed, but later on the same design was loaded again (cloned) and then once more (another clone). I noticed this during the second clone after coming back in after checking the sky for incoming clouds and then let the exposures complete and for a third time, HAL had loaded a cloned design, so I stopped auto-pilot at once. > > 05:57Z - When I looked back at the Kronos Plan Observing page, the Obs Queue was still showing 621638 without any strike-through text, so I get the feeling HAL was just loading what it was told hadn't yet been loaded by Kronos (?). > > I clicked on the Plan Observing tab (didn't refresh the whole page) and then Kronos presented the "Congratulations you found a bug!" message. I assume that this bug was related to 621638 not properly indicating it had been loaded at any point. > > At this point I reloaded the page with the browser refresh button, but Kronos still put 621638 at the top of the queue. I manually 'X'-ed it from the queue and then preloaded the next design (613728). > > I issued the goto-field for 613728, but it was getting very cloudy, so took a quick look at clouds outside again and we ended-up not observing this design because it was too cloudy. > > However, Kronos was now showing 613728 (which was loaded) without strike-through text as if it also hadn't been loaded. > > Again, the Obs Queue was definitely not properly displaying the status of the loaded bright_time designs tonight, but it did properly display the RM design when it had loaded at the beginning of the night. > > ---- Observing Log ---- > > ============================================================================ > AFTERNOON CHECKOUT > ============================================================================ > > All systems alive and responsive. > > Disk space: > > Filesystem Size Used Avail Use% Mounted on > /dev/sdb1 2.0T 464G 1.6T 23% /home > /dev/sdd1 22T 15T 5.9T 72% /data > > FPS powered on. Positioners are home/folded: > > 23:23:20 APO.doravetz_3 30 jaeger i folded=T > 23:23:20 APO.doravetz_3 30 jaeger : text="All positioners are folded." > > Loaded a manual folded configuration: > > 23:23:36 CmdStarted: APO.doravetz_3 31 jaeger configuration load --from-positions > 23:23:39 APO.doravetz_3 31 jaeger d text="Resetting cherno offsets." > 23:23:39 APO.doravetz_3 31 jaeger d text="Focal plane scale: 1.0." > 23:23:39 APO.doravetz_3 31 jaeger d configuration_loaded=15023,-999,-999,-999.0,-999.0,0,-999.0,-999.0,/home/sdss5/software/sdsscore/main/apo/summary_files/0150XX/confSummary-15023.par,F > 23:23:40 .jaeger 0 jaeger d text="Setting chiller to 18.0 C" > 23:23:43 APO.doravetz_3 31 jaeger i configuration_snapshot=/data/logs/jaeger/configuration_snapshots/60486/configuration_snapshot_60486_15023.pdf > 23:23:43 APO.doravetz_3 31 jaeger d design_preloaded=-999 > 23:23:43 APO.doravetz_3 31 jaeger : text="Configuration 15023 loaded." > 23:23:43 CmdDone: APO.doravetz_3 31 jaeger configuration load --from-positions > > Took a '--no-apply' FVC image. > 23:24:39 FVC Loop > fvc_rms= 40.71 > fvc_deltarms= 20.97 > fvc_filename=/data/fcam/60486/fimg-fvc1n-0001.fits > fvc_perc_90= 60.50 > fvc_percent_reached= 2.20 > fvc iterations= 1 > > Scattered-light check: > > 23:25:13 CmdStarted: APO.doravetz_3 33 fliswarm talk -n fvc expose 50 > > fimg-fvc1n-0002.fits shows no evidence of light artifacts. > > Exercised positioners: > > 23:26:25 CmdStarted: APO.doravetz_3 34 jaeger configuration random --danger > 23:28:13 APO.doravetz_3 34 jaeger d trajectory_dump_file=/data/logs/jaeger/trajectories/60485/trajectory-60485-0028.json > 23:28:14 CmdDone: APO.doravetz_3 34 jaeger configuration random --danger > 23:28:18 .jaeger 0 jaeger i snapshot=/data/logs/jaeger/snapshots/60485/fps_snapshot_60485_0028.pdf > > 23:29:35 CmdStarted: APO.doravetz_3 35 jaeger configuration reverse > > GFA biases/darks: > > 23:32:37 CmdStarted: APO.doravetz_3 36 fliswarm talk -c gfa expose --bias --count 10 > 23:33:39 CmdStarted: APO.doravetz_3 37 fliswarm talk -c gfa expose --dark --count 5 5 > > Ventilation delayed due to dust, clouds, and radar returns. > > ============================================================================ > NIGHT > ============================================================================ > > Opening delayed due to dust, clouds, and radar returns. > > 04:31:33 2.5m Enclosure: OPEN. BOSS LN2 dewars have already been force-filled. Mostly clear. > > APOGEE dome flats: > > 04:33:43 APOGEE expNum=49240011, reads=5, dithPos=B, DomeFlat > 04:35:03 APOGEE expNum=49240012, reads=5, dithPos=B, DomeFlat > 04:36:23 APOGEE expNum=49240013, reads=5, dithPos=B, DomeFlat > > BOSS hartmanns: > > 04:38:11 BOSS expNum=370589, expTime= 4.0s > 04:38:51 BOSS expNum=370590, expTime= 4.0s > 04:39:34 sp1.motor.move= 1160, 1160, 1160 > 04:39:34 Hartmann collimate output > sp1: offset: r= 329 (In Focus); b= -6.2 (Out of focus) > sp1: pred. move: spAverageMove= 1160 > sp1: pred. resid: r= -831, b= -2.6, txt= OK, spTemp = 10.2 > > BOSS flats: > > 04:40:25 BOSS expNum=370591, expTime= 25.0s > 04:42:23 BOSS expNum=370592, expTime= 25.0s > 04:44:11 BOSS expNum=370593, expTime= 25.0s > > **************************************************************** > 04:45:06 Field-Design-Configuration: 112359-403333-15024 > **************************************************************** > 04:45:46 hal goto-field: slew, reconfigure, fvc, reslew, flat, arc, acquire, guide > 04:45:50 CmdStarted: jaeger configuration execute > 04:45:54 CmdStarted: tcc track (slew begins) > 04:46:39 CmdDone: jaeger configuration execute > 04:46:55 CmdDone: Slew finished ('tcc track' command finished) > 04:46:56 CmdStarted: FVC loop > 04:48:03 CmdDone: FVC loop > FVC loop results: > fvc_rms= 6.78 > fvc_deltarms= 43.82 > fvc_filename=/data/fcam/60486/fimg-fvc1n-0004.fits > fvc_perc_90= 10.00 > fvc_percent_reached= 54.80 > fvc iterations= 2 > 04:48:04 CmdStarted: tcc track (slew begins) > 04:48:12 CmdDone: Slew finished ('tcc track' command finished) > 04:48:31 BOSS expNum=370594, expTime= 25.0s > 04:49:00 HgCd lamps turned ON > 04:49:00 Ne lamps turned ON > 04:51:08 BOSS expNum=370595, expTime= 4.0s > 04:51:17 HgCd lamps turned OFF > 04:51:17 Ne lamps turned OFF > 04:51:26 CmdStarted: cherno acquire > 04:53:47 CmdDone: cherno acquire > 04:53:49 CmdStarted: 'cherno guide' > 04:53:59 hal expose: BOSS science, APOGEE science > 04:54:29 APOGEE expNum=49240014, reads=43, dithPos=B, Object > 04:54:35 BOSS expNum=370596, expTime= 900.0s > 05:02:15 APOGEE expNum=49240015, reads=43, dithPos=A, Object > > **************************************************************** > 05:09:49 Field-Design-Configuration: 103240-621638-15025 > **************************************************************** > 05:09:55 hal goto-field: slew, reconfigure, fvc, reslew, arc, acquire, guide > 05:09:59 CmdStarted: jaeger configuration reverse > 05:10:03 CmdStarted: tcc track (slew begins) > 05:10:06 HgCd lamps turned ON > 05:10:48 CmdDone: jaeger configuration reverse > 05:10:48 CmdStarted: jaeger configuration execute > 05:12:53 FVC Loop > fvc_rms= 10.57 > fvc_deltarms= 45.67 > fvc_filename=/data/fcam/60486/fimg-fvc1n-0006.fits > fvc_perc_90= 10.70 > fvc_percent_reached= 53.40 > fvc iterations= 2 > 05:13:37 BOSS expNum=370597, expTime= 4.0s > 05:15:23 New-field overhead: 0:05:33 > 05:15:23 HAL expose: BOSS science, APOGEE science > 05:15:52 APOGEE expNum=49240016, reads=35, dithPos=A, Object > 05:15:57 BOSS expNum=370598, expTime= 730.0s > 05:22:08 APOGEE expNum=49240017, reads=35, dithPos=B, Object > > Looks like clouds to the north are going to be encroaching soon. A look outside shows a pretty good sky for the time being to the south and southwest, but the satellite and the first-person view of the clouds with the moonlight to help illuminate clouds doesn't look promising for a whole lot longer. > > **************************************************************** > 05:28:20 Field-Design-Configuration: 103240-621638-15026 > **************************************************************** > 05:28:26 HAL auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup > 05:28:26 HAL auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup > 05:29:15 HAL expose: BOSS science, APOGEE science > 05:29:51 BOSS expNum=370599, expTime= 730.0s > 05:29:54 APOGEE expNum=49240018, reads=35, dithPos=A, Object > 05:36:10 APOGEE expNum=49240019, reads=35, dithPos=B, Object > 05:42:24 HAL auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup > > **************************************************************** > 05:42:29 Field-Design-Configuration: 103240-621638-15027 > **************************************************************** > 05:43:09 HAL expose: BOSS science, APOGEE science > 05:43:41 BOSS expNum=370600, expTime= 730.0s > 05:43:43 APOGEE expNum=49240020, reads=35, dithPos=A, Object > 05:50:00 APOGEE expNum=49240021, reads=35, dithPos=B, Object > 05:56:13 HAL auto-pilot: auto_pilot, prepare, load, goto_field, expose, cleanup > > I noticed the same bright_time design (621638) has been observed/cloned twice since the first observation. I let the exposures finish on this second clone and then watched to see what would happen on the subsequent preload/load. > > The same design was once again loaded/cloned, so I stopped the auto-pilot loop at once. > > 05:57Z - When I looked back at the Kronos Plan Observing page, the Obs Queue was still showing 621638 without any strike-through text. I clicked on the Plan Observing tab (didn't refresh the whole page) and then Kronos presented the "Congratulations you found a bug!" message. > > At this point I reloaded the page with the browser refresh button, but Kronos still put 621638 at the top of the queue. I manually 'X'-ed it from the queue and then preloaded the next design (613728). > > I issued the goto-field for 613728, but it was getting very cloudy, so took a quick look at clouds outside again. > > 06:07Z - Too cloudy. Acquire failed. Probably going to close soon based on the satellite. > > 06:15Z - Checked on the clouds again. When I came back in I was going to slew to stow, but noticed the axes were all out of closed loop. What was also odd was the FFS state was reportedly 'Open' for all 8 petals, but the petals were, in fact, all closed. > > I tried several times to clear the axes and try small commanded moves via both TCC and MCP, but none of those things worked. I went down to the telescope and tried a couple of stop buttons, but the axis states in the MCP did not register any stop buttons engaged, nor did the E-stops themselves actually light up. > > The relevant error message in the STUI Log that was recurring was a bad AZ state, but also an error reading the SLC (PLC). If the PLC isn't behaving properly, that better explains the goofy states of things like FF petals and E-stops not reporting, etc. > > 06:39:04 .mcp 0 mcp i trace="MGT: bad az state ABORT_EVENT: ID_ERROR_LIMIT" > 06:39:04 .mcp 0 mcp i trace="tbrakes azimuth: error reading slc: 0x0004" > 06:39:04 .mcp 0 mcp i trace="tbrakes altitude: error reading slc: 0x0004" > 06:39:06 .mcp 0 mcp i trace="MGT: bad az state ABORT_EVENT: ID_ERROR_LIMIT" > 06:39:06 .mcp 0 mcp i trace="tbrakes azimuth: error reading slc: 0x0004" > 06:39:06 .mcp 0 mcp i trace="tbrakes altitude: error reading slc: 0x0004" > > Called John and we talked about restarting the MCP as that was the best thing we could come up with to try at this point, even though there were doubts about it working given the errors about the SLC. However, if restarting the MCP re-established comms with the SLC, then it was worth a shot. > > I turned off the axes amplifiers, plugged-in the manual pendant, and then switched from MCP mode to ALT mode and made sure the windbaffle brake was engaged. (With no E-stops working, this was the next best thing for the peace of mind that a brake was on and I had control of it.) > > FYI, before restarting MCP the reported axes positioners were: AZ=366, ALT=43, ROT=228 > > Restarting the MCP almost worked. It went as smooth as one could ask for given we haven't had to touch it in quite some time, but it didn't do the trick. Unfortunately, after the restart there was a recurring "Broken pipe" error message and the corresponding process ID to which it referenced was the SLC process: > > mcp@mcp:~$ log_message write : Broken pipe > log_message write pid 26014 > log_message write : Broken pipe > > mcp 26014 5.8 0.0 11700 1000 pts/0 R 01:17 0:04 /home/mcp/sdss/mcp_linux/src/slc_dc > > Tried one more time for good measure, but after another MCP restart the same problem referencing the SLC occurred: > > log_message write : Broken pipe > log_message write pid 26147 > > mcp 26147 6.0 0.0 11700 1008 pts/0 S 01:23 0:03 /home/mcp/sdss/mcp_linux/src/slc_dc > > Called John once more and we decided to just go ahead and close using the windbaffle and the hand pendant to move AZ and then ALT as needed, and the building-interlock bypass. > > I moved ROT by hand to what should be very close to normal 0-deg position so that the LN2 autofill line could reach BOSS. > > I did not get APOGEE dome flats before disconnecting the gang connector because the FF lamps weren't controllable given the broken state of things anyway. > > 09:04Z - The positioners are home/folded. FPS is powered-off. APOGEE morning cals started. > > ---- Mirror Numbers ---- > > PRIMARY: > -------- > Scale: 1.000000 > > MIGS TONIGHT NOMINAL > Axial A: 0.2000 0.2000 > Axial B: -0.0010 -0.0011 > Axial C: 0.0640 0.0630 > Trans D: -0.0080 -0.0090 > Lateral E: 0.3380 0.3380 > Lateral F: 0.5455 0.5455 > > GALILS > Commanded: 9800 -300 6750 19150 -3550 -3250 > Actual: 9800 -300 6750 19150 -3550 -3250 > > SETMIR VALUES > PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 > PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 > > SECONDARY: > ---------- > Focus: 0.0000 > Air temp: 0.00 > Altitude: 30.0 > > MIGS TONIGHT NOMINAL > Axial A: -1.0400 -1.0390 > Axial B: 0.2200 0.2180 > Axial C: -1.1090 -1.1100 > Trans D: -0.0030 -0.0200 > > GALILS > Commanded: 5886750 5811900 5947300 -1500 -7250 > Actual: 5883775 5811725 5906206 -1500 -7250 > > SETMIR VALUES > SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 > SecOrient: 4657.06 -24.5 -17.54 17.67 125.26 > Saved new NOMINALS given the battery for M2 Trans D was changed today. > > PRIMARY: > -------- > Scale: 1.000000 > > MIGS TONIGHT NOMINAL > Axial A: 0.2010 0.2010 > Axial B: -0.0010 -0.0010 > Axial C: 0.0650 0.0650 > Trans D: -0.0120 -0.0120 > Lateral E: 0.3380 0.3380 > Lateral F: 0.5455 0.5455 > > GALILS > Commanded: 9800 -300 6750 19150 -3550 -3250 > Actual: 9800 -300 6750 19150 -3550 -3250 > > SETMIR VALUES > PrimDesOrient: 0.0 -18.5 -7.8 -427.0 -998.0 > PrimOrient: 0.0 -18.23 -7.9 -427.17 -999.0 > > SECONDARY: > ---------- > > Focus: 0.0000 > Air temp: 0.00 > Altitude: 30.0 > > MIGS TONIGHT NOMINAL > Axial A: -1.0410 -1.0410 > Axial B: 0.2170 0.2170 > Axial C: -1.1110 -1.1110 > Trans D: -0.0010 -0.0010 > > GALILS > Commanded: 5888150 5814000 5948200 -1500 -7250 > Actual: 5883681 5811700 5906125 -1500 -7250 > > SETMIR VALUES > SecDesOrient: 4657.0 -24.5 -17.5 18.0 125.41 > SecOrient: 4657.01 -24.5 -17.52 17.68 125.26 > > ---- Telescope Offsets and Scale ---- > > Time Field-Design Az Alt Rot RA Off Dec Off Rot Off Guide RMS (um) > ================================================================================ > 05:09:39 nan-nan 218.5 60.0 85.7 -0.009 0.007 -0.098 0.358 > 05:28:09 103240-621638 393.3 40.3 236.4 -0.005 0.014 -0.078 0.364 > 05:43:31 103240-621638 388.9 41.9 232.9 -0.004 0.015 -0.078 0.280 > 06:00:00 103240-621638 383.9 43.4 228.8 -0.004 0.015 -0.076 0.349 > > ---- Focus Log ---- > > Time Field-Design M1 M2 Focus Az Alt Temp Wind Dir > ================================================================================ > 05:09:39 nan-nan 0.00 4419.42 -133 218.5 60.0 0.0 9 90 > 05:28:09 103240-621638 0.00 4458.77 -157 393.3 40.3 0.0 8 100 > 05:43:31 103240-621638 0.00 4452.54 -157 388.9 41.9 0.0 6 95 > 06:00:00 103240-621638 0.00 4435.50 -168 383.9 43.4 0.0 4 162 > > ---- Telescope Status ---- > > Status at: 09:07Z > > Telescope stowed at: as close to stow as I could eyeball > Instrument mounted: FPS > Counterweights at: 692 695 712 712 > LN2 autofill systems: Connected and turned on > 180L LN2 dewar scales: SP1 158 lbs, 18.6 psi > > FPS Status > ========== > CP10 Loop = 7.54 L/min > GFA Loop = 2.28 L/min > Supply pressure = 42.4 psi > Return pressure = 29.2 psi > > Chiller Status > ============== > Set temperarature = 16.8 C > Flow = 4.9 Gpm > > ---- Software Used ---- > > STUI : 1.7.19 > alerts : 2.3.0 > boss : 6.0.2 > -boss.specMec : 3.0.4 > apo : 4.1.0 > fliswarm : 0.5.1a0 > jaeger : 1.7.5 > kaiju : 1.4.0 > coordio : 1.11.2 > fps_calibrati : 2024.04.01+91998f8 > cherno : 1.0.0 > astrometrynet : 0.93 > hal : 1.3.5 > hartmann : 2.0.2 > hub : 5.0.1a0 > mcp : 2.1-1-ge461 > -mcp.plc : 35 > -mcp.azFiduci : v1_149 > -mcp.altFiduc : v1_149 > -mcp.rotFiduc : v1_149 > tcc : 1.4.0 > apogee : not available > apogeecal : not available > apogeeql : v1_7_0dev > STUIAdditions : 3.1.18 > boson : 0.4.9 > sdss-obstools : 3.9.3 > TPM : 1.2-31-gd7eb > MCP : 2.1-1-ge461 > PLC : V35 > Fiducials : v1_149 > idlspec2d : v6_1_3 > Kronos : 1.7.0 > roboscheduler : 1.4.0 > > ---- APOGEE Data Summary ---- > > 60485 morning cals: > > num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 > -------- ------ ------ ------ ------------ ------- ----- ------- ----- ------- ------- ------ ------ ---- ---- > 49230035 100568 613489 15022 DARK 13.5 60 17 True -- -- -- -- -- -- > 49230036 100568 613489 15022 DARK 13.5 60 8 True -- -- -- -- -- -- > 49230037 100568 613489 15022 DARK 13.5 60 -2 True -- -- -- -- -- -- > 49230038 100568 613489 15022 QUARTZFLAT 13.5 10 107 True -- -0.20 -- 0.86 -- -- > 49230039 100568 613489 15022 QUARTZFLAT 13.5 10 109 True -- -0.18 -- 0.87 -- -- > 49230040 100568 613489 15022 QUARTZFLAT 13.5 10 109 True -- -0.17 -- 0.86 -- -- > 49230041 100568 613489 15022 ARCLAMP-THAR 13.5 12 108 True 1.04 0.63 1.08 0.90 -- -- > 49230042 100568 613489 15022 ARCLAMP-UNE 13.5 40 91 True 0.64 0.45 1.11 0.88 -- -- > 49230043 100568 613489 15022 ARCLAMP-THAR 13.0 12 97 True 0.53 0.68 1.06 0.88 -- -- > 49230044 100568 613489 15022 ARCLAMP-UNE 13.0 40 89 True 0.14 0.44 1.10 0.90 -- -- > 49230045 100568 613489 15022 DARK 13.0 30 17 False -- -- -- -- -- -- > 49230046 100568 613489 15022 INTERNALFLAT 13.0 30 99 True -- -- -- -- -- -- > 49230047 100568 613489 15022 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- > 49230048 100568 613489 15022 INTERNALFLAT 13.0 30 100 True -- -- -- -- -- -- > 49230049 100568 613489 15022 DARK 13.0 30 260 True -- -- -- -- -- -- > 49230050 100568 613489 15022 FPI 13.0 30 127 True 11.99 0.37 1.16 0.88 -- -- > 49230051 100568 613489 15022 FPI 13.5 30 127 True -12.13 0.39 1.15 0.89 -- -- > num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 > -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- > 49230038 QUARTZFLAT 10 70 76 84 93 93 88 90 91 91 84 > 49230039 QUARTZFLAT 10 72 78 85 93 94 89 91 91 91 85 > 49230040 QUARTZFLAT 10 73 79 85 94 95 89 91 92 92 86 > (observers39) [observer@sdss5-hub: ~]$ check_ap 60486 > Using nfowler=1 for DOMEFLAT > Using nfowler=1 for DOMEFLAT > Using nfowler=1 for DOMEFLAT > num field design config exptype dithpix nread relflux okay dx dy sigx sigy fpi1 fpi2 > -------- ------ ------ ------ ------------ ------- ----- ------- ---- ------- ------- ------ ------ ---- ---- > 49240001 100568 613489 15022 DARK 13.5 3 67 True -- -- -- -- -- -- > 49240002 -999 -999 15023 DARK 13.5 60 -22 True -- -- -- -- -- -- > 49240003 -999 -999 15023 QUARTZFLAT 13.5 10 105 True -- -0.27 -- 0.86 -- -- > 49240004 -999 -999 15023 ARCLAMP-THAR 13.5 12 100 True 1.05 0.54 1.07 0.89 -- -- > 49240005 -999 -999 15023 ARCLAMP-UNE 13.5 40 91 True 0.69 0.39 1.09 0.88 -- -- > 49240006 -999 -999 15023 ARCLAMP-THAR 13.0 12 92 True 0.53 0.59 1.06 0.87 -- -- > 49240007 -999 -999 15023 ARCLAMP-UNE 13.0 40 93 True 0.15 0.37 1.10 0.89 -- -- > 49240008 -999 -999 15023 DARK 13.0 60 7 True -- -- -- -- -- -- > 49240009 -999 -999 15023 FPI 13.0 30 135 True 11.98 0.33 1.17 0.88 -- -- > 49240010 -999 -999 15023 FPI 13.5 30 133 True -12.11 0.33 1.17 0.88 -- -- > 49240011 -999 -999 15023 DOMEFLAT 13.5 5 104 True -- -0.24 -- 0.84 -- -- > 49240012 -999 -999 15023 DOMEFLAT 13.5 5 109 True -- -0.23 -- 0.85 -- -- > 49240013 -999 -999 15023 DOMEFLAT 13.5 5 110 True -- -0.24 -- 0.85 -- -- > 49240014 112359 403333 15024 OBJECT 13.5 43 58 True 1.08 -0.27 1.76 0.91 102 104 > 49240015 112359 403333 15024 OBJECT 13.0 43 62 True 0.53 -0.26 1.76 0.90 105 105 > 49240016 103240 621638 15025 OBJECT 13.0 35 83 True 0.57 -0.25 1.71 0.90 102 105 > 49240017 103240 621638 15025 OBJECT 13.5 35 71 True 1.07 -0.22 1.71 0.89 98 103 > 49240018 103240 621638 15026 OBJECT 13.0 35 73 True 0.57 -0.25 1.70 0.89 105 105 > 49240019 103240 621638 15026 OBJECT 13.5 35 82 True 1.08 -0.21 1.70 0.88 100 104 > 49240020 103240 621638 15027 OBJECT 13.0 35 73 True 0.58 -0.25 1.70 0.90 103 103 > 49240021 103240 621638 15027 OBJECT 13.5 35 65 True 1.07 -0.21 1.70 0.89 98 105 > num exptype nread mtp1 mtp2 mtp3 mtp4 mtp5 mtp6 mtp7 mtp8 mtp9 mtp10 > -------- ---------- ----- ---- ---- ---- ---- ---- ---- ---- ---- ---- ----- > 49240003 QUARTZFLAT 10 69 74 83 92 93 88 90 91 91 83 > 49240011 DOMEFLAT 5 74 80 91 99 107 95 100 105 99 93 > 49240012 DOMEFLAT 5 80 88 98 106 114 101 107 112 106 100 > 49240013 DOMEFLAT 5 79 88 97 104 112 99 105 109 104 99 > 49240025 QUARTZFLAT 10 69 75 83 93 93 88 88 91 91 84 > > ---- BOSS Data Summary ---- > > Reading FITS from /data/spectro//60486 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > n Time Filename design-field cfg flav shu Exp colA colB colC Hrt > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > 1 23:21 b1-00370582 ----------- ---- bia Clo 0.0 5202 5205 5196 > 2 23:21 r1-00370582 ----------- ---- bia Clo 0.0 5202 5205 5196 > 3 23:35 b1-00370583 -999--999 15023 arc Lef 4.1 5202 5205 5196 HART > 4 23:35 r1-00370583 -999--999 15023 arc Lef 4.1 5202 5205 5196 HART > 5 23:36 b1-00370584 -999--999 15023 arc Rig 4.1 5202 5205 5196 HART > 6 23:36 r1-00370584 -999--999 15023 arc Rig 4.1 5202 5205 5196 HART > 7 23:37 b1-00370585 -999--999 15023 arc Lef 4.1 5875 5879 5873 HART > 8 23:37 r1-00370585 -999--999 15023 arc Lef 4.1 5875 5879 5873 HART > 9 23:38 r1-00370586 -999--999 15023 arc Rig 4.1 5875 5879 5873 HART > 10 23:38 b1-00370586 -999--999 15023 arc Rig 4.1 5875 5879 5873 HART > 11 23:40 r1-00370587 -999--999 15023 arc Out 4.1 5899 5905 5896 > 12 23:40 b1-00370587 -999--999 15023 arc Out 4.1 5899 5905 5896 > 13 23:42 b1-00370588 -999--999 15023 fla Out 25.1 5899 5905 5896 > 14 23:42 r1-00370588 -999--999 15023 fla Out 25.1 5899 5905 5896 > 15 04:38 b1-00370589 -999--999 15023 arc Lef 4.1 5899 5905 5896 HART > 16 04:38 r1-00370589 -999--999 15023 arc Lef 4.1 5899 5905 5896 HART > 17 04:38 r1-00370590 -999--999 15023 arc Rig 4.1 5899 5905 5896 HART > 18 04:38 b1-00370590 -999--999 15023 arc Rig 4.1 5899 5905 5896 HART > 19 04:40 b1-00370591 -999--999 15023 fla Out 25.1 7059 7065 7056 > 20 04:40 r1-00370591 -999--999 15023 fla Out 25.1 7059 7065 7056 > 21 04:42 r1-00370592 -999--999 15023 fla Out 25.1 7059 7065 7056 > 22 04:42 b1-00370592 -999--999 15023 fla Out 25.1 7059 7065 7056 > 23 04:44 r1-00370593 -999--999 15023 fla Out 25.1 7059 7065 7056 > 24 04:44 b1-00370593 -999--999 15023 fla Out 25.1 7059 7065 7056 > 25 04:48 b1-00370594 403333-112359 15024 fla Out 25.1 7059 7065 7056 > 26 04:48 r1-00370594 403333-112359 15024 fla Out 25.1 7059 7065 7056 > 27 04:51 r1-00370595 403333-112359 15024 arc Out 4.1 7059 7065 7056 > 28 04:51 b1-00370595 403333-112359 15024 arc Out 4.1 7059 7065 7056 > 29 04:54 r1-00370596 403333-112359 15024 sci Out 900.1 7059 7065 7056 > 30 04:54 b1-00370596 403333-112359 15024 sci Out 900.1 7059 7065 7056 > 31 05:13 b1-00370597 621638-103240 15025 arc Out 4.1 7059 7065 7056 > 32 05:13 r1-00370597 621638-103240 15025 arc Out 4.1 7059 7065 7056 > 33 05:15 r1-00370598 621638-103240 15025 sci Out 730.1 7059 7065 7056 > 34 05:15 b1-00370598 621638-103240 15025 sci Out 730.1 7059 7065 7056 > 35 05:29 r1-00370599 621638-103240 15026 sci Out 730.1 7059 7065 7056 > 36 05:29 b1-00370599 621638-103240 15026 sci Out 730.1 7059 7065 7056 > 37 05:43 r1-00370600 621638-103240 15027 sci Out 730.1 7059 7065 7056 > 38 05:43 b1-00370600 621638-103240 15027 sci Out 730.1 7059 7065 7056 > 39 06:04 r1-00370601 613728-100572 15029 arc Out 4.1 7059 7065 7056 > 40 06:04 b1-00370601 613728-100572 15029 arc Out 4.1 7059 7065 7056 > > ---- BOSS Spectrographs Collimation ---- > > Afternoon focus check > ===================== > 23:37:41 BOSS expNum=370585, expTime= 4.0s > 23:38:21 BOSS expNum=370586, expTime= 4.0s > 23:39:02 sp1.motor.move= 24, 26, 23 > 23:39:02 Hartmann collimate output > sp1: offset: r= -659 (In Focus); b= -2.2 (In Focus) > sp1: pred. move: spAverageMove= 24 > sp1: pred. resid: r= -683, b= -2.1, txt= OK, spTemp = 10.9 > 23:38:59 APO.doravetz_3 39 hartmann i r1MeanOffset=-0.10,"In Focus" > 23:38:59 APO.doravetz_3 39 hartmann i b1MeanOffset=0.11,"In Focus" > > Night > ===== > Time TSP1 R1 B1 Move1 B1Resid > ============================================= > 04:39:20 9.8 329 -6.2 1160 -2.6 > > ---- Weather Log ---- > > Time Field-Design Temp DP Diff Humid Wind Dir 1μm Dust IRSC σ IRSC μ > ================================================================================ > 05:09:39 nan-nan nan nan 0.0 0.0 0.0 100 9 90.4 nan > 05:28:09 103240-621638 0.0 0.0 0.0 100.0 7.8 100 nan 3.4 341 > 05:43:31 103240-621638 0.0 0.0 0.0 100.0 5.6 95 nan 52.8 351 > 06:00:00 103240-621638 0.0 0.0 0.0 100.0 4.3 162 nan 61.6 354 > > ---- Time Tracking ---- > > *=*=*= Observatory: APO > *=*=*= MJD: 60486 > Fill night time in Hours below > *=*=*= Total bright time : 6.6 > *=*=*= Bright time, queued observations : 0.8 > *=*=*= Bright time, commissioning/engineering: 0 > *=*=*= Bright time lost to weather : 2.8 > *=*=*= Bright time lost to technical issues : 3 > > *=*=*= Total dark time : 1.4 > *=*=*= Dark time, queued observations : 0.3 > *=*=*= Dark time, commissioning/engineering : 0 > *=*=*= Dark time lost to weather : 1.1 > *=*=*= Dark time lost to technical issues : 0 > > > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html