Date: Fri, 11 Nov 2022 11:07:57 -0700 Subject: Re: sdss-gfa4 -- [sloan-25mlog 6341] Re: 2.5m obslog 20221110 (SJD From: "Daniel Oravetz" I experienced similar problems a few weeks ago. Below is the sequence I went through to eventually get sdss-gfa4 back. Ultimately I had to go into the IEB and manually power-cycle sdss-gfa4, then I followed that with a soft reboot to fix a "No containers" error following the hard reboot, and then I restarted fliswarm from systemctl after that soft reboot. Since this sounds like it's a recurring issue, I can post a pared-down version of this on the wiki for reference. Dan From 59867: After powering-on FPS, the 'fliswarm reconnect' produced an error: 23:52:10 APO.doravetz 406 fliswarm f error="Command 'reconnect' failed because of an uncaught error 'ConnectionRefusedError: [Errno 111] Connect call failed ('10.25.1.125', 19995)'. See traceback in the log for more information." 23:52:10 CmdDone: APO.doravetz 406 fliswarm reconnect Pinged 10.25.1.125 just to makes sure it was responsive and it returned pings. Tried the 'fliswarm reconnect' again, but now sdss-gfa4 was producing a connection timeout: 23:52:47 APO.doravetz 407 fliswarm f error="Command 'reconnect' failed because of an uncaught error 'ConnectTimeout: HTTPConnectionPool(host='sdss-gfa4', port=2375): Max retries exceeded with url: /v1.41/containers/json?limit=-1&all=0&size=0&trunc_cmd=0&filters=%7B%22name%22%3A+%5B%22flicamera-gfa4%22%5D%2C+%22status%22%3A+%5B%22running%22%5D%7D (Caused by ConnectTimeoutError(, 'Connection to sdss-gfa4 timed out. (connect timeout=3)'))'. See traceback in the log for more information." 23:52:47 CmdDone: APO.doravetz 407 fliswarm reconnect 23:55:11 APO.doravetz 413 fliswarm w text="Node sdss-gfa4 is not pinging back." 23:56:35 APO.doravetz 414 fliswarm w text="Node gfa4 is not pinging back or the Docker daemon is not running. Try rebooting the computer." Tried a reboot of sdss-gfa4: 23:57:15 CmdStarted: APO.doravetz 416 fliswarm reboot -n sdss-gfa4 Still no luck; same sdss-gfa4 error as above. Tried a hard reboot: 23:59:13 CmdStarted: APO.doravetz 421 fliswarm reboot -n sdss-gfa4 -f That also didn't fix it. At the telescope I removed the IEB cover and verified all the connections to and from sdss-gfa4 were solid, then I manually rebooted sdss-gfa4. Next I issued a "soft" reboot of sdss-gfa4 after the power cycle because there had been an error about "No containers running to sdss-gfa4." 00:09:18 CmdStarted: APO.telescope 491 fliswarm reboot -n sdss-gfa4 The manual power cycle and the subsequent soft reboot evidently fixed the connection problem with sdss-gfa4 because sdss-gfa4 was back according to 'fliswarm status', but now there's a "No route to host." error instead: 00:12:57 APO.doravetz 424 fliswarm f error="Command 'reconnect' failed because of an uncaught error 'OSError: [Errno 113] No route to host'. See traceback in the log for more information." Restarted fliswarm from systemctl and now 'fliswarm reconnect' completes successfully. > Something is causing the connection for GFA4 to drop intermittently during > the night (last night and tonight). Sometimes a 'fliswarm reconnect' works > to get it talking again, but most of the time it does not. > > Does anyone have any ideas as to why this could be happening, have any > recommendations on how we can stop it from happening again, and/or have > any other troubleshooting measures that we can try during the night? > > Audrey > >> >> APACHE POINT OBSERVATORY SDSS 2.5M >> >> OBSERVING LOG >> Thursday November 10, 2022 >> >> (SJD 59894) >> >> ---- Observing Team ---- >> >> Audrey Oravetz >> Dmitry Bizyaev >> >> José Sánchez-Gallego (afternoon support) >> >> ---- Observing Plan ---- >> >> Thu 10 Nov 2022 59894 0.92 01:00 01:15 12:19 21:32 08:38 --- bright >> (11.3 >> h, w/ 0.2 h for 12 deg design) >> >> >> ---- Observing Summary ---- >> >> Clear for the first half of the night (seeing ~1.5"), then mostly cloudy >> after 06:30Z. The rest of the night was lost due to clouds and an early >> morning humidity spike (loss of 6.1hrs). >> >> There were no instances of software delays tonight! >> >> Observed: >> --------- >> Time field-dsgn-conf Design type BOSS/APOGEE exp complete? >> Comment >> =========================================================================== >> 00:27:58 100461-37788-7414 1X900 + 1XAB DONE Clear >> 00:57:04 101782-119646-7415 1X900 + 1XAB DONE “ >> 01:19:48 100461-37788-7416 1X900 + 1XAB DONE “ >> 01:41:20 102470-122369-7417 1X900 + 1XAB DONE “ >> 02:03:30 100456-114083-7418 1X900 + 1XAB DONE “ >> 02:26:30 100456-114084-7419 1X900 + 1XAB DONE “ >> 02:48:28 104436-128413-7420 1X900 + 1XAB DONE “ >> 03:12:00 103561-125638-7421 1X900 + 1XAB DONE “ >> 03:34:33 100619-115217-7422 1X900 + 1XAB DONE “ >> 03:56:57 100595-115095-7423 1X900 + 1XAB DONE “ >> 04:19:27 103558-125634-7424 1X900 + 1XAB **not done Partly/mostly >> cloudy >> 04:42:14 101731-119037-7425 2X900 + 2XAB DONE Partly cloudy >> 05:21:32 101731-119038-7426 1X900 + 1XAB DONE “ >> 05:43:09 101731-119039-7427 1X900 + 1XAB DONE “ >> 06:05:46 101731-119040-7428 1X900 + 1XAB not done Mostly cloudy for >> the >> last dither; could not take another exposure because it was past the >> observing window >> TOO CLOUDY 06:27Z - 12:34Z >> >> * Many systems needed to be restarted in the afternoon / early evening >> (apogeefpi, hal, cerebro, influx, cherno, kronos, apogeeql), probably >> due >> to the tron restart earlier in the day. No time was lost. >> >> * However, exposure apRaw-43320016.fits did not make it properly into >> /data/apogee/archive/59894/ because of an apogeeql restart from above. >> >> * GFA4 disconnected several times throughout the night. The first time >> (04:44Z) it happened, a simple 'fliswarm reconnect' got the camera >> reconnected but the second time (06:30Z), it was not able to be >> reconnected. >> 04:44:46 .fliswarm 0 fliswarm w text="Docker client on node sdss-gfa4 is >> not connected." >> >> ** We were not guiding for the last ~3 minutes of the >> apRaw-43320034/r1b1-00348248 exposures due to thick clouds interfering >> with guider corrections: >> 04:41:00 APO.aoravetz_2 64 cherno f error="Acquisition failed: Timed out >> updating camera status." >> Repeated attempts to get it restarted failed until cherno was restarted >> with systemctl. However, while working to solve that problem, gotoField >> was already started for the next field before checking the signal >> collected on the previous field. Officially we did not collect enough >> signal on 103558-125634-7424 for completion, and it is too late to go >> back >> since it is at the tail end of it's observing window. >> >> >> ---- Automatic Summary ---- >> >> >> Time Field-Design-Conf DesignType BOSS/APOGEE Seeing Overhead >> Complete? Comment >> 00:27:58 100461-37788-7414 ........... 1x900/1xAB 1.4" 0:00 >> 00:57:04 101782-119646-7415 ........... 1x900/0xAB 1.4" 7:32 >> 01:19:48 100461-37788-7416 ........... 1x900/1xAB 1.4" 6:48 >> 01:41:20 102470-122369-7417 ........... 1x900/1xAB 1.3" 6:56 >> 02:03:30 100456-114083-7418 ........... 1x900/1xAB 2.0" 8:02 >> 02:26:31 100456-114084-7419 ........... 1x900/1xAB 2.0" 7:06 >> 02:48:28 104436-128413-7420 ........... 1x900/1xAB 1.9" 8:32 >> 03:12:00 103561-125638-7421 ........... 1x900/1xAB 1.3" 7:31 >> 03:34:33 100619-115217-7422 ........... 1x900/1xAB 1.8" 7:14 >> 03:56:57 100595-115095-7423 ........... 1x900/1xAB 1.9" 7:36 >> 04:19:27 103558-125634-7424 ........... 1x900/1xAB 1.3" 7:45 >> 04:42:14 101731-119037-7425 ........... 2x900/2xAB 1.4" 7:50 >> 05:21:32 101731-119038-7426 ........... 1x900/1xAB 1.4" 6:36 >> 05:43:09 101731-119039-7427 ........... 1x900/1xAB 1.3" 7:10 >> 06:05:46 101731-119040-7428 ........... 1x900/1xAB 1.6" 6:56 >> >> >> >> ---- Engineering Issues ---- >> >> >> * The FPI cleaning technique worked. Signal in the fibers are now ~6000 >> instead of maybe ~1000-2000 from before. Thank you. >> >> >> ---- Observing Log ---- >> >> Afternoon: >> ========== >> >> Median spectrographs temperatures: >> Tsp1= 4.0 >> Outside air temperature= 92.8; >> ---------- >> Boss CCD Temp : >> sp1r0= -124.6 (-126.6); >> sp1b2= -100.0 (-100.0); >> sp1LN2: 10.0; >> >> Discovered that the APOGEE evening cal script did not start because >> there >> was no connection to the fpi shutter. >> >> 22:37:00 CmdStarted: jose.jose 10 apogeefpi status >> 22:37:01 jose.jose 10 apogeefpi > >> 22:37:01 jose.jose 10 apogeefpi : shutter_position=closed >> 22:37:01 CmdDone: jose.jose 10 apogeefpi status >> 22:37:05 CmdStarted: jose.jose 10 apogeefpi open >> 22:37:05 jose.jose 10 apogeefpi > >> 22:37:05 jose.jose 10 apogeefpi f error="Command 'open' failed because >> of >> an uncaught error 'CluError: cannot connect to tron.'. See traceback in >> the log for more information." >> 22:37:05 CmdDone: jose.jose 10 apogeefpi open >> >> I connected the Jose through Slack. He helped restart the actor through >> hub. There was a complete restart to tron this morning. The new >> procedure >> to do this is now found on the software restart page, as well as in the >> observer tips page. >> >> Hal was also found not connected to tron. >> >> I restarted it through systemctl on hub. >> >> 22:39:52 APO.aoravetz 32 hal w text="Script eveningcals: 10.0 apogeecal >> allOff" >> 22:39:52 APO.aoravetz 32 hal f error="cannot connect to tron." >> 22:39:52 CmdDone: APO.aoravetz 32 hal script run eveningcals >> >> Restarted APOGEE evening cals. >> >> Checked mirror numbers, APOGEE cals, BOSS cals, and FPS cals. >> >> 22:24:56 boss exposure 20.0, file=348212 >> >> 22:36:02 sp1.motor.move= -2255, -2255, -2254 >> 22:36:02 Hartmann collimate output >> sp1: offset: r= -2968 (Out of focus); b= 4.8 (Out of focus) >> sp1: pred. move: spAverageMove= -2255 >> sp1: pred. resid: r= -713, b= -2.2, txt= OK, spTemp = 4.1 >> 22:41:41 boss exposure 4.0, file=348217 >> 22:41:48 apogee.expose=apRaw-43320002, 60, A, Dark >> 22:44:17 boss exposure 25.0, file=348218 >> 22:52:41 apogee.expose=apRaw-43320003, 10, A, QuartzFlat >> >> 22:54:46 APO.aoravetz_2 28 jaeger i >> configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59894/configuration_snapshot_59894_7413.pdf >> 22:54:46 APO.aoravetz_2 28 jaeger : text="Configuration 7413 loaded." >> 22:54:47 CmdDone: APO.aoravetz_2 28 jaeger configuration load >> --from-positions >> >> 22:58:21 APO.aoravetz_2 32 fliswarm i >> filename=fvc,fvc1n,/data/fcam/59894/fimg-fvc1n-0001.fits >> 22:58:22 APO.aoravetz_2 32 fliswarm i >> exposure_state=fvc,fvc1n,idle,NA,0.0,0,0,0 >> 22:58:22 APO.aoravetz_2 32 fliswarm i >> filename_bundle=/data/fcam/59894/fimg-fvc1n-0001.fits >> 22:58:22 APO.aoravetz_2 32 fliswarm : >> 22:58:22 CmdDone: APO.aoravetz_2 32 fliswarm talk -n fvc expose 50 >> >> 22:58:42 boss exposure 300.0, file=348219 >> >> No light seen within images. >> >> 23:15Z - Started ventilating. Telescope checkout went ok. Clear skies. >> >> (observers39) [observer@sdss5-hub: ~]$ list_ap >> APOGEE data list, mjd=59894, all >> raw_data: /data/apogee/utr_cdr/59894/apRaw-*.fits >> archive: /data/apogee/archive/59894/apR-[a,b,c]-*.apz >> quickred: /data/apogee/quickred/59894/apq-*.fits >> ------------------------------------------------------------------------------ >> mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset >> %Flux/FPI Comm >> ------------------------------------------------------------------------------ >> 59894 1 21:33 43320001 Dark 3 A 13.0 ---------- a-b-c - >> - >> 59894 2 22:52 43320002 Dark 60 A 13.0 ---------- a-b-c - >> - >> 59894 3 22:54 43320003 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 59894 4 22:56 43320004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.07 >> - >> 59894 5 23:03 43320005 Arc-Une 40 B 13.5 ---------- a-b-c - >> - >> 59894 6 23:05 43320006 Arc-Thar 12 A 13.0 ---------- a-b-c 2.60 >> - >> 59894 7 23:13 43320007 Arc-Une 40 A 13.0 ---------- a-b-c - >> - >> 59894 8 23:24 43320008 Dark 60 A 13.0 ---------- a-b-c - >> - >> 59894 9 23:29 43320009 Arc---- 30 A 13.0 ---------- a-b-c - >> 90 FPI >> ------------------------------------------------------------------------------ >> >> Night: >> ====== >> >> 00:37Z - Opened enclosure and filled spectrographs. >> >> root@sdss5-influx needed to be restarted since no weather or telescope >> hardware data was within GRAFANA. >> >> 00:12:45 apogee.expose=apRaw-43320010, 5, A, DomeFlat >> 00:13:59 apogee.expose=apRaw-43320011, 5, A, DomeFlat >> 00:15:14 apogee.expose=apRaw-43320012, 5, A, DomeFlat >> 00:16:25 boss exposure 25.0, file=348220 >> 00:18:14 boss exposure 25.0, file=348221 >> 00:20:02 boss exposure 25.0, file=348222 >> >> ******************************************************** >> 00:27:58 Field-Design-Configuration: 100461-37788-7414 >> ******************************************************** >> 00:28:11 GOTO-FIELD >> 00:30:51 FVC LOOP AND BOSS CALS >> fvc_rms= 9.72 >> fvc_deltarms= 27.21 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0003.fits >> fvc_perc_90= 10.00 >> fvc_percent_reached= 52.60 >> 00:31:20 boss exposure 25.0, file=348223 >> 00:33:16 boss exposure 4.0, file=348224 >> 00:33:56 boss exposure 4.0, file=348225 >> 00:34:44 sp1.motor.move= 3289, 3289, 3287 >> 00:34:44 Hartmann collimate output >> sp1: offset: r= 3298 (Out of focus); b= -10.3 (Out of focus) >> sp1: pred. move: spAverageMove= 3288 >> sp1: pred. resid: r= 10, b= 0.0, txt= OK, spTemp = 2.8 >> 00:35:03 boss exposure 4.0, file=348226 >> 00:36:50 CmdStarted: APO.aoravetz_2 52 cherno acquire -t 7.0 --count 5 >> --full >> 00:36:50 APO.aoravetz_2 52 cherno > >> 00:36:50 APO.aoravetz_2 52 cherno w text="FFS petals are not open." >> 00:36:50 .cherno 0 cherno i guider_status=0x1 >> 00:36:50 .cherno 0 cherno i guider_status=0x2 >> 00:36:53 .tcc 0 tcc i TCCPos=-53.920032,62.618429,166.633223; >> AxePos=-53.920320,62.618254,166.634200; >> ObjInstAng=31.748900,0.000000,5174843459.8113012; >> SpiderInstAng=-12.999693,-0.004484,5174843821.6627235 >> 00:36:53 .tcc 0 tcc i AxisCmdState=Tracking,Tracking,Tracking; >> AxisErrCode=OK,OK,OK; AxisNoTrack=FFF >> 00:36:53 .alerts 0 alerts i activeAlerts >> 00:36:53 .alerts 0 alerts i disabledAlertRules >> 00:36:53 .cherno 0 cherno i guider_status=0x21 >> 00:36:53 APO.aoravetz_2 52 cherno f error="Acquisition failed: Timed out >> updating camera status." >> 00:36:53 CmdDone: APO.aoravetz_2 52 cherno acquire -t 7.0 --count 5 >> --full >> >> Yes the petals were open. Cherno was getting false information, probably >> related to the software restarts from earlier in the day. Cherno was >> restarted. >> >> 00:41:43 apogee.expose=apRaw-43320013, 43, A, Object >> 00:41:49 boss exposure 900.0, file=348227 >> 00:49:24 apogee.expose=apRaw-43320014, 43, B, Object >> >> ******************************************************** >> 00:57:04 Field-Design-Configuration: 101782-119646-7415 >> ******************************************************** >> 00:57:55 GOTO-FIELD >> >> 01:01:03 FVC LOOP AND BOSS CALS >> fvc_rms= 13.54 >> fvc_deltarms= 23.18 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0005.fits >> fvc_perc_90= 9.70 >> fvc_percent_reached= 54.10 >> 01:01:51 boss exposure 4.0, file=348228 >> 01:03:52 New-field overhead: 0:05:57 >> 01:03:52 START SCIENCE EXPOSURES >> 01:04:15 apogee.expose=apRaw-43320015, 43, B, Object >> 01:04:22 boss exposure 900.0, file=348229 >> 01:12:01 apogee.expose=apRaw-43320016, 43, A, Object >> >> ******************************************************** >> 01:19:48 Field-Design-Configuration: 100461-37788-7416 >> ******************************************************** >> 01:20:26 GOTO-FIELD >> >> It seems as if the apogeeql S/N information is not making it into >> Kronos, >> and Kronos does not change the observing plan after reloading the queue. >> So, Kronos was restarted. >> >> In the meantime, only one exposure pair will be taken with each design, >> given the history of the signal and based off the pristine guiding and >> sky >> conditions. >> >> Now Kronos is updating, but no APOGEE signal is seen yet. Apogeeql was >> restarted next. >> >> 01:23:20 FVC LOOP AND BOSS CALS >> fvc_rms= 12.86 >> fvc_deltarms= 33.2 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0007.fits >> fvc_perc_90= 10.50 >> fvc_percent_reached= 52.80 >> 01:24:06 boss exposure 4.0, file=348230 >> 01:25:42 New-field overhead: 0:05:16 >> 01:25:42 START SCIENCE EXPOSURES >> 01:26:05 apogee.expose=apRaw-43320017, 43, A, Object >> 01:26:11 boss exposure 900.0, file=348231 >> 01:33:46 apogee.expose=apRaw-43320018, 43, B, Object >> >> Yes! We can see APOGEE's signal again. >> >> ******************************************************** >> 01:41:20 Field-Design-Configuration: 102470-122369-7417 >> ******************************************************** >> 01:42:15 GOTO-FIELD >> 01:45:06 FVC LOOP AND BOSS CALS >> fvc_rms= 7.29 >> fvc_deltarms= 28.29 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0009.fits >> fvc_perc_90= 9.90 >> fvc_percent_reached= 50.80 >> 01:45:58 boss exposure 4.0, file=348232 >> 01:47:39 New-field overhead: 0:05:24 >> 01:47:39 START SCIENCE EXPOSURES >> 01:48:02 apogee.expose=apRaw-43320019, 43, B, Object >> 01:48:08 boss exposure 900.0, file=348233 >> 01:55:47 apogee.expose=apRaw-43320020, 43, A, Object >> >> ******************************************************** >> 02:03:30 Field-Design-Configuration: 100456-114083-7418 >> ******************************************************** >> 02:04:12 GOTO-FIELD >> 02:08:20 FVC LOOP AND BOSS CALS >> fvc_rms= 8.9 >> fvc_deltarms= 29.35 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0011.fits >> fvc_perc_90= 9.60 >> fvc_percent_reached= 55.10 >> 02:09:06 boss exposure 4.0, file=348234 >> 02:10:42 New-field overhead: 0:06:29 >> 02:10:42 START SCIENCE EXPOSURES >> 02:11:04 apogee.expose=apRaw-43320021, 43, A, Object >> 02:11:11 boss exposure 900.0, file=348235 >> 02:18:45 apogee.expose=apRaw-43320022, 43, B, Object >> >> ******************************************************** >> 02:26:30 Field-Design-Configuration: 100456-114084-7419 >> ******************************************************** >> 02:27:16 GOTO-FIELD >> 02:30:19 FVC LOOP AND BOSS CALS >> fvc_rms= 6.18 >> fvc_deltarms= 36.21 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0013.fits >> fvc_perc_90= 8.80 >> fvc_percent_reached= 57.20 >> 02:32:49 Same-field overhead: 0:05:33 >> 02:33:11 apogee.expose=apRaw-43320023, 43, B, Object >> 02:33:18 boss exposure 900.0, file=348236 >> 02:40:58 apogee.expose=apRaw-43320024, 43, A, Object >> >> ******************************************************** >> 02:48:28 Field-Design-Configuration: 104436-128413-7420 >> ******************************************************** >> 02:49:23 GOTO-FIELD >> 02:52:18 FVC LOOP AND BOSS CALS >> fvc_rms= 43.63 >> fvc_deltarms= -6.86 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0015.fits >> fvc_perc_90= 9.40 >> fvc_percent_reached= 56.00 >> 02:52:49 boss exposure 4.0, file=348237 >> 02:53:30 boss exposure 4.0, file=348238 >> 02:54:22 sp1.motor.move= -1913, -1912, -1914 >> 02:54:22 Hartmann collimate output >> sp1: offset: r= -1319 (Out of focus); b= 7.9 (Out of focus) >> sp1: pred. move: spAverageMove= -1912 >> sp1: pred. resid: r= 593, b= 1.9, txt= OK, spTemp = 2.1 >> 02:54:44 boss exposure 4.0, file=348239 >> 02:56:22 New-field overhead: 0:06:59 >> 02:56:22 START SCIENCE EXPOSURES >> 02:56:44 apogee.expose=apRaw-43320025, 43, A, Object >> 02:56:51 boss exposure 900.0, file=348240 >> 03:04:26 apogee.expose=apRaw-43320026, 43, B, Object >> >> ******************************************************** >> 03:12:00 Field-Design-Configuration: 103561-125638-7421 >> ******************************************************** >> 03:12:55 GOTO-FIELD >> 03:16:08 FVC LOOP AND BOSS CALS >> fvc_rms= 6.36 >> fvc_deltarms= 31.66 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0017.fits >> fvc_perc_90= 9.70 >> fvc_percent_reached= 55.10 >> 03:16:59 boss exposure 4.0, file=348241 >> 03:18:54 New-field overhead: 0:05:58 >> 03:18:54 START SCIENCE EXPOSURES >> 03:19:17 apogee.expose=apRaw-43320027, 43, B, Object >> 03:19:23 boss exposure 900.0, file=348242 >> 03:27:03 apogee.expose=apRaw-43320028, 43, A, Object >> >> ******************************************************** >> 03:34:33 Field-Design-Configuration: 100619-115217-7422 >> ******************************************************** >> 03:35:29 GOTO-FIELD >> 03:38:22 FVC LOOP AND BOSS CALS >> fvc_rms= 9.35 >> fvc_deltarms= 27.02 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0019.fits >> fvc_perc_90= 9.90 >> fvc_percent_reached= 56.70 >> 03:39:10 boss exposure 4.0, file=348243 >> 03:41:09 New-field overhead: 0:05:40 >> 03:41:09 START SCIENCE EXPOSURES >> 03:41:32 apogee.expose=apRaw-43320029, 43, A, Object >> 03:41:38 boss exposure 900.0, file=348244 >> 03:49:13 apogee.expose=apRaw-43320030, 43, B, Object >> >> ******************************************************** >> 03:56:57 Field-Design-Configuration: 100595-115095-7423 >> ******************************************************** >> 03:57:42 GOTO-FIELD >> 04:00:35 FVC LOOP AND BOSS CALS >> fvc_rms= 10.86 >> fvc_deltarms= 31.37 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0021.fits >> fvc_perc_90= 10.20 >> fvc_percent_reached= 54.80 >> 04:01:24 boss exposure 4.0, file=348245 >> 04:03:46 New-field overhead: 0:06:03 >> 04:03:46 START SCIENCE EXPOSURES >> 04:04:08 apogee.expose=apRaw-43320031, 43, B, Object >> 04:04:15 boss exposure 900.0, file=348246 >> 04:11:54 apogee.expose=apRaw-43320032, 43, A, Object >> >> ******************************************************** >> 04:19:27 Field-Design-Configuration: 103558-125634-7424 >> ******************************************************** >> 04:20:19 GOTO-FIELD >> 04:23:46 FVC LOOP AND BOSS CALS >> fvc_rms= 25.14 >> fvc_deltarms= 15.88 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0023.fits >> fvc_perc_90= 11.00 >> fvc_percent_reached= 49.20 >> 04:24:33 boss exposure 4.0, file=348247 >> 04:26:31 New-field overhead: 0:06:12 >> 04:26:31 START SCIENCE EXPOSURES >> 04:26:54 apogee.expose=apRaw-43320033, 43, A, Object >> 04:27:01 boss exposure 900.0, file=348248 >> 04:34:35 apogee.expose=apRaw-43320034, 43, B, Object >> >> 04:41:00 APO.aoravetz_2 64 cherno f error="Acquisition failed: Timed out >> updating camera status." >> 04:41:00 CmdDone: APO.aoravetz_2 64 cherno acquire -t 15.0 >> >> The guider was not guiding for the last ~3 minutes. Cherno was >> restarted. >> May have been a timeout due to clouds interfering with the correction. >> >> I'm sorry. I was working on getting the guider fixed when I started >> gotoField for the next field. I did not check the signal collected on >> the >> previous field before switching. Officially we did not collect enough >> signal because of clouds and the guider not updating. >> >> ******************************************************** >> 04:42:14 Field-Design-Configuration: 101731-119037-7425 >> ******************************************************** >> 04:43:07 GOTO-FIELD >> 04:46:27 FVC LOOP AND BOSS CALS >> fvc_rms= 22.49 >> fvc_deltarms= 15.39 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0025.fits >> fvc_perc_90= 10.20 >> fvc_percent_reached= 55.40 >> 04:47:15 boss exposure 4.0, file=348249 >> >> 04:44:46 .fliswarm 0 fliswarm w text="Docker client on node sdss-gfa4 is >> not connected." >> 04:44:46 .fliswarm 0 fliswarm i >> node=gfa4,sdss-gfa4,tcp://sdss-gfa4:2375,T,F >> >> fliswarm reconnect >> >> GFA4 began talking again. >> >> 04:49:23 New-field overhead: 0:06:16 >> 04:49:23 START SCIENCE EXPOSURES >> 04:49:46 apogee.expose=apRaw-43320035, 43, B, Object >> 04:49:52 boss exposure 900.0, file=348250 >> 04:57:31 apogee.expose=apRaw-43320036, 43, A, Object >> 05:06:16 apogee.expose=apRaw-43320037, 43, A, Object >> 05:06:23 boss exposure 900.0, file=348251 >> 05:13:57 apogee.expose=apRaw-43320038, 43, B, Object >> >> ******************************************************** >> 05:21:32 Field-Design-Configuration: 101731-119038-7426 >> ******************************************************** >> 05:22:27 GOTO-FIELD >> 05:25:07 FVC LOOP AND BOSS CALS >> fvc_rms= 40.15 >> fvc_deltarms= 2.64 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0027.fits >> fvc_perc_90= 10.60 >> fvc_percent_reached= 51.40 >> 05:27:31 Same-field overhead: 0:05:04 >> 05:27:31 START SCIENCE EXPOSURES >> 05:27:54 apogee.expose=apRaw-43320039, 43, B, Object >> 05:28:00 boss exposure 900.0, file=348252 >> 05:35:40 apogee.expose=apRaw-43320040, 43, A, Object >> >> ******************************************************** >> 05:43:09 Field-Design-Configuration: 101731-119039-7427 >> ******************************************************** >> 05:44:04 GOTO-FIELD >> 05:46:54 FVC LOOP AND BOSS CALS >> fvc_rms= 8.46 >> fvc_deltarms= 37.04 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0029.fits >> fvc_perc_90= 10.00 >> fvc_percent_reached= 50.80 >> 05:49:42 Same-field overhead: 0:05:37 >> 05:49:42 START SCIENCE EXPOSURES >> 05:50:05 apogee.expose=apRaw-43320041, 43, A, Object >> 05:50:11 boss exposure 900.0, file=348253 >> 05:57:46 apogee.expose=apRaw-43320042, 43, B, Object >> >> ******************************************************** >> 06:05:46 Field-Design-Configuration: 101731-119040-7428 >> ******************************************************** >> 06:06:16 GOTO-FIELD >> 06:09:37 FVC LOOP AND BOSS CALS >> fvc_rms= 17.68 >> fvc_deltarms= 29.75 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0031.fits >> fvc_perc_90= 11.20 >> fvc_percent_reached= 50.40 >> 06:11:38 Same-field overhead: 0:05:22 >> 06:11:39 START SCIENCE EXPOSURES >> 06:12:01 apogee.expose=apRaw-43320043, 43, B, Object >> 06:12:08 boss exposure 900.0, file=348254 >> 06:19:46 apogee.expose=apRaw-43320044, 43, A, Object >> >> ******************************************************** >> 06:27:16 Field-Design-Configuration: 103699-126123-7429 >> ******************************************************** >> >> Very cloudy. Waiting until the clouds get better before continuing. >> >> 06:30:28 APO.aoravetz_2 74 fliswarm w text="Node gfa4 is not pinging >> back >> or the Docker daemon is not running. Try rebooting the computer." >> >> systemctl fliswarm >> >> 06:41:16 hal.hal 188 jaeger f error="Another command >> jaeger-command-parser_fvc_loop is already running." >> 06:41:16 CmdDone: hal.hal 188 jaeger fvc loop >> 06:41:16 APO.aoravetz 44 hal e error="FVC loop failed." >> >> jaeger fvc loop --stop >> >> fliswarm reconnect >> >> Cannot connect to GFA4 this time. >> >> 06:43:48 boss exposure 25.0, file=348255 >> 06:46:26 boss exposure 4.0, file=348256 >> >> Switching to a new -better- field while the clouds were still around. >> >> ******************************************************** >> 06:50:47 Field-Design-Configuration: 103701-126142-7430 >> ******************************************************** >> 06:51:05 GOTO-FIELD >> 06:55:05 FVC LOOP AND BOSS CALS >> fvc_rms= 16.29 >> fvc_deltarms= 21.4 >> fvc_filename=/data/fcam/59894/fimg-fvc1n-0037.fits >> fvc_perc_90= 10.00 >> fvc_percent_reached= 54.80 >> 06:55:53 boss exposure 4.0, file=348257 >> >> 07:45Z - Attempted to guide again, without any luck. The satellite looks >> like it is getting worse, so I slewed to stow for the time being. >> >> 07:50:06 apogee.expose=apRaw-43320045, 5, A, DomeFlat >> 07:51:22 apogee.expose=apRaw-43320046, 5, A, DomeFlat >> 07:52:27 apogee.expose=apRaw-43320047, 5, A, DomeFlat >> >> 08:38Z - Now there are little radar specks within a different county. >> The >> enclosure was pulled over the telescope but we remained ventilating. >> >> 11:05Z - The clouds persisted. There was a humidity spike as well. >> Stopped >> ventilating. Turned off the FPS. Started morning cals, but accidentally >> clicked the 'stop' button within the APOGEE window during the first >> dark. >> I needed to abort the remaining exposures that were queued and then >> start >> over. >> >> >> ---- Mirror Numbers ---- >> >> >> PRIMARY: >> -------- >> Scale: 1.000000 >> >> MIGS TONIGHT NOMINAL >> Axial A: 0.0040 0.0060 >> Axial B: -0.0030 -0.0030 >> Axial C: -0.0320 0.0000 >> Trans D: 0.0190 0.0270 >> Lateral E: 0.3315 0.3315 >> Lateral F: 0.5350 0.5335 >> >> GALILS >> Commanded: 12500 -1600 5300 47050 -8800 -8400 >> Actual: 12500 -1600 5300 47050 -8800 -8400 >> >> SETMIR VALUES >> PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 >> PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 >> >> SECONDARY: >> ---------- >> >> Focus: 0.0000 >> Air temp: 92.83 >> Altitude: 30.0 >> >> MIGS TONIGHT NOMINAL >> Axial A: -0.0470 -0.0470 >> Axial B: -0.0440 -0.0440 >> Axial C: -0.0590 -0.0630 >> Trans D: -0.2420 -0.2810 >> >> GALILS >> Commanded: 4912950 4901650 4990850 -49 -3449 >> Actual: 4859236 4859274 4859079 0 -3400 >> >> SETMIR VALUES >> SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 >> SecOrient: 3856.98 0.05 -0.01 -0.01 42.84 >> >> >> >> ---- Telescope Status ---- >> >> 23:00Z Started ventilation >> 00:00Z encl25m : open >> 08:30Z encl25m : closed >> >> No interlocks bypassed >> >> Status at: 11:50Z >> Telescope stowed at: 121.0, 30.0, 0.0 mount >> Instrument mounted: FPS >> Counterweights at: 695 695 715 715 >> LN2 autofill systems: Connected and turned on >> 180L LN2 dewar scales: SP1 194 lbs, 21 psi >> >> FPS Status >> ========== >> CP10 Loop = 7.20 L/min >> GFA Loop = 1.56 L/min >> Supply pressure = 41.8 psi >> Return pressure = 28.9 psi >> >> Chiller Status >> ============== >> Set temperarature = 1.0 C >> Flow = 5.3 Gpm >> >> >> >> ---- Software Used ---- >> >> Get Versions, 11/10/22, 22:33:07 >> -- STUI software: >> STUI : 1.7.18b1 >> alerts : 2.3.0 >> boss : 6.0.2-alpha.0 >> -boss.specMec : 3.0.4 >> apo : 3.0.1 >> fliswarm : 0.3.1 >> jaeger : 1.2.0b1 >> kaiju : 1.3.0 >> coordio : 1.5.0b1 >> fps_calibrati : 2022.09.11+7ffa0ab >> cherno : 0.4.0b3 >> astrometrynet : 0.91 >> hal : 0.3.1a0 >> hartmann : 2.0.2 >> hub : 5.0.1a0 >> mcp : 2.1 >> -mcp.plc : 35 >> -mcp.azFiduci : v1_148 >> -mcp.altFiduc : v1_148 >> -mcp.rotFiduc : v1_148 >> tcc : 1.4.0 >> apogee : not availble >> apogeecal : not availble >> apogeeql : v1_7_0dev >> STUIAdditions : FAILED >> Boson : 0.1.2-beta.1 >> -- done -- >> --------------Other Versions-------------- >> sdss-obstools : 3.9.1 >> TPM : 1.2-31-gd7eb >> MCP : 2.1 >> PLC : V35 >> Fiducials : v1_148 >> idlspec2d : v6_0_9 >> Kronos : 1.5.0 >> roboscheduler : 1.2.2 >> >> /dev/sdb1 2.0T 222G 1.8T 11% /home >> 10.25.1.230:/data 21T 17T 3.4T 84% /data >> >> >> >> ---- APOGEE Data Summary ---- >> >> list_ap -m 59893 - morning >> >> 59893 76 12:59 43310076 Dark 60 A 13.0 ---------- a-b-c - >> - >> 59893 77 13:10 43310077 Dark 60 A 13.0 ---------- a-b-c - >> - >> 59893 78 13:21 43310078 Dark 60 A 13.0 ---------- a-b-c - >> - >> 59893 79 13:23 43310079 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 59893 80 13:25 43310080 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 59893 81 13:26 43310081 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 59893 82 13:29 43310082 Arc-Thar 12 B 13.5 ---------- a-b-c 3.11 >> - >> 59893 83 13:36 43310083 Arc-Une 40 B 13.5 ---------- a-b-c - >> - >> 59893 84 13:38 43310084 Arc-Thar 12 A 13.0 ---------- a-b-c 2.61 >> - >> 59893 85 13:45 43310085 Arc-Une 40 A 13.0 ---------- a-b-c - >> - >> 59893 86 13:51 43310086 Dark 30 A 13.0 ---------- a-b-c - >> - >> 59893 87 13:56 43310087 InteFlat 30 A 13.0 ---------- a-b-c - >> - >> 59893 88 14:01 43310088 InteFlat 30 A 13.0 ---------- a-b-c - >> - >> 59893 89 14:07 43310089 InteFlat 30 A 13.0 ---------- a-b-c - >> - >> 59893 90 14:12 43310090 Dark 30 A 13.0 ---------- a-b-c - >> - >> 59893 91 14:18 43310091 Arc---- 30 A 13.0 ---------- a-b-c - >> 91 FPI >> APOGEE data list, mjd=59894, all >> raw_data: /data/apogee/utr_cdr/59894/apRaw-*.fits >> archive: /data/apogee/archive/59894/apR-[a,b,c]-*.apz >> quickred: /data/apogee/quickred/59894/apq-*.fits >> ------------------------------------------------------------------------------ >> mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset >> %Flux/FPI Comm >> ------------------------------------------------------------------------------ >> 59894 1 21:33 43320001 Dark 3 A 13.0 ---------- a-b-c - >> - >> 59894 2 22:52 43320002 Dark 60 A 13.0 ---------- a-b-c - >> - >> 59894 3 22:54 43320003 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 59894 4 22:56 43320004 Arc-Thar 12 B 13.5 ---------- a-b-c 3.07 >> - >> 59894 5 23:03 43320005 Arc-Une 40 B 13.5 ---------- a-b-c - >> - >> 59894 6 23:05 43320006 Arc-Thar 12 A 13.0 ---------- a-b-c 2.60 >> - >> 59894 7 23:13 43320007 Arc-Une 40 A 13.0 ---------- a-b-c - >> - >> 59894 8 23:24 43320008 Dark 60 A 13.0 ---------- a-b-c - >> - >> 59894 9 23:29 43320009 Arc---- 30 A 13.0 ---------- a-b-c - >> 90 FPI >> 59894 10 00:13 43320010 DomeFlat 5 A 13.0 -999-7413 a-b-c - >> 147 >> 59894 11 00:14 43320011 DomeFlat 5 A 13.0 -999-7413 a-b-c - >> 145 >> 59894 12 00:15 43320012 DomeFlat 5 A 13.0 -999-7413 a-b-c - >> 144 >> 59894 13 00:48 43320013 Object 43 A 13.0 37788-7414 a-b-c 1.42 >> 1,1 >> 59894 14 00:56 43320014 Object 43 B 13.5 37788-7414 a-b-c 1.97 >> 1,1 >> 59894 15 01:11 43320015 Object 43 B 13.5 119646-7415 a-b-c >> 1.88 >> 1,1 >> cannot read file : /data/apogee/utr_cdr/59894/apRaw-43320016.fits >> 59894 17 01:33 43320017 Object 43 A 13.0 37788-7416 a-b-c 1.43 >> 1,1 >> 59894 18 01:40 43320018 Object 43 B 13.5 37788-7416 a-b-c 1.97 >> 1,1 >> 59894 19 01:55 43320019 Object 43 B 13.5 122369-7417 a-b-c >> 1.34 >> 1,1 >> 59894 20 02:03 43320020 Object 43 A 13.0 122369-7417 a-b-c >> 1.39 >> 1,1 >> 59894 21 02:18 43320021 Object 43 A 13.0 114083-7418 a-b-c >> 2.08 >> 1,1 >> 59894 22 02:25 43320022 Object 43 B 13.5 114083-7418 a-b-c >> 2.59 >> 1,1 >> 59894 23 02:40 43320023 Object 43 B 13.5 114084-7419 a-b-c >> 2.40 >> 1,1 >> 59894 24 02:48 43320024 Object 43 A 13.0 114084-7419 a-b-c >> 1.87 >> 1,1 >> 59894 25 03:03 43320025 Object 43 A 13.0 128413-7420 a-b-c >> 1.35 >> 1,1 >> 59894 26 03:11 43320026 Object 43 B 13.5 128413-7420 a-b-c >> 1.86 >> 1,1 >> 59894 27 03:26 43320027 Object 43 B 13.5 125638-7421 a-b-c >> 1.85 >> 1,1 >> 59894 28 03:34 43320028 Object 43 A 13.0 125638-7421 a-b-c >> 1.32 >> 1,1 >> 59894 29 03:48 43320029 Object 43 A 13.0 115217-7422 a-b-c >> 1.57 >> 1,1 >> 59894 30 03:56 43320030 Object 43 B 13.5 115217-7422 a-b-c >> 2.05 >> 1,1 >> 59894 31 04:11 43320031 Object 43 B 13.5 115095-7423 a-b-c >> 1.94 >> 1,1 >> 59894 32 04:19 43320032 Object 43 A 13.0 115095-7423 a-b-c >> 1.43 >> 1,1 >> 59894 33 04:34 43320033 Object 43 A 13.0 125634-7424 a-b-c >> 1.41 >> 1,1 >> 59894 34 04:41 43320034 Object 43 B 13.5 125634-7424 a-b-c >> 1.94 >> 1,1 >> 59894 35 04:56 43320035 Object 43 B 13.5 119037-7425 a-b-c >> 1.95 >> 1,1 >> 59894 36 05:04 43320036 Object 43 A 13.0 119037-7425 a-b-c >> 1.48 >> 1,1 >> 59894 37 05:13 43320037 Object 43 A 13.0 119037-7425 a-b-c >> 1.47 >> 1,1 >> 59894 38 05:21 43320038 Object 43 B 13.5 119037-7425 a-b-c >> 1.93 >> 1,1 >> 59894 39 05:35 43320039 Object 43 B 13.5 119038-7426 a-b-c >> 1.88 >> 1,1 >> 59894 40 05:42 43320040 Object 43 A 13.0 119038-7426 a-b-c >> 1.35 >> 1,1 >> 59894 41 05:57 43320041 Object 43 A 13.0 119039-7427 a-b-c >> 1.36 >> 1,1 >> 59894 42 06:04 43320042 Object 43 B 13.5 119039-7427 a-b-c >> 1.89 >> 1,1 >> 59894 43 06:19 43320043 Object 43 B 13.5 119040-7428 a-b-c >> 1.89 >> 1,1 >> 59894 44 06:27 43320044 Object 43 A 13.0 119040-7428 a-b-c >> 1.39 >> 1,1 >> 59894 45 07:50 43320045 DomeFlat 5 A 13.0 126142-7430 a-b-c - >> 136 >> 59894 46 07:51 43320046 DomeFlat 5 A 13.0 126142-7430 a-b-c - >> 0 >> 59894 47 07:52 43320047 DomeFlat 5 A 13.0 126142-7430 a-b-c - >> 135 >> ------------------------------------------------------------------------------ >> >> >> >> ---- eBOSS Data Summary ---- >> >> Reading FITS from /data/spectro//59894 >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - >> - - >> n Time Filename design-field cfg flav shu Exp colA colB >> colC Hrt >> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - >> - >> - - >> 1 22:21 b1-00348211 ----------- ---- bia Clo 0.0 -1959 -1823 >> -3157 >> 2 22:21 r1-00348211 ----------- ---- bia Clo 0.0 -1959 -1823 >> -3157 >> 3 22:24 b1-00348212 ----------- ---- dar Clo 20.0 -1959 -1823 >> -3157 >> 4 22:24 r1-00348212 ----------- ---- dar Clo 20.0 -1959 -1823 >> -3157 >> 5 22:32 b1-00348213 123397-102977 7412 arc Lef 4.1 -1959 -1823 >> -3157 HART >> 6 22:32 r1-00348213 123397-102977 7412 arc Lef 4.1 -1959 -1823 >> -3157 HART >> 7 22:33 b1-00348214 123397-102977 7412 arc Rig 4.1 -1959 -1823 >> -3157 HART >> 8 22:33 r1-00348214 123397-102977 7412 arc Rig 4.1 -1959 -1823 >> -3157 HART >> 9 22:34 b1-00348215 123397-102977 7412 arc Lef 4.1 -1959 -1823 >> -3157 HART >> 10 22:34 r1-00348215 123397-102977 7412 arc Lef 4.1 -1959 -1823 >> -3157 HART >> 11 22:35 b1-00348216 123397-102977 7412 arc Rig 4.1 -1959 -1823 >> -3157 HART >> 12 22:35 r1-00348216 123397-102977 7412 arc Rig 4.1 -1959 -1823 >> -3157 HART >> 13 22:41 r1-00348217 123397-102977 7412 arc Out 4.1 -4214 -4078 >> -5411 >> 14 22:41 b1-00348217 123397-102977 7412 arc Out 4.1 -4214 -4078 >> -5411 >> 15 22:44 r1-00348218 123397-102977 7412 fla Out 25.1 -4214 -4078 >> -5411 >> 16 22:44 b1-00348218 123397-102977 7412 fla Out 25.1 -4214 -4078 >> -5411 >> 17 22:58 r1-00348219 -999--999 7413 sci Out 300.1 -4214 -4078 -5411 >> 18 22:58 b1-00348219 -999--999 7413 sci Out 300.1 -4214 -4078 -5411 >> 19 00:16 r1-00348220 -999--999 7413 fla Out 25.1 -4214 -4078 -5411 >> 20 00:16 b1-00348220 -999--999 7413 fla Out 25.1 -4214 -4078 -5411 >> 21 00:18 b1-00348221 -999--999 7413 fla Out 25.1 -4214 -4078 -5411 >> 22 00:18 r1-00348221 -999--999 7413 fla Out 25.1 -4214 -4078 -5411 >> 23 00:20 r1-00348222 -999--999 7413 fla Out 25.1 -4214 -4078 -5411 >> 24 00:20 b1-00348222 -999--999 7413 fla Out 25.1 -4214 -4078 -5411 >> 25 00:31 r1-00348223 37788-100461 7414 fla Out 25.1 -4214 -4078 >> -5411 >> 26 00:31 b1-00348223 37788-100461 7414 fla Out 25.1 -4214 -4078 >> -5411 >> 27 00:33 b1-00348224 37788-100461 7414 arc Lef 4.1 -4214 -4078 >> -5411 HART >> 28 00:33 r1-00348224 37788-100461 7414 arc Lef 4.1 -4214 -4078 >> -5411 HART >> 29 00:33 b1-00348225 37788-100461 7414 arc Rig 4.1 -4214 -4078 >> -5411 HART >> 30 00:33 r1-00348225 37788-100461 7414 arc Rig 4.1 -4214 -4078 >> -5411 HART >> 31 00:35 b1-00348226 37788-100461 7414 arc Out 4.1 -925 -789 >> -2124 >> 32 00:35 r1-00348226 37788-100461 7414 arc Out 4.1 -925 -789 >> -2124 >> 33 00:41 b1-00348227 37788-100461 7414 sci Out 900.1 -925 -789 >> -2124 >> 34 00:41 r1-00348227 37788-100461 7414 sci Out 900.1 -925 -789 >> -2124 >> 35 01:01 b1-00348228 119646-101782 7415 arc Out 4.1 -925 -789 >> -2124 >> 36 01:01 r1-00348228 119646-101782 7415 arc Out 4.1 -925 -789 >> -2124 >> 37 01:04 r1-00348229 119646-101782 7415 sci Out 900.1 -925 -789 >> -2124 >> 38 01:04 b1-00348229 119646-101782 7415 sci Out 900.1 -925 -789 >> -2124 >> 39 01:24 b1-00348230 37788-100461 7416 arc Out 4.1 -925 -789 >> -2124 >> 40 01:24 r1-00348230 37788-100461 7416 arc Out 4.1 -925 -789 >> -2124 >> 41 01:26 b1-00348231 37788-100461 7416 sci Out 900.1 -925 -789 >> -2124 >> 42 01:26 r1-00348231 37788-100461 7416 sci Out 900.1 -925 -789 >> -2124 >> 43 01:45 r1-00348232 122369-102470 7417 arc Out 4.1 -925 -789 >> -2124 >> 44 01:45 b1-00348232 122369-102470 7417 arc Out 4.1 -925 -789 >> -2124 >> 45 01:48 r1-00348233 122369-102470 7417 sci Out 900.1 -925 -789 >> -2124 >> 46 01:48 b1-00348233 122369-102470 7417 sci Out 900.1 -925 -789 >> -2124 >> 47 02:09 r1-00348234 114083-100456 7418 arc Out 4.1 -925 -789 >> -2124 >> 48 02:09 b1-00348234 114083-100456 7418 arc Out 4.1 -925 -789 >> -2124 >> 49 02:11 r1-00348235 114083-100456 7418 sci Out 900.1 -925 -789 >> -2124 >> 50 02:11 b1-00348235 114083-100456 7418 sci Out 900.1 -925 -789 >> -2124 >> 51 02:33 b1-00348236 114084-100456 7419 sci Out 900.1 -925 -789 >> -2124 >> 52 02:33 r1-00348236 114084-100456 7419 sci Out 900.1 -925 -789 >> -2124 >> 53 02:52 r1-00348237 128413-104436 7420 arc Lef 4.1 -925 -789 >> -2124 HART >> 54 02:52 b1-00348237 128413-104436 7420 arc Lef 4.1 -925 -789 >> -2124 HART >> 55 02:53 b1-00348238 128413-104436 7420 arc Rig 4.1 -925 -789 >> -2124 HART >> 56 02:53 r1-00348238 128413-104436 7420 arc Rig 4.1 -925 -789 >> -2124 HART >> 57 02:54 b1-00348239 128413-104436 7420 arc Out 4.1 -2838 -2701 >> -4038 >> 58 02:54 r1-00348239 128413-104436 7420 arc Out 4.1 -2838 -2701 >> -4038 >> 59 02:56 b1-00348240 128413-104436 7420 sci Out 900.1 -2838 -2701 >> -4038 >> 60 02:56 r1-00348240 128413-104436 7420 sci Out 900.1 -2838 -2701 >> -4038 >> 61 03:16 b1-00348241 125638-103561 7421 arc Out 4.1 -2838 -2701 >> -4038 >> 62 03:16 r1-00348241 125638-103561 7421 arc Out 4.1 -2838 -2701 >> -4038 >> 63 03:19 r1-00348242 125638-103561 7421 sci Out 900.1 -2838 -2701 >> -4038 >> 64 03:19 b1-00348242 125638-103561 7421 sci Out 900.1 -2838 -2701 >> -4038 >> 65 03:39 r1-00348243 115217-100619 7422 arc Out 4.1 -2838 -2701 >> -4038 >> 66 03:39 b1-00348243 115217-100619 7422 arc Out 4.1 -2838 -2701 >> -4038 >> 67 03:41 b1-00348244 115217-100619 7422 sci Out 900.1 -2838 -2701 >> -4038 >> 68 03:41 r1-00348244 115217-100619 7422 sci Out 900.1 -2838 -2701 >> -4038 >> 69 04:01 r1-00348245 115095-100595 7423 arc Out 4.1 -2838 -2701 >> -4038 >> 70 04:01 b1-00348245 115095-100595 7423 arc Out 4.1 -2838 -2701 >> -4038 >> 71 04:04 r1-00348246 115095-100595 7423 sci Out 900.1 -2838 -2701 >> -4038 >> 72 04:04 b1-00348246 115095-100595 7423 sci Out 900.1 -2838 -2701 >> -4038 >> 73 04:24 r1-00348247 125634-103558 7424 arc Out 4.1 -2838 -2701 >> -4038 >> 74 04:24 b1-00348247 125634-103558 7424 arc Out 4.1 -2838 -2701 >> -4038 >> 75 04:27 r1-00348248 125634-103558 7424 sci Out 900.1 -2838 -2701 >> -4038 >> 76 04:27 b1-00348248 125634-103558 7424 sci Out 900.1 -2838 -2701 >> -4038 >> 77 04:47 r1-00348249 119037-101731 7425 arc Out 4.1 -2838 -2701 >> -4038 >> 78 04:47 b1-00348249 119037-101731 7425 arc Out 4.1 -2838 -2701 >> -4038 >> 79 04:49 r1-00348250 119037-101731 7425 sci Out 900.1 -2838 -2701 >> -4038 >> 80 04:49 b1-00348250 119037-101731 7425 sci Out 900.1 -2838 -2701 >> -4038 >> 81 05:06 b1-00348251 119037-101731 7425 sci Out 900.1 -2838 -2701 >> -4038 >> 82 05:06 r1-00348251 119037-101731 7425 sci Out 900.1 -2838 -2701 >> -4038 >> 83 05:28 b1-00348252 119038-101731 7426 sci Out 900.1 -2838 -2701 >> -4038 >> 84 05:28 r1-00348252 119038-101731 7426 sci Out 900.1 -2838 -2701 >> -4038 >> 85 05:50 b1-00348253 119039-101731 7427 sci Out 900.1 -2838 -2701 >> -4038 >> 86 05:50 r1-00348253 119039-101731 7427 sci Out 900.1 -2838 -2701 >> -4038 >> 87 06:12 b1-00348254 119040-101731 7428 sci Out 900.1 -2838 -2701 >> -4038 >> 88 06:12 r1-00348254 119040-101731 7428 sci Out 900.1 -2838 -2701 >> -4038 >> 89 06:43 b1-00348255 126123-103699 7429 fla Out 25.1 -2838 -2701 >> -4038 >> 90 06:43 r1-00348255 126123-103699 7429 fla Out 25.1 -2838 -2701 >> -4038 >> 91 06:46 b1-00348256 126123-103699 7429 arc Out 4.1 -2838 -2701 >> -4038 >> 92 06:46 r1-00348256 126123-103699 7429 arc Out 4.1 -2838 -2701 >> -4038 >> 93 06:55 b1-00348257 126142-103701 7430 arc Out 4.1 -2838 -2701 >> -4038 >> 94 06:55 r1-00348257 126142-103701 7430 arc Out 4.1 -2838 -2701 >> -4038 >> >> >> ---- BOSS Spectrographs Collimation ---- >> >> Time TSP1 R1 B1 Move1 B1Resid >> ============================================= >> >> >> ---- Weather Log ---- >> >> Wind 1 um DIMM IRSC >> Time Temp C Dewp C MPH Dir Dust fwhm mean sigma >> >> 19:00 4.7 -7.9 15.3 236.0 355 >> 19:15 6.2 -7.9 14.6 246.5 438 >> 19:30 5.9 -8.4 11.5 244.7 337 >> 19:45 6.4 -8.1 16.2 237.9 350 >> 20:00 6.0 -8.6 15.2 253.1 346 >> 20:15 4.7 -7.5 11.4 252.6 393 >> 20:30 4.5 -7.5 14.9 277.7 378 >> 20:45 3.9 -8.1 15.3 265.4 377 >> 21:00 6.1 -6.5 11.8 277.6 354 >> 21:15 4.8 -6.2 9.8 251.1 398 >> 21:30 5.1 -6.5 13.9 246.3 417 >> 21:45 7.2 -5.6 16.6 264.3 459 >> 22:00 5.0 -6.6 8.1 265.0 473 >> 22:15 8.1 -4.9 12.7 251.2 432 >> 22:30 4.4 -5.6 11.1 266.0 380 >> 22:45 4.2 -5.7 9.9 263.8 444 >> 23:00 3.7 -5.7 12.2 268.7 421 >> 23:15 4.2 -4.8 10.9 264.5 409 >> 23:30 3.9 -5.6 8.4 267.2 499 >> 23:45 4.4 -4.6 8.6 262.9 441 >> 00:00 3.2 -4.9 13.3 268.4 473 >> 00:15 2.4 -4.7 10.9 265.8 480 >> 00:30 2.4 -4.7 11.2 259.8 486 342 6.51 >> 00:45 2.3 -4.4 11.2 259.8 513 342 3.05 >> 01:00 2.3 -4.5 8.3 262.1 521 342 3.76 >> 01:15 2.1 -4.6 9.4 262.5 562 342 2.51 >> 01:30 2.1 -4.4 11.2 258.3 517 342 2.44 >> 01:45 1.9 -4.6 8.9 228.0 509 323 2.44 >> 02:00 1.9 -4.3 8.9 245.4 559 342 2.34 >> 02:15 1.9 -4.4 7.1 256.0 521 342 2.71 >> 02:30 1.7 -4.6 16.8 239.2 526 342 4.52 >> 02:45 2.4 -4.7 15.0 247.6 528 342 5.88 >> 03:00 2.7 -4.6 19.2 218.8 617 342 7.16 >> 03:15 2.4 -4.3 18.7 269.7 652 342 5.55 >> 03:30 2.1 -4.0 15.8 269.0 598 254 8.35 >> 03:45 2.1 -4.1 12.5 270.8 538 273 10.37 >> 04:00 2.0 -3.8 13.3 269.0 549 244 13.84 >> 04:15 2.3 -3.3 18.9 283.8 684 344 31.24 >> 04:30 2.0 -3.8 13.7 266.0 654 355 79.02 >> 04:45 1.9 -3.7 15.2 267.7 612 355 60.24 >> 05:00 2.2 -3.3 16.1 271.5 755 363 76.60 >> 05:15 1.9 -3.6 17.4 268.0 645 351 62.56 >> 05:30 2.3 -3.1 16.1 268.4 697 363 49.16 >> 05:45 2.4 -3.1 16.4 268.7 756 341 31.20 >> 06:00 1.9 -3.6 13.9 280.7 706 357 44.70 >> 06:15 2.0 -3.3 12.8 271.8 723 250 83.21 >> 06:30 1.9 -3.6 12.2 280.3 665 382 70.99 >> 06:45 2.0 -3.7 19.5 271.1 688 364 59.09 >> 07:00 1.9 -3.9 15.3 273.5 649 395 43.72 >> 07:15 2.0 -3.8 11.7 268.1 613 361 61.97 >> 07:30 1.9 -3.8 17.4 232.8 630 380 60.41 >> 07:45 1.7 -3.8 10.3 254.8 603 369 69.00 >> 08:00 1.8 -3.3 18.7 247.7 701 367 70.16 >> 08:15 1.6 -3.8 15.8 273.1 600 365 57.16 >> 08:30 1.6 -3.3 14.9 266.4 739 357 62.04 >> 08:45 1.5 -3.6 15.9 277.8 682 293 54.73 >> 09:00 1.4 -3.6 10.3 266.3 663 360 59.87 >> 09:15 1.3 -3.7 16.6 271.0 652 356 62.76 >> 09:30 1.1 -3.7 17.9 283.9 617 371 56.27 >> 09:45 1.0 -3.9 20.0 294.2 589 364 60.07 >> 10:00 0.8 -4.1 16.2 269.9 534 385 80.05 >> 10:15 0.5 -4.2 15.9 268.7 555 401 83.44 >> 10:30 0.3 -4.2 16.3 261.8 557 381 64.21 >> >> >> ---- Time Tracking ---- >> >> *=*=*= Observatory: APO >> *=*=*= MJD: 59894 >> Fill night time in Hours below >> *=*=*= Total bright time : 11.5 >> *=*=*= Bright time, queued observations : 5.4 >> *=*=*= Bright time, commissioning/engineering: 0 >> *=*=*= Bright time lost to weather : 6.1 >> *=*=*= Bright time lost to technical issues : 0 >> >> *=*=*= Total dark time : 0 >> *=*=*= Dark time, queued observations : 0 >> *=*=*= Dark time, commissioning/engineering : 0 >> *=*=*= Dark time lost to weather : 0 >> *=*=*= Dark time lost to technical issues : 0 >> >> >> >> >> ____________________________________________________________ >> Logs of Nightly APO Activities >> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html >> > > > > > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html > > >