From: Conor Sayres Subject: Re: [sloan-25mlog 6409] Re: 2.5m obslog 20221214 (SJD 59928) Date: Thu, 15 Dec 2022 13:32:18 -0500 --Apple-Mail=_F20BE176-3F3D-4F6D-B375-4CD4ADF12413 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 I think the issue is a little more pathological. I recall that you can = connect (eg the port is up), but the other end is always silent, so = pretty much everything just times out. > On Dec 15, 2022, at 1:30 PM, Jos=C3=A9 S=C3=A1nchez-Gallego = wrote: >=20 > Forgot to mention that we did that. Or I think we did. The = instructions are here >=20 > = https://wiki.sdss.org/display/APO/Procedure+to+Logout+Hung+Communication+P= orts = >=20 > and there's an script in ~sdss5/bin that John wrote to automate the = process. >=20 > My recollection was that when a port is hung it won't let you connect = to it, not that it will sometimes disconnect, but in any case clearing = those ports may be another thing to try. >=20 > Jos=C3=A9 >=20 >=20 > On Dec 15, 2022 at 10:19:24, Conor Sayres > wrote: >> =46rom ancient history, I recall that the xyplex terminal server = needs to have =E2=80=9Cports cleared=E2=80=9D, otherwise we see = occasional hanging connections. Do the observers know (remember?) how = to do this? =20 >>=20 >>> On Dec 15, 2022, at 1:00 PM, Jos=C3=A9 S=C3=A1nchez-Gallego = > wrote: >>>=20 >>> Hi all, >>>=20 >>> With respect to the BOSS (and sometimes axis) issues that kept us = from observing yesterday ... well they are very annoyingly not showing = up today. I took several 900s images and they all completed well. Also = seems that there weren't any additional disconnections last night after = we closed. >>>=20 >>> I don't believe in coincidences, so I would suspect that something = that we did yesterday caused this, but the code itself hasn't changed = (only the format of /data and where it's mounted, but that should not = matter to sdss5-boss-icc or sdss5-tcc), and when it works it works fine = as usual. >>>=20 >>> The problem seems to be that sometimes the ICC or TCC lose = connection to the controllers on the dome side. I think BOSS is more = obvious because it lacks automatic reconnection capabilities, while the = TCC does some reconnecting by itself. >>>=20 >>> All the devices that seem to be failing are connected through the = xyplex terminal server, and we saw some weird behaviour with it = yesterday, in which at some point it stopped letting us into the = management console, but this morning it was fine again. We saw a weird = case with the FPS, but I'm willing to attribute that to something else, = but it's worth looking into other hardware that enable communication = with the telescope. >>>=20 >>> Has there been any change on the telescope side that could explain = these very intermittent disconnects? I could believe that something = related with UPS or power could be at play here. >>>=20 >>> If not, I suggest we try observing tonight ... If it starts failing = again my first advice would be to reboot the xyplex (I think yesterday = it was an edge server that was restarted yesterday). >>>=20 >>> I think Shane is monitoring that device now, maybe we'll learn = something if it happens again. >>>=20 >>> Thanks, >>>=20 >>> Jos=C3=A9 >>>=20 >>>=20 >>> On Dec 15, 2022 at 04:04:36, asimmons@apo.nmsu.edu = wrote: >>>>=20 >>>> APACHE POINT OBSERVATORY SDSS 2.5M >>>>=20 >>>> OBSERVING LOG >>>> Wednesday December 14, 2022 >>>>=20 >>>> (SJD 59928) >>>>=20 >>>> ---- Observing Team ---- >>>>=20 >>>> Audrey Oravetz >>>>=20 >>>> Jos=C3=A9 S=C3=A1nchez-Gallego (online support) >>>> John Donor (online support) >>>> Shane Thomas (on site support) >>>>=20 >>>> ---- Observing Plan ---- >>>>=20 >>>> Wed 14 Dec 2022 59928 0.61 00:56 01:11 12:45 23:42 11:18 --- dark = until 05:52 ( 4.7 h), then bright ( 7.1 h, w/ 0.3 h for 12 deg design) >>>>=20 >>>>=20 >>>> ---- Observing Summary ---- >>>>=20 >>>> The first half of the night could be treated as a flexible = engineering due to the problems we encountered from the /data disk = transfer. >>>>=20 >>>> The first 1.3 hrs were used to get all systems back running and = afternoon tasks completed.=20 >>>>=20 >>>> After we opened, we operated smoothly for the most part until just = before 04:00Z when we began witnessing multiple BOSS camera controller = disconnections. Mostly it was sp1cam, but once was sp1mech. To get them = reconnected, we logged out ports and restarted the bossICC. We also = witnessed the M2 mirror controller hanging, as well as some suspicious = behavior with the FPS during a trajectory unwind, again pointing to a = potential network or server error. The terminal server was then rebooted = with a very small reset button, but did not resolve the issues (unless a = physical unplug/replug of the power cord is needed instead). More = investigation is needed tomorrow.=20 >>>>=20 >>>> The humidity spiked and remained high the rest of the night past = 07:00Z. >>>>=20 >>>> Observed: >>>> --------- >>>> Time field-dsgn-conf Design type BOSS/APOGEE exp complete? = Comment >>>> = =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= >>>> 01:36:15 104666-130833-7898 dark_monit. 2x900 + 2XAB. Not done. = * one exposure taken with this design was mistakenly loaded into the = next design, but the robots never moved. =20 >>>> 02:31:00 104666-130834-7899 dark_monit. 2x900 + 2XAB. DONE >>>> 03:40:35 104666-130835-7900 dark_monit. 1x900 + 3XAB. Not done >>>>=20 >>>>=20 >>>> ---- Observing Log ---- >>>>=20 >>>> Afternoon: >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>>=20 >>>> Checked actors, mirror numbers. Attempted to begin taking afternoon = cals, and hence restarting a while bunch of systems, but mistakenly = interrupted the on-going work with the /data transfer. Verified with = Jose that everything is ok and they will continue to work. Note that we = will be delayed getting on sky on time tonight.=20 >>>>=20 >>>> 23:40Z - Started ventilating.=20 >>>>=20 >>>> 00:02Z - Opened the telescope and filled the cameras now in an = attempt to save time later. >>>>=20 >>>> Night: >>>> =3D=3D=3D=3D=3D=3D >>>>=20 >>>> Took BOSS cals and APOGEE cals. Turned on the FPS.=20 >>>>=20 >>>> The DAQ was not responding properly after the /data work. No = exposures were being written to disk. Everything would hang after = integration. Jose came online. I reset the DAQ on the specrograph = itself. Jose did his magic. We could take images again.=20 >>>>=20 >>>> 00:56:34 boss exposure 4.0, file=3D349872=20 >>>> 00:57:17 boss exposure 4.0, file=3D349873=20 >>>> 00:58:06 sp1.motor.move=3D 2298, 2298, 2298 >>>> 00:58:06 Hartmann collimate output >>>> sp1: offset: r=3D 1319 (Out of focus); b=3D -10.3 (Out of focus)=20= >>>> sp1: pred. move: spAverageMove=3D 2298 >>>> sp1: pred. resid: r=3D -979, b=3D -3.1, txt=3D OK, spTemp =3D = -0.5 >>>> 00:58:39 boss exposure 4.0, file=3D349874=20 >>>> 01:00:11 boss exposure 25.0, file=3D349875=20 >>>> 01:02:08 apogee.expose=3DapRaw-43660006, 10, A, QuartzFlat >>>>=20 >>>> ******************************************************** >>>> 01:03:35 Field-Design-Configuration: -999--999-7897 >>>> ******************************************************** >>>> 01:04:04 apogee.expose=3DapRaw-43660007, 12, B, ArcLamp >>>> 01:05:06 boss exposure 300.0, file=3D349876=20 >>>> 01:06:16 apogee.expose=3DapRaw-43660008, 40, B, ArcLamp >>>> 01:13:33 apogee.expose=3DapRaw-43660009, 12, A, ArcLamp >>>> 01:13:52 boss exposure 25.0, file=3D349877=20 >>>> 01:15:46 boss exposure 25.0, file=3D349878=20 >>>> 01:15:46 apogee.expose=3DapRaw-43660010, 40, A, ArcLamp >>>> 01:17:43 boss exposure 25.0, file=3D349879=20 >>>> 01:23:12 apogee.expose=3DapRaw-43660011, 60, A, Dark >>>> 01:34:04 apogee.expose=3DapRaw-43660012, 30, A, ArcLamp >>>>=20 >>>> 01:03:39 APO.aoravetz 46 jaeger i = configuration_snapshot=3D/data/logs/jaeger/configuration_snapshots/59928/c= onfiguration_snapshot_59928_7897.pdf >>>> 01:03:39 APO.aoravetz 46 jaeger : text=3D"Configuration 7897 = loaded." >>>> 01:03:39 CmdDone: APO.aoravetz 46 jaeger configuration load = --from-positions >>>>=20 >>>> 01:04:44 APO.aoravetz 45 fliswarm i = filename=3Dfvc,fvc1n,/data/fcam/59928/fimg-fvc1n-0001.fits >>>> 01:04:44 APO.aoravetz 45 fliswarm i = exposure_state=3Dfvc,fvc1n,idle,NA,0.0,0,0,0 >>>> 01:04:44 APO.aoravetz 45 fliswarm i = filename_bundle=3D/data/fcam/59928/fimg-fvc1n-0001.fits >>>> 01:04:44 APO.aoravetz 45 fliswarm :=20 >>>> 01:04:44 CmdDone: APO.aoravetz 45 fliswarm talk -n fvc expose 50 >>>>=20 >>>> (observers39) [observer@sdss5-hub: ~]$ list_ap >>>> APOGEE data list, mjd=3D59928, all >>>> raw_data: /data/apogee/utr_cdr/59928/apRaw-*.fits >>>> archive: /data/apogee/archive/59928/apR-[a,b,c]-*.apz >>>> quickred: /data/apogee/quickred/59928/apq-*.fits >>>> = --------------------------------------------------------------------------= ---- >>>> mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch = Offset %Flux/FPI Comm >>>> = --------------------------------------------------------------------------= ---- >>>> 59928 1 no file found: = /data/apogee/utr_cdr/59928/apRaw-43660001.fits >>>> 59928 2 no file found: = /data/apogee/utr_cdr/59928/apRaw-43660002.fits >>>> 59928 3 no file found: = /data/apogee/utr_cdr/59928/apRaw-43660003.fits >>>> 59928 4 23:42 43660004 Dark 60 A 13.0 ---------- a-b-c = - - >>>> 59928 5 01:01 43660005 Dark 60 A 13.0 ---------- a-b-c = - - >>>> 59928 6 01:03 43660006 QuarFlat 10 A 13.0 ---------- a-b-c = - - >>>> 59928 7 01:05 43660007 Arc---- 12 B 13.5 ---------- a-b-c = - 13 >>>> 59928 8 01:12 43660008 Arc---- 40 B 13.5 ---------- a-b-c = - 6 >>>> 59928 9 01:15 43660009 Arc---- 12 A 13.0 ---------- a-b-c = - 12 >>>> 59928 10 01:22 43660010 Arc---- 40 A 13.0 ---------- a-b-c = - 8 >>>> 59928 11 01:33 43660011 Dark 60 A 13.0 ---------- a-b-c = - - >>>> 59928 12 01:38 43660012 Arc---- 30 A 13.0 ---------- a-b-c = - 45 FPI >>>> 59928 13 01:43 43660013 DomeFlat 5 A 13.0 --------- a-b-c = - 130 >>>> 59928 14 01:44 43660014 DomeFlat 5 A 13.0 --------- a-b-c = - 140 >>>> 59928 15 01:45 43660015 DomeFlat 5 A 13.0 --------- a-b-c = - 131 >>>>=20 >>>> The FPI afternoon cal was a bit faint at 45%. Not sure what the = value was yesterday. The dome flats seem ok.=20 >>>>=20 >>>> BSONO was not preloading the designs from KRONOS properly. Jose is = aware and is working on creating another BOSON version. >>>>=20 >>>> ******************************************************** >>>> 01:36:15 Field-Design-Configuration: 104666-130833-7898 >>>> ******************************************************** >>>>=20 >>>> 01:42:43 apogee.expose=3DapRaw-43660013, 5, A, DomeFlat >>>> 01:43:58 apogee.expose=3DapRaw-43660014, 5, A, DomeFlat >>>> 01:45:12 apogee.expose=3DapRaw-43660015, 5, A, DomeFlat >>>>=20 >>>> 01:46:12 GOTO-FIELD >>>> 01:48:57 FVC LOOP AND BOSS CALS >>>> fvc_rms=3D 7.95 >>>> fvc_deltarms=3D 29.53 >>>> fvc_filename=3D/data/fcam/59928/fimg-fvc1n-0003.fits >>>> fvc_perc_90=3D 11.30 >>>> fvc_percent_reached=3D 50.30 >>>> 01:49:26 boss exposure 25.0, file=3D349880=20 >>>> 01:51:21 boss exposure 4.0, file=3D349881=20 >>>> 01:52:02 boss exposure 4.0, file=3D349882=20 >>>> 01:52:44 sp1.motor.move=3D 36, 37, 31 >>>> 01:52:44 Hartmann collimate output >>>> sp1: offset: r=3D 329 (In Focus); b=3D 0.8 (In Focus)=20 >>>> sp1: pred. move: spAverageMove=3D 36 >>>> sp1: pred. resid: r=3D 293, b=3D 0.9, txt=3D OK, spTemp =3D -1.4 >>>> 01:53:04 boss exposure 4.0, file=3D349883=20 >>>>=20 >>>> 02:04:17 .alerts 0 alerts e alert=3Dboss.camCheck.stale,critical,"at = 2022-12-15 01:56:01; no change for 615 sec, with None",enabled,noack,0 >>>>=20 >>>> Reloaded the controllers. Not sure why this alert spawned, but it = was real.=20 >>>>=20 >>>> 02:14:47 apogee.expose=3DapRaw-43660016, 43, A, Object >>>> 02:14:54 boss exposure 900.0, file=3D349884=20 >>>> 02:22:29 apogee.expose=3DapRaw-43660017, 43, B, Object >>>>=20 >>>> ******************************************************** >>>> 02:30:07 Field-Design-Configuration: 104666-130834-7899 >>>> ******************************************************** >>>>=20 >>>> ***** The pre-loading test for Jose was accidentally implemented = and the S?N was not checked beforehand. **No robots were moved into a = new configuration**. The exposures below are associated with the = previous configuration and design.=20 >>>>=20 >>>>=20 >>>> 02:31:08 Same-field overhead: 0:00:08 >>>> 02:31:22 apogee.expose=3DapRaw-43660018, 43, B, Object >>>> 02:31:29 boss exposure 900.0, file=3D349885=20 >>>> 02:39:08 apogee.expose=3DapRaw-43660019, 43, A, Object >>>>=20 >>>> ******************************************************** >>>> 02:30:07 Field-Design-Configuration: 104666-130834-7899 >>>> ******************************************************** >>>>=20 >>>> The following images ARE associated with the new design and = configuration.=20 >>>>=20 >>>> 02:31:08 Same-field overhead: 0:00:08 >>>> 02:31:22 apogee.expose=3DapRaw-43660018, 43, B, Object >>>> 02:31:29 boss exposure 900.0, file=3D349885=20 >>>> 02:39:08 apogee.expose=3DapRaw-43660019, 43, A, Object >>>> 02:47:34 GOTO-FIELD >>>> 02:50:19 FVC LOOP AND BOSS CALS >>>> fvc_rms=3D 8.33 >>>> fvc_deltarms=3D 30.68 >>>> fvc_filename=3D/data/fcam/59928/fimg-fvc1n-0005.fits >>>> fvc_perc_90=3D 10.00 >>>> fvc_percent_reached=3D 51.70 >>>> 02:51:59 START SCIENCE EXPOSURES >>>> 02:52:21 apogee.expose=3DapRaw-43660020, 43, A, Object >>>> 02:52:29 boss exposure 900.0, file=3D349886=20 >>>> 03:00:03 apogee.expose=3DapRaw-43660021, 43, B, Object >>>> 03:08:50 apogee.expose=3DapRaw-43660022, 43, B, Object >>>> 03:08:58 boss exposure 900.0, file=3D349887=20 >>>> 03:16:37 apogee.expose=3DapRaw-43660023, 43, A, Object >>>> 03:25:20 apogee.expose=3DapRaw-43660024, 43, A, Object >>>> 03:25:28 boss exposure 900.0, file=3D349888=20 >>>> 03:33:01 apogee.expose=3DapRaw-43660025, 43, B, Object >>>>=20 >>>> ******************************************************** >>>> 03:40:35 Field-Design-Configuration: 104666-130835-7900 >>>> ******************************************************** >>>>=20 >>>> 03:41:35 GOTO-FIELD >>>> 03:43:56 FVC LOOP AND BOSS CALS >>>> fvc_rms=3D 9.43 >>>> fvc_deltarms=3D 29.13 >>>> fvc_filename=3D/data/fcam/59928/fimg-fvc1n-0007.fits >>>> fvc_perc_90=3D 10.20 >>>> fvc_percent_reached=3D 51.80 >>>> 03:44:48 Same-field overhead: 0:03:12 >>>>=20 >>>> Only APOGEE started because BOSS was having cam and mech connection = errors. >>>>=20 >>>> 03:44:48 START SCIENCE EXPOSURES >>>> 03:45:10 apogee.expose=3DapRaw-43660026, 43, B, Object >>>> 03:52:57 apogee.expose=3DapRaw-43660027, 43, A, Object >>>>=20 >>>> There are continuous connection errors in the logs, but yet the = system seemed ok for awhile. >>>>=20 >>>> 04:04:54 .boss 0 boss w text=3D"Controller sp1cam disconnected." >>>> 04:04:54 .boss 0 boss w text=3D"sp1cam removed from available = hardware." >>>>=20 >>>> Logged out ports and reloaded the controllers >>>>=20 >>>> Critical boss.specMechProtocol at 2022-12-15 04:08:55 UT found 0 >>>>=20 >>>> Restarted the bossICC.=20 >>>>=20 >>>> Things are working again.=20 >>>>=20 >>>> 04:08:55 sp1.motor.move=3D None, None, None >>>> 04:09:33 sp1.motor.move=3D None, None, None >>>> 04:11:36 apogee.expose=3DapRaw-43660028, 43, A, Object >>>> 04:11:47 boss exposure 900.0, file=3D349889=20 >>>>=20 >>>> 04:19:18 apogee.expose=3DapRaw-43660029, 43, B, Object >>>> 04:28:23 apogee.expose=3DapRaw-43660030, 43, B, Object >>>> 04:28:37 boss exposure 900.0, file=3D349892=20 >>>> 04:36:09 apogee.expose=3DapRaw-43660031, 43, A, Object >>>>=20 >>>> 04:48:07 CmdStarted: APO.aoravetz 162 boss exposure recover >>>> 04:48:07 APO.aoravetz 162 boss f text=3D"No exposure to act on." >>>> 04:48:07 CmdDone: APO.aoravetz 162 boss exposure recover >>>>=20 >>>> The exposure was not able to be recovered. >>>>=20 >>>> HORRIBLE CRAZY NOISES WHEN ALERTS WENT HAYWIRE >>>>=20 >>>> 04:52:31 APO.aoravetz 1338 stui F Timeout; Actor=3D'alerts'; = Cmd=3D"status" >>>>=20 >>>> Critical boss.heartbeat at 2022-12-15 04:54:09; last seen 135 sec = ago >>>> Serious mcp.heartbeat at 2022-12-15 04:53:45; last seen 160 sec ago >>>> Warn boss.camCheck.stale at 2022-12-15 04:54:10; no change for 135 = sec, with None >>>>=20 >>>> Axes halted. Stop button in. By 05:05:23, everything was back to = normal. Connections were re-established all on their own.=20 >>>>=20 >>>> ******************************************************** >>>> 05:08:15 Field-Design-Configuration: 103743-126372-7901 >>>> ******************************************************** >>>> 05:08:56 GOTO-FIELD >>>> 05:09:40 GOTO-FIELD >>>> 05:11:27 GOTO-FIELD >>>> 05:13:41 FVC LOOP AND BOSS CALS >>>> fvc_rms=3D 11.84 >>>> fvc_deltarms=3D 23.85 >>>> fvc_filename=3D/data/fcam/59928/fimg-fvc1n-0009.fits >>>> fvc_perc_90=3D 11.10 >>>> fvc_percent_reached=3D 45.00 >>>>=20 >>>> Discovered that the secondary mirror was still moving, 1600 = sesconds. tcc talk sec stop was issued, but it retunred with 'TIMEOUT'. = I was in the process of looking back through the notes to enter into the = mirror controllers when it magically reconnected and went back to idle.=20= >>>>=20 >>>> The gotofield hung again at Flushing for the ARC.=20 >>>>=20 >>>> At this point, Jose recommended that we reboot the terminal server = completely.=20 >>>>=20 >>>> It was rebooted. The TCC disconnected and reconnected.=20 >>>>=20 >>>> ******************************************************** >>>> 05:50:53 Field-Design-Configuration: 103721-126260-7902 >>>> ******************************************************** >>>> 05:50:58 GOTO-FIELD >>>>=20 >>>> 05:51:09 hal.hal 363 jaeger f error=3D"Trajectory failed with = error: Something went wrong sending the trajectory: Failed sending = SEND_NEW_TRAJECTORY" >>>> 05:51:09 CmdDone: hal.hal 363 jaeger configuration reverse >>>> 05:51:09 APO.HushedOrangutan 30 hal e error=3D"Command jaeger = configuration reverse failed." >>>>=20 >>>> jaeger reload re-established the connection to the CAN devices. >>>>=20 >>>> Shane was contacted for his opinion.=20 >>>>=20 >>>> After an hour or so, Jose and Shane were out of ideas and could not = understand why the connections were poor.=20 >>>>=20 >>>> 06:45:33 apogee.expose=3DapRaw-43660032, 5, A, DomeFlat >>>> 06:46:47 apogee.expose=3DapRaw-43660033, 5, A, DomeFlat >>>> 06:48:02 apogee.expose=3DapRaw-43660034, 5, A, DomeFlat >>>>=20 >>>> 06:53:41 encl25m : closed; Due to high humidity. Turned off the = FPS. >>>>=20 >>>> It remained humid the rest of the night.=20 >>>>=20 >>>>=20 >>>> ---- Mirror Numbers ---- >>>>=20 >>>>=20 >>>> PRIMARY: >>>> -------- >>>> Scale: 1.000000 >>>>=20 >>>> MIGS TONIGHT NOMINAL >>>> Axial A: 0.0080 0.0080 >>>> Axial B: 0.0030 0.0040 >>>> Axial C: -0.0310 -0.0290 >>>> Trans D: 0.0320 0.0320 >>>> Lateral E: 0.3315 0.3320 >>>> Lateral F: 0.5355 0.5350 >>>>=20 >>>> GALILS >>>> Commanded: 12500 -1600 5300 47050 -8800 -8400 >>>> Actual: 12500 -1600 5300 47050 -8800 -8400 >>>>=20 >>>> SETMIR VALUES >>>> PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 >>>> PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 >>>>=20 >>>> SECONDARY: >>>> ---------- >>>>=20 >>>> Focus: 0.0000 >>>> Air temp: 92.83 >>>> Altitude: 30.0 >>>>=20 >>>> MIGS TONIGHT NOMINAL >>>> Axial A: -0.0320 -0.0330 >>>> Axial B: -0.0280 -0.0290 >>>> Axial C: -0.0450 -0.0450 >>>> Trans D: -0.2670 -0.3170 >>>>=20 >>>> GALILS >>>> Commanded: 4905050 4881950 4974150 0 -3400 >>>> Actual: 4859268 4859148 4859262 0 -3400 >>>>=20 >>>> SETMIR VALUES >>>> SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 >>>> SecOrient: 3857.01 -0.02 -0.03 -0.02 41.65 >>>>=20 >>>>=20 >>>>=20 >>>> ---- Telescope Status ---- >>>>=20 >>>> 23:45Z Started ventilation >>>> 00:05Z encl25m : open >>>> 06:53Z encl25m : closed >>>>=20 >>>> No interlocks bypassed >>>>=20 >>>> Status at: 11:14Z >>>> Telescope stowed at: 121.0, 30.0, 0.0 mount >>>> Instrument mounted: FPS =20 >>>> Counterweights at: 685/710 =20 >>>> LN2 autofill systems: Connected and turned on >>>> 180L LN2 dewar scales: SP1 171 lbs, 19.7 psi >>>>=20 >>>> FPS Status >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> CP10 Loop =3D 7.19 L/min >>>> GFA Loop =3D 1.71 L/min >>>> Supply pressure =3D 42.2 psi >>>> Return pressure =3D 29.2 psi >>>>=20 >>>> Chiller Status >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> Set temperarature =3D 0.9 C >>>> Flow =3D 5.3 Gpm >>>>=20 >>>>=20 >>>>=20 >>>> ---- Software Used ---- >>>>=20 >>>> Get Versions, 12/14/22, 23:55:29 >>>> -- STUI software:=20 >>>> STUI : 1.7.18b1 >>>> alerts : 2.3.0 >>>> boss : 6.0.2-alpha.0 >>>> -boss.specMec : 3.0.4 >>>> apo : 3.0.1 >>>> fliswarm : 0.3.1 >>>> jaeger : 1.2.0b2 >>>> kaiju : 1.2.4-alpha.0 >>>> coordio : 1.5.0-beta.2 >>>> fps_calibrati : 2022.09.11+c5432fa >>>> cherno : 0.4.0b3 >>>> astrometrynet : 0.91 >>>> hal : 0.3.1a0 >>>> hartmann : 2.0.2 >>>> hub : 5.0.1a0 >>>> mcp : 2.1 >>>> -mcp.plc : 35 >>>> -mcp.azFiduci : v1_148 >>>> -mcp.altFiduc : v1_148 >>>> -mcp.rotFiduc : v1_148 >>>> tcc : 1.4.0 >>>> apogee : not availble >>>> apogeecal : not availble >>>> apogeeql : v1_7_0dev >>>> STUIAdditions : FAILED >>>> -- done -- >>>> (observers39) [observer@sdss5-hub: ~]$ spVersion >>>> --------------Other Versions-------------- >>>> sdss-obstools : 3.9.2 >>>> TPM : 1.2-31-gd7eb >>>> MCP : 2.1 >>>> PLC : V35 >>>> Fiducials : v1_148 >>>> idlspec2d : v6_0_9 >>>> Kronos : 1.5.0 >>>> roboscheduler : 1.2.2 >>>>=20 >>>> /dev/sdb1 2.0T 235G 1.8T 12% /home >>>> /dev/sdd1 22T 230G 21T 2% /data >>>>=20 >>>>=20 >>>> ---- APOGEE Data Summary ---- >>>>=20 >>>> APOGEE data list, mjd=3D59928, all >>>> raw_data: /data/apogee/utr_cdr/59928/apRaw-*.fits >>>> archive: /data/apogee/archive/59928/apR-[a,b,c]-*.apz >>>> quickred: /data/apogee/quickred/59928/apq-*.fits >>>> = --------------------------------------------------------------------------= ----=20 >>>> mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch = Offset %Flux/FPI Comm=20 >>>> = --------------------------------------------------------------------------= ---- >>>> 59928 1 no file found: = /data/apogee/utr_cdr/59928/apRaw-43660001.fits=20 >>>> 59928 2 no file found: = /data/apogee/utr_cdr/59928/apRaw-43660002.fits=20 >>>> 59928 3 no file found: = /data/apogee/utr_cdr/59928/apRaw-43660003.fits=20 >>>> 59928 4 23:42 43660004 Dark 60 A 13.0 ---------- a-b-c = - - =20 >>>> 59928 5 01:01 43660005 Dark 60 A 13.0 ---------- a-b-c = - - =20 >>>> 59928 6 01:03 43660006 QuarFlat 10 A 13.0 ---------- a-b-c = - - =20 >>>> 59928 7 01:05 43660007 Arc---- 12 B 13.5 ---------- a-b-c = - 13=20 >>>> 59928 8 01:12 43660008 Arc---- 40 B 13.5 ---------- a-b-c = - 6=20 >>>> 59928 9 01:15 43660009 Arc---- 12 A 13.0 ---------- a-b-c = - 12=20 >>>> 59928 10 01:22 43660010 Arc---- 40 A 13.0 ---------- a-b-c = - 8=20 >>>> 59928 11 01:33 43660011 Dark 60 A 13.0 ---------- a-b-c = - - =20 >>>> 59928 12 01:38 43660012 Arc---- 30 A 13.0 ---------- a-b-c = - 45 FPI >>>> 59928 13 01:43 43660013 DomeFlat 5 A 13.0 --------- a-b-c = - 130=20 >>>> 59928 14 01:44 43660014 DomeFlat 5 A 13.0 --------- a-b-c = - 140=20 >>>> 59928 15 01:45 43660015 DomeFlat 5 A 13.0 --------- a-b-c = - 131=20 >>>> 59928 16 02:22 43660016 Object 43 A 13.0 ---------- a-b-c = - 1,1=20 >>>> 59928 17 02:29 43660017 Object 43 B 13.5 ---------- a-b-c = - 1,1=20 >>>> 59928 18 02:38 43660018 Object 43 B 13.5 ---------- a-b-c = - 1,1=20 >>>> 59928 19 02:46 43660019 Object 43 A 13.0 130834-7899 a-b-c = 0.09 1,1=20 >>>> 59928 20 02:59 43660020 Object 43 A 13.0 130834-7899 a-b-c = 0.40 1,1=20 >>>> 59928 21 03:07 43660021 Object 43 B 13.5 130834-7899 a-b-c = 0.66 1,1=20 >>>> 59928 22 03:16 43660022 Object 43 B 13.5 130834-7899 a-b-c = 0.64 1,1=20 >>>> 59928 23 03:23 43660023 Object 43 A 13.0 130834-7899 a-b-c = 0.40 1,1=20 >>>> 59928 24 03:32 43660024 Object 43 A 13.0 130834-7899 a-b-c = 0.32 1,1=20 >>>> 59928 25 03:40 43660025 Object 43 B 13.5 130834-7899 a-b-c = 0.67 1,1=20 >>>> 59928 26 03:52 43660026 Object 43 B 13.5 130835-7900 a-b-c = 0.65 1,1=20 >>>> 59928 27 04:00 43660027 Object 43 A 13.0 130835-7900 a-b-c = 0.41 1,1=20 >>>> 59928 28 04:18 43660028 Object 43 A 13.0 130835-7900 a-b-c = 0.31 1,1=20 >>>> 59928 29 04:26 43660029 Object 43 B 13.5 130835-7900 a-b-c = 0.66 1,1=20 >>>> 59928 30 04:35 43660030 Object 43 B 13.5 130835-7900 a-b-c = 0.66 1,1=20 >>>> 59928 31 04:43 43660031 Object 43 A 13.0 130835-7900 a-b-c = 0.30 1,1=20 >>>> 59928 32 06:46 43660032 DomeFlat 5 A 13.0 126260-7903 a-b-c = - 128=20 >>>> 59928 33 06:47 43660033 DomeFlat 5 A 13.0 126260-7903 a-b-c = - 133=20 >>>> 59928 34 06:48 43660034 DomeFlat 5 A 13.0 126260-7903 a-b-c = - 135=20 >>>> cannot read file : = /data/apogee/utr_cdr/59928/apRaw-43660035.fits >>>> = --------------------------------------------------------------------------= ----=20 >>>>=20 >>>>=20 >>>>=20 >>>> ---- BOSS Data Summary ---- >>>>=20 >>>> Reading FITS from /data/spectro//59928 >>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - = - - - - -=20 >>>> n Time Filename design-field cfg flav shu Exp colA = colB colC Hrt >>>> - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - = - - - - -=20 >>>> 1 00:35 b1-00349865 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 2 00:35 r1-00349865 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 3 00:41 r1-00349866 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 4 00:41 b1-00349866 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 5 00:47 r1-00349867 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 6 00:47 b1-00349867 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 7 00:51 b1-00349868 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 8 00:51 r1-00349868 ----------- ---- bia Clo 0.0 -7088 -7138 = -8377=20 >>>> 9 00:52 r1-00349869 ----------- ---- dar Clo 20.0 -7088 -7138 = -8377=20 >>>> 10 00:52 b1-00349869 ----------- ---- dar Clo 20.0 -7088 -7138 = -8377=20 >>>> 11 00:54 b1-00349870 127572-104222 7896 arc Lef 4.1 -7088 = -7138 -8377 HART >>>> 12 00:54 r1-00349870 127572-104222 7896 arc Lef 4.1 -7088 = -7138 -8377 HART >>>> 13 00:55 r1-00349871 127572-104222 7896 arc Rig 4.1 -7088 = -7138 -8377 HART >>>> 14 00:55 b1-00349871 127572-104222 7896 arc Rig 4.1 -7088 = -7138 -8377 HART >>>> 15 00:56 r1-00349872 127572-104222 7896 arc Lef 4.1 -7088 = -7138 -8377 HART >>>> 16 00:56 b1-00349872 127572-104222 7896 arc Lef 4.1 -7088 = -7138 -8377 HART >>>> 17 00:57 r1-00349873 127572-104222 7896 arc Rig 4.1 -7088 = -7138 -8377 HART >>>> 18 00:57 b1-00349873 127572-104222 7896 arc Rig 4.1 -7088 = -7138 -8377 HART >>>> 19 00:58 b1-00349874 127572-104222 7896 arc Out 4.1 -4790 = -4840 -6079=20 >>>> 20 00:58 r1-00349874 127572-104222 7896 arc Out 4.1 -4790 = -4840 -6079=20 >>>> 21 01:00 r1-00349875 127572-104222 7896 fla Out 25.1 -4790 = -4840 -6079=20 >>>> 22 01:00 b1-00349875 127572-104222 7896 fla Out 25.1 -4790 = -4840 -6079=20 >>>> 23 01:05 b1-00349876 -999--999 7897 sci Out 300.1 -4790 -4840 = -6079=20 >>>> 24 01:05 r1-00349876 -999--999 7897 sci Out 300.1 -4790 -4840 = -6079=20 >>>> 25 01:13 r1-00349877 -999--999 7897 fla Out 25.1 -4790 -4840 = -6079=20 >>>> 26 01:13 b1-00349877 -999--999 7897 fla Out 25.1 -4790 -4840 = -6079=20 >>>> 27 01:15 r1-00349878 -999--999 7897 fla Out 25.1 -4790 -4840 = -6079=20 >>>> 28 01:15 b1-00349878 -999--999 7897 fla Out 25.1 -4790 -4840 = -6079=20 >>>> 29 01:17 r1-00349879 -999--999 7897 fla Out 25.1 -4790 -4840 = -6079=20 >>>> 30 01:17 b1-00349879 -999--999 7897 fla Out 25.1 -4790 -4840 = -6079=20 >>>> 31 01:49 b1-00349880 130833-104666 7898 fla Out 25.1 -4790 = -4840 -6079=20 >>>> 32 01:49 r1-00349880 130833-104666 7898 fla Out 25.1 -4790 = -4840 -6079=20 >>>> 33 01:51 r1-00349881 130833-104666 7898 arc Lef 4.1 -4790 = -4840 -6079 HART >>>> 34 01:51 b1-00349881 130833-104666 7898 arc Lef 4.1 -4790 = -4840 -6079 HART >>>> 35 01:52 b1-00349882 130833-104666 7898 arc Rig 4.1 -4790 = -4840 -6079 HART >>>> 36 01:52 r1-00349882 130833-104666 7898 arc Rig 4.1 -4790 = -4840 -6079 HART >>>> 37 01:53 r1-00349883 130833-104666 7898 arc Out 4.1 -4754 = -4803 -6048=20 >>>> 38 01:53 b1-00349883 130833-104666 7898 arc Out 4.1 -4754 = -4803 -6048=20 >>>> 39 02:14 r1-00349884 130833-104666 7898 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 40 02:14 b1-00349884 130833-104666 7898 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 41 02:31 r1-00349885 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 42 02:31 b1-00349885 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 43 02:52 b1-00349886 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 44 02:52 r1-00349886 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 45 03:08 b1-00349887 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 46 03:08 r1-00349887 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 47 03:25 r1-00349888 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 48 03:25 b1-00349888 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 49 04:11 b1-00349891 130835-104666 7900 sci Out 900.1 -4754 = -4803 -6048=20 >>>> 50 04:11 r1-00349891 130835-104666 7900 sci Out 900.1 -4754 = -4803 -6048=20 >>>>=20 >>>>=20 >>>> ---- BOSS Spectrographs Collimation ---- >>>>=20 >>>> Time TSP1 R1 B1 Move1 B1Resid >>>> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >>>> 00:57:50 -0.9 1649 -10.3 2298 -3.1 >>>> 01:52:40 -1.8 1319 0.8 36 0.9 >>>>=20 >>>>=20 >>>> ---- Time Tracking ---- >>>>=20 >>>> *=3D*=3D*=3D Observatory: APO >>>> *=3D*=3D*=3D MJD: 59928 >>>> Fill night time in Hours below >>>> *=3D*=3D*=3D Total bright time : 7.4 >>>> *=3D*=3D*=3D Bright time, queued observations : 0 >>>> *=3D*=3D*=3D Bright time, commissioning/engineering: 1.3 >>>> *=3D*=3D*=3D Bright time lost to weather : 6.1 >>>> *=3D*=3D*=3D Bright time lost to technical issues : 0 >>>>=20 >>>> *=3D*=3D*=3D Total dark time : 4.7 >>>> *=3D*=3D*=3D Dark time, queued observations : 2.7 >>>> *=3D*=3D*=3D Dark time, commissioning/engineering : 2.0 >>>> *=3D*=3D*=3D Dark time lost to weather : 0 >>>> *=3D*=3D*=3D Dark time lost to technical issues : 0 >>>>=20 >>>>=20 >>>> ____________________________________________________________ >>>> Logs of Nightly APO Activities > >>>> = https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/priv= ate/sloan-25mlog/index.html__;!!K-Hz7m0Vt54!gjc17ylValeuSwPji4hP8cx9zth7p4= K8JYisv1elBG3jWAcbiWmwWLx_8I1rW-7GZ-ps84fgHthj6R4M3TnZZg$ = =20 >>> ____________________________________________________________ >>> Logs of Nightly APO Activities > >>> = https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/priv= ate/sloan-25mlog/index.html__;!!K-Hz7m0Vt54!iuq18v8DKzwpLrZszVCO_rjRVE6Pnb= 8bZBNwFchFyFL0xnjHC5Ok22qdMlV-xiGR4sp_DyU43jXoixo$ = =20 >>=20 --Apple-Mail=_F20BE176-3F3D-4F6D-B375-4CD4ADF12413 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 I = think the issue is a little more pathological. I recall that you = can connect (eg the port is up), but the other end is always silent, so = pretty much everything just times out.On Dec = 15, 2022, at 1:30 PM, Jos=C3=A9 S=C3=A1nchez-Gallego = wrote: Forgot to mention that we did that. Or I think we did. The = instructions are herehttps://wiki.sdss.org/display/APO/Procedure+to+Logout+Hung+Comm= unication+Portsand there's an script in = ~sdss5/bin that John wrote to automate the process.My = recollection was that when a port is hung it won't let you connect to = it, not that it will sometimes disconnect, but in any case clearing = those ports may be another thing to try.Jos=C3=A9 On Dec 15, 2022 at 10:19:24, = Conor Sayres wrote: =46rom ancient history, I recall that the xyplex terminal = server needs to have =E2=80=9Cports cleared=E2=80=9D, otherwise we see = occasional hanging connections. Do the observers know (remember?) = how to do this? On Dec = 15, 2022, at 1:00 PM, Jos=C3=A9 S=C3=A1nchez-Gallego = wrote: Hi all,With respect to the BOSS (and sometimes axis) = issues that kept us from observing yesterday ... well they are very = annoyingly not showing up today. I took several 900s images and they all = completed well. Also seems that there weren't any additional = disconnections last night after we closed.I don't = believe in coincidences, so I would suspect that something that we did = yesterday caused this, but the code itself hasn't changed (only the = format of /data and where it's mounted, but that should not matter to = sdss5-boss-icc or sdss5-tcc), and when it works it works fine as = usual.The problem seems to be that sometimes the ICC or = TCC lose connection to the controllers on the dome side. I think BOSS is = more obvious because it lacks automatic reconnection capabilities, while = the TCC does some reconnecting by itself.All the = devices that seem to be failing are connected through the xyplex = terminal server, and we saw some weird behaviour with it yesterday, in = which at some point it stopped letting us into the management console, = but this morning it was fine again. We saw a weird case with the FPS, = but I'm willing to attribute that to something else, but it's worth = looking into other hardware that enable communication with the = telescope.Has there been any change on the telescope side = that could explain these very intermittent disconnects? I could believe = that something related with UPS or power could be at play = here.If not, I suggest we try observing tonight ... If = it starts failing again my first advice would be to reboot the xyplex (I = think yesterday it was an edge server that was restarted = yesterday).I think Shane is monitoring that device now, = maybe we'll learn something if it happens again.Thanks,Jos=C3=A9 On Dec 15, 2022 at 04:04:36, = asimmons@apo.nmsu.edu wrote: =20 = &n= bsp; APACHE POINT = OBSERVATORY SDSS 2.5M = &n= bsp; &nbs= p; OBSERVING LOG = &n= bsp; &nbs= p;Wednesday December 14, 2022 = &n= bsp; &nbs= p; (SJD 59928)---- Observing Team ----Audrey OravetzJos=C3=A9 = S=C3=A1nchez-Gallego (online support)John Donor (online = support)Shane Thomas (on site support)---- Observing Plan ----Wed 14 = Dec 2022 59928 0.61 00:56 01:11 12:45 23:42 11:18 --- dark until 05:52 ( = 4.7 h), then bright ( 7.1 h, w/ 0.3 h for 12 deg design)---- Observing Summary ----The first half of the night could be treated as a flexible = engineering due to the problems we encountered from the /data disk = transfer.The first 1.3 hrs were used to = get all systems back running and afternoon tasks completed. After we opened, we operated smoothly for the = most part until just before 04:00Z when we began witnessing multiple = BOSS camera controller disconnections. Mostly it was sp1cam, but once = was sp1mech. To get them reconnected, we logged out ports and restarted = the bossICC. We also witnessed the M2 mirror controller hanging, as well = as some suspicious behavior with the FPS during a trajectory unwind, = again pointing to a potential network or server error. The terminal = server was then rebooted with a very small reset button, but did not = resolve the issues (unless a physical unplug/replug of the power cord is = needed instead). More investigation is needed tomorrow. The humidity spiked and remained high the rest of the night = past 07:00Z.Observed:---------Time = field-dsgn-conf Design type = BOSS/APOGEE exp complete? Comment=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D01:36:15 104666-130833-7898 dark_monit. = 2x900 + 2XAB. Not done. * one exposure taken with this = design was mistakenly loaded into the next design, but the robots never = moved. 02:31:00 104666-130834-7899 = dark_monit. 2x900 + 2XAB. DONE03:40:35 = 104666-130835-7900 dark_monit. 1x900 + 3XAB. Not done---- Observing Log ----Afternoon:=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3DChecked actors, mirror numbers. = Attempted to begin taking afternoon cals, and hence restarting a while = bunch of systems, but mistakenly interrupted the on-going work with the = /data transfer. Verified with Jose that everything is ok and they will = continue to work. Note that we will be delayed getting on sky on time = tonight. 23:40Z - Started ventilating. 00:02Z - Opened the telescope and filled the = cameras now in an attempt to save time later.Night:=3D=3D=3D=3D=3D=3DTook BOSS cals and APOGEE cals. Turned on the FPS. The DAQ was not responding properly after the = /data work. No exposures were being written to disk. Everything would = hang after integration. Jose came online. I reset the DAQ on the = specrograph itself. Jose did his magic. We could take images again. 00:56:34 boss exposure = 4.0, file=3D349872 00:57:17 boss = exposure 4.0, file=3D349873 00:58:06 = sp1.motor.move=3D 2298, 2298, 229800:58:06 = Hartmann collimate output sp1: offset: = r=3D 1319 (Out of focus); b=3D -10.3 (Out of focus) = sp1: pred. move: spAverageMove=3D 2298 = sp1: pred. resid: r=3D -979, b=3D -3.1, txt=3D OK, spTemp =3D = -0.500:58:39 boss exposure 4.0, = file=3D349874 01:00:11 boss exposure = 25.0, file=3D349875 01:02:08 = apogee.expose=3DapRaw-43660006, 10, A, QuartzFlat********************************************************01:03:35 Field-Design-Configuration: -999--999-7897********************************************************01:04:04 apogee.expose=3DapRaw-43660007, 12, B, = ArcLamp01:05:06 boss exposure 300.0, = file=3D349876 01:06:16 = apogee.expose=3DapRaw-43660008, 40, B, ArcLamp01:13:33 apogee.expose=3DapRaw-43660009, 12, A, = ArcLamp01:13:52 boss exposure 25.0, = file=3D349877 01:15:46 boss exposure = 25.0, file=3D349878 01:15:46 = apogee.expose=3DapRaw-43660010, 40, A, ArcLamp01:17:43 boss exposure 25.0, file=3D349879 = 01:23:12 apogee.expose=3DapRaw-43660011, 60, A, = Dark01:34:04 apogee.expose=3DapRaw-43660012, 30, A, = ArcLamp01:03:39 APO.aoravetz 46 jaeger i = configuration_snapshot=3D/data/logs/jaeger/configuration_snapshots/59928/c= onfiguration_snapshot_59928_7897.pdf01:03:39 APO.aoravetz = 46 jaeger : text=3D"Configuration 7897 loaded."01:03:39 = CmdDone: APO.aoravetz 46 jaeger configuration load --from-positions01:04:44 APO.aoravetz 45 fliswarm i = filename=3Dfvc,fvc1n,/data/fcam/59928/fimg-fvc1n-0001.fits01:04:44 APO.aoravetz 45 fliswarm i = exposure_state=3Dfvc,fvc1n,idle,NA,0.0,0,0,001:04:44 = APO.aoravetz 45 fliswarm i = filename_bundle=3D/data/fcam/59928/fimg-fvc1n-0001.fits01:04:44 APO.aoravetz 45 fliswarm : 01:04:44 = CmdDone: APO.aoravetz 45 fliswarm talk -n fvc expose 50(observers39) [observer@sdss5-hub: ~]$ list_apAPOGEE data list, mjd=3D59928, allraw_data: /data/apogee/utr_cdr/59928/apRaw-*.fitsarchive: = /data/apogee/archive/59928/apR-[a,b,c]-*.apzquickred:= /data/apogee/quickred/59928/apq-*.fits---------------------------------------------------------------= --------------- mjd i UT = File/Exp Imtype Nread A/B Dth Design-Cfg = Arch Offset %Flux/FPI Comm = --------------------------------------------------------------------------= ----59928 1 no = file found: /data/apogee/utr_cdr/59928/apRaw-43660001.fits59928 2 no file = found: /data/apogee/utr_cdr/59928/apRaw-43660002.fits59928 3 no file = found: /data/apogee/utr_cdr/59928/apRaw-43660003.fits59928 4 23:42 43660004 = Dark 60 A 13.0 = ---------- a-b-c - -59928 5 01:01 43660005 = Dark 60 A 13.0 = ---------- a-b-c - -59928 6 01:03 43660006 QuarFlat 10 = A 13.0 ---------- a-b-c - = -59928 7 01:05 43660007 = Arc---- 12 B 13.5 ---------- a-b-c = - 1359928 8 = 01:12 43660008 Arc---- 40 B 13.5 = ---------- a-b-c - 659928 9 01:15 43660009 Arc---- = 12 A 13.0 ---------- a-b-c = - 1259928 10 01:22 = 43660010 Arc---- 40 A 13.0 = ---------- a-b-c - 859928 11 01:33 43660011 Dark = 60 A 13.0 ---------- a-b-c = - -59928 12 01:38 = 43660012 Arc---- 30 A 13.0 = ---------- a-b-c - 45 = FPI59928 13 01:43 43660013 = DomeFlat 5 A 13.0 --------- a-b-c = - 13059928 14 01:44 = 43660014 DomeFlat 5 A 13.0 --------- = a-b-c - 14059928 15 = 01:45 43660015 DomeFlat 5 A 13.0 = --------- a-b-c - 131The FPI afternoon cal was a bit faint at 45%. = Not sure what the value was yesterday. The dome flats seem ok. BSONO was not preloading the designs from = KRONOS properly. Jose is aware and is working on creating another BOSON = version.********************************************************01:36:15 Field-Design-Configuration: = 104666-130833-7898********************************************************01:42:43 apogee.expose=3DapRaw-43660013, = 5, A, DomeFlat01:43:58 = apogee.expose=3DapRaw-43660014, 5, A, DomeFlat01:45:12 apogee.expose=3DapRaw-43660015, 5, A, = DomeFlat01:46:12 GOTO-FIELD01:48:57 FVC LOOP AND BOSS CALS = fvc_rms=3D = 7.95 = fvc_deltarms=3D= 29.53 = fvc_filename=3D= /data/fcam/59928/fimg-fvc1n-0003.fits = fvc_perc_90=3D= 11.30 = fvc_percent_re= ached=3D 50.3001:49:26 boss exposure = 25.0, file=3D349880 01:51:21 boss = exposure 4.0, file=3D349881 01:52:02 = boss exposure 4.0, file=3D349882 01:52:44 sp1.motor.move=3D 36, 37, 3101:52:44 Hartmann collimate output = sp1: offset: r=3D 329 (In Focus); b=3D 0.8 (In Focus) = sp1: pred. move: spAverageMove=3D 36 sp1: pred. resid: r=3D 293, b=3D 0.9, txt=3D OK, = spTemp =3D -1.401:53:04 boss exposure = 4.0, file=3D349883 02:04:17 .alerts 0 alerts e = alert=3Dboss.camCheck.stale,critical,"at 2022-12-15 01:56:01; no change = for 615 sec, with None",enabled,noack,0Reloaded the controllers. Not sure why this alert spawned, = but it was real. 02:14:47 = apogee.expose=3DapRaw-43660016, 43, A, Object02:14:54= boss exposure 900.0, file=3D349884 02:22:29 = apogee.expose=3DapRaw-43660017, 43, B, Object********************************************************02:30:07 Field-Design-Configuration: = 104666-130834-7899************************************************************* The pre-loading test for Jose was = accidentally implemented and the S?N was not checked beforehand. **No = robots were moved into a new configuration**. The exposures below are = associated with the previous configuration and design. 02:31:08 Same-field overhead: 0:00:0802:31:22 apogee.expose=3DapRaw-43660018, 43, B, = Object02:31:29 boss exposure 900.0, = file=3D349885 02:39:08 = apogee.expose=3DapRaw-43660019, 43, A, Object********************************************************02:30:07 Field-Design-Configuration: = 104666-130834-7899********************************************************The following images ARE associated with the = new design and configuration. 02:31:08 = Same-field overhead: 0:00:0802:31:22 = apogee.expose=3DapRaw-43660018, 43, B, Object02:31:29= boss exposure 900.0, file=3D349885 02:39:08 = apogee.expose=3DapRaw-43660019, 43, A, Object02:47:34= GOTO-FIELD02:50:19 FVC LOOP AND BOSS CALS = fvc_rms=3D = 8.33 = fvc_deltarms=3D= 30.68 = fvc_filename=3D= /data/fcam/59928/fimg-fvc1n-0005.fits = fvc_perc_90=3D= 10.00 = fvc_percent_re= ached=3D 51.7002:51:59 START SCIENCE EXPOSURES02:52:21 apogee.expose=3DapRaw-43660020, 43, A, = Object02:52:29 boss exposure 900.0, = file=3D349886 03:00:03 = apogee.expose=3DapRaw-43660021, 43, B, Object03:08:50= apogee.expose=3DapRaw-43660022, 43, B, Object03:08:58 boss exposure 900.0, file=3D349887 03:16:37 apogee.expose=3DapRaw-43660023, 43, A, = Object03:25:20 apogee.expose=3DapRaw-43660024, 43, = A, Object03:25:28 boss exposure 900.0, = file=3D349888 03:33:01 = apogee.expose=3DapRaw-43660025, 43, B, Object********************************************************03:40:35 Field-Design-Configuration: = 104666-130835-7900********************************************************03:41:35 GOTO-FIELD03:43:56= FVC LOOP AND BOSS CALS = fvc_rms=3D = 9.43 = fvc_deltarms=3D= 29.13 = fvc_filename=3D= /data/fcam/59928/fimg-fvc1n-0007.fits = fvc_perc_90=3D= 10.20 = fvc_percent_re= ached=3D 51.8003:44:48 Same-field overhead: = 0:03:12Only APOGEE started because BOSS = was having cam and mech connection errors.03:44:48 START SCIENCE EXPOSURES03:45:10 = apogee.expose=3DapRaw-43660026, 43, B, Object03:52:57= apogee.expose=3DapRaw-43660027, 43, A, ObjectThere are continuous connection errors in the logs, but yet = the system seemed ok for awhile.04:04:54 = .boss 0 boss w text=3D"Controller sp1cam disconnected."04:04:54 .boss 0 boss w text=3D"sp1cam removed from available = hardware."Logged out ports and reloaded = the controllersCritical = boss.specMechProtocol at 2022-12-15 04:08:55 UT found 0Restarted the bossICC. Things = are working again. 04:08:55 = sp1.motor.move=3D None, None, None04:09:33 = sp1.motor.move=3D None, None, None04:11:36 = apogee.expose=3DapRaw-43660028, 43, A, Object04:11:47= boss exposure 900.0, file=3D349889 04:19:18 apogee.expose=3DapRaw-43660029, 43, B, = Object04:28:23 apogee.expose=3DapRaw-43660030, 43, = B, Object04:28:37 boss exposure 900.0, = file=3D349892 04:36:09 = apogee.expose=3DapRaw-43660031, 43, A, Object04:48:07 CmdStarted: APO.aoravetz 162 boss exposure = recover04:48:07 APO.aoravetz 162 boss f text=3D"No = exposure to act on."04:48:07 CmdDone: APO.aoravetz 162 = boss exposure recoverThe exposure was not = able to be recovered.HORRIBLE CRAZY NOISES = WHEN ALERTS WENT HAYWIRE04:52:31 = APO.aoravetz 1338 stui F Timeout; Actor=3D'alerts'; Cmd=3D"status"Critical boss.heartbeat at 2022-12-15 = 04:54:09; last seen 135 sec agoSerious mcp.heartbeat at = 2022-12-15 04:53:45; last seen 160 sec agoWarn = boss.camCheck.stale at 2022-12-15 04:54:10; no change for 135 sec, with = NoneAxes halted. Stop button in. By = 05:05:23, everything was back to normal. Connections were re-established = all on their own. ********************************************************05:08:15 Field-Design-Configuration: = 103743-126372-7901********************************************************05:08:56 GOTO-FIELD05:09:40 = GOTO-FIELD05:11:27 GOTO-FIELD05:13:41 FVC LOOP AND BOSS CALS = fvc_rms=3D = 11.84 = fvc_deltarms=3D= 23.85 = fvc_filename=3D= /data/fcam/59928/fimg-fvc1n-0009.fits = fvc_perc_90=3D= 11.10 = fvc_percent_re= ached=3D 45.00Discovered that the = secondary mirror was still moving, 1600 sesconds. tcc talk sec stop was = issued, but it retunred with 'TIMEOUT'. I was in the process of looking = back through the notes to enter into the mirror controllers when it = magically reconnected and went back to idle. The gotofield hung again at Flushing for the ARC. At this point, Jose recommended that we reboot = the terminal server completely. It was = rebooted. The TCC disconnected and reconnected. ********************************************************05:50:53 Field-Design-Configuration: = 103721-126260-7902********************************************************05:50:58 GOTO-FIELD05:51:09= hal.hal 363 jaeger f error=3D"Trajectory failed with error: Something = went wrong sending the trajectory: Failed sending = SEND_NEW_TRAJECTORY"05:51:09 CmdDone: hal.hal 363 jaeger = configuration reverse05:51:09 APO.HushedOrangutan 30 hal = e error=3D"Command jaeger configuration reverse failed."jaeger reload re-established the connection to the CAN = devices.Shane was contacted for his = opinion. After an hour or so, Jose and = Shane were out of ideas and could not understand why the connections = were poor. 06:45:33 = apogee.expose=3DapRaw-43660032, 5, A, DomeFlat06:46:47 apogee.expose=3DapRaw-43660033, 5, A, = DomeFlat06:48:02 apogee.expose=3DapRaw-43660034, 5, = A, DomeFlat06:53:41 encl25m : = closed; Due to high humidity. Turned off the FPS.It remained humid the rest of the night. ---- Mirror Numbers ----PRIMARY:--------Scale: 1.000000MIGS = TONIGHT NOMINALAxial A: 0.0080 0.0080Axial B: 0.0030 0.0040Axial C: -0.0310 = -0.0290Trans D: 0.0320 0.0320Lateral E: = 0.3315 0.3320Lateral F: 0.5355 0.5350GALILSCommanded: 12500 = -1600 5300 47050 -8800 -8400Actual: = 12500 -1600 5300 47050 -8800 -8400SETMIR VALUESPrimDesOrient: 0.0 = -59.5 -9.15 -512.0 1906.3PrimOrient: = 1.07 -59.51 -9.21 -512.13 1906.31SECONDARY:----------Focus: 0.0000Air = temp: 92.83Altitude: = 30.0MIGS TONIGHT = NOMINALAxial A: -0.0320 -0.0330Axial B: = -0.0280 -0.0290Axial C: -0.0450 -0.0450Trans D: -0.2670 -0.3170GALILSCommanded: 4905050 4881950 4974150 0 = -3400Actual: 4859268 4859148 4859262 0 = -3400SETMIR VALUESSecDesOrient: 3857.0 0.0 0.0 0.0 42.22SecOrient:= 3857.01 -0.02 -0.03 -0.02 41.65---- Telescope Status ----23:45Z Started = ventilation00:05Z encl25m : = open06:53Z encl25m : closedNo interlocks bypassedStatus at: 11:14ZTelescope stowed at: = 121.0, 30.0, 0.0 mountInstrument mounted: FPS = Counterweights at: = 685/710 LN2= autofill systems: Connected and turned on180L = LN2 dewar scales: SP1 171 lbs, = 19.7 psiFPS Status=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DCP10 Loop =3D = 7.19 L/minGFA Loop =3D 1.71 L/minSupply pressure =3D 42.2 psiReturn = pressure =3D 29.2 psiChiller = Status=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DSet temperarature =3D 0.9 CFlow =3D = 5.3 Gpm---- Software Used ----Get = Versions, 12/14/22, 23:55:29 -- STUI software: STUI : = 1.7.18b1alerts = : 2.3.0boss = : 6.0.2-alpha.0-boss.specMec : 3.0.4apo = : 3.0.1fliswarm : 0.3.1jaeger : 1.2.0b2kaiju : = 1.2.4-alpha.0coordio = : 1.5.0-beta.2fps_calibrati : 2022.09.11+c5432facherno = : 0.4.0b3astrometrynet : 0.91hal = : 0.3.1a0hartmann : 2.0.2hub = : 5.0.1a0mcp = : 2.1-mcp.plc : 35-mcp.azFiduci : v1_148-mcp.altFiduc : = v1_148-mcp.rotFiduc : v1_148tcc = : 1.4.0apogee : not = availbleapogeecal : not = availbleapogeeql : = v1_7_0devSTUIAdditions : FAILED -- = done --(observers39) [observer@sdss5-hub: ~]$ = spVersion--------------Other Versions--------------sdss-obstools : 3.9.2TPM = &n= bsp; : 1.2-31-gd7ebMCP = &n= bsp; : 2.1PLC = &n= bsp; : V35Fiducials = : v1_148idlspec2d = : v6_0_9Kronos = &n= bsp;: 1.5.0roboscheduler = : 1.2.2/dev/sdb1 = &n= bsp; 2.0T 235G 1.8T 12% /home/dev/sdd1 = &n= bsp; 22T 230G 21T 2% = /data---- APOGEE Data = Summary ----APOGEE data list, = mjd=3D59928, allraw_data: = /data/apogee/utr_cdr/59928/apRaw-*.fitsarchive: = /data/apogee/archive/59928/apR-[a,b,c]-*.apzquickred:= /data/apogee/quickred/59928/apq-*.fits---------------------------------------------------------------= --------------- mjd i UT = File/Exp Imtype Nread A/B Dth Design-Cfg = Arch Offset %Flux/FPI Comm = --------------------------------------------------------------------------= ----59928 1 no = file found: /data/apogee/utr_cdr/59928/apRaw-43660001.fits 59928 2 no file = found: /data/apogee/utr_cdr/59928/apRaw-43660002.fits 59928 3 no file = found: /data/apogee/utr_cdr/59928/apRaw-43660003.fits 59928 4 23:42 43660004 = Dark 60 A 13.0 = ---------- a-b-c - - 59928 5 01:01 43660005 = Dark 60 A 13.0 = ---------- a-b-c - - 59928 6 01:03 43660006 QuarFlat 10 = A 13.0 ---------- a-b-c - = - 59928 7 01:05 = 43660007 Arc---- 12 B 13.5 = ---------- a-b-c - 13 59928 8 01:12 43660008 Arc---- = 40 B 13.5 ---------- a-b-c = - 6 59928 = 9 01:15 43660009 Arc---- 12 A = 13.0 ---------- a-b-c - = 12 59928 10 01:22 43660010 = Arc---- 40 A 13.0 ---------- a-b-c = - 8 59928 11 = 01:33 43660011 Dark 60 A = 13.0 ---------- a-b-c - = - 59928 12 01:38 43660012 = Arc---- 30 A 13.0 ---------- a-b-c = - 45 FPI59928 = 13 01:43 43660013 DomeFlat 5 A 13.0 = --------- a-b-c - 130 59928 14 01:44 43660014 DomeFlat 5 = A 13.0 --------- a-b-c - = 140 59928 15 01:45 43660015 = DomeFlat 5 A 13.0 --------- a-b-c = - 131 59928 16 02:22 = 43660016 Object 43 A 13.0 = ---------- a-b-c - 1,1 59928 17 02:29 43660017 Object = 43 B 13.5 ---------- a-b-c = - 1,1 59928 18 02:38 = 43660018 Object 43 B 13.5 = ---------- a-b-c - 1,1 59928 19 02:46 43660019 Object = 43 A 13.0 130834-7899 a-b-c 0.09 1,1 = 59928 20 02:59 43660020 Object = 43 A 13.0 130834-7899 a-b-c 0.40 1,1 = 59928 21 03:07 43660021 Object = 43 B 13.5 130834-7899 a-b-c 0.66 1,1 = 59928 22 03:16 43660022 Object = 43 B 13.5 130834-7899 a-b-c 0.64 1,1 = 59928 23 03:23 43660023 Object = 43 A 13.0 130834-7899 a-b-c 0.40 1,1 = 59928 24 03:32 43660024 Object = 43 A 13.0 130834-7899 a-b-c 0.32 1,1 = 59928 25 03:40 43660025 Object = 43 B 13.5 130834-7899 a-b-c 0.67 1,1 = 59928 26 03:52 43660026 Object = 43 B 13.5 130835-7900 a-b-c 0.65 1,1 = 59928 27 04:00 43660027 Object = 43 A 13.0 130835-7900 a-b-c 0.41 1,1 = 59928 28 04:18 43660028 Object = 43 A 13.0 130835-7900 a-b-c 0.31 1,1 = 59928 29 04:26 43660029 Object = 43 B 13.5 130835-7900 a-b-c 0.66 1,1 = 59928 30 04:35 43660030 Object = 43 B 13.5 130835-7900 a-b-c 0.66 1,1 = 59928 31 04:43 43660031 Object = 43 A 13.0 130835-7900 a-b-c 0.30 1,1 = 59928 32 06:46 43660032 DomeFlat = 5 A 13.0 126260-7903 a-b-c = - 128 59928 33 06:47 = 43660033 DomeFlat 5 A 13.0 = 126260-7903 a-b-c - 133 59928 34 06:48 43660034 DomeFlat 5 = A 13.0 126260-7903 a-b-c - = 135 cannot read file : = /data/apogee/utr_cdr/59928/apRaw-43660035.fits---------------------------------------------------------------= --------------- ---- BOSS Data Summary ----Reading FITS from /data/spectro//59928- - - - = - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - = n Time Filename = design-field cfg flav shu Exp = colA colB colC Hrt- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - = - - - - - - - - 1 00:35 b1-00349865 ----------- = ---- bia Clo 0.0 -7088 -7138 -8377 = 2 00:35 r1-00349865 ----------- ---- bia Clo = 0.0 -7088 -7138 -8377 = 3 00:41 r1-00349866 ----------- ---- bia Clo 0.0 = -7088 -7138 -8377 4 00:41 = b1-00349866 ----------- ---- bia Clo 0.0 -7088 = -7138 -8377 5 00:47 r1-00349867 = ----------- ---- bia Clo 0.0 -7088 -7138 = -8377 6 00:47 b1-00349867 ----------- ---- = bia Clo 0.0 -7088 -7138 -8377 7 00:51 b1-00349868 ----------- ---- bia Clo = 0.0 -7088 -7138 -8377 = 8 00:51 r1-00349868 ----------- ---- bia Clo 0.0 = -7088 -7138 -8377 9 00:52 = r1-00349869 ----------- ---- dar Clo 20.0 -7088 = -7138 -8377 10 00:52 b1-00349869 ----------- = ---- dar Clo 20.0 -7088 -7138 -8377 11 00:54 b1-00349870 127572-104222 7896 arc Lef = 4.1 -7088 -7138 -8377 HART 12 00:54 r1-00349870 127572-104222 7896 arc Lef = 4.1 -7088 -7138 -8377 HART 13 00:55 r1-00349871 127572-104222 7896 arc Rig = 4.1 -7088 -7138 -8377 HART 14 00:55 b1-00349871 127572-104222 7896 arc Rig = 4.1 -7088 -7138 -8377 HART 15 00:56 r1-00349872 127572-104222 7896 arc Lef = 4.1 -7088 -7138 -8377 HART 16 00:56 b1-00349872 127572-104222 7896 arc Lef = 4.1 -7088 -7138 -8377 HART 17 00:57 r1-00349873 127572-104222 7896 arc Rig = 4.1 -7088 -7138 -8377 HART 18 00:57 b1-00349873 127572-104222 7896 arc Rig = 4.1 -7088 -7138 -8377 HART 19 00:58 b1-00349874 127572-104222 7896 arc Out = 4.1 -4790 -4840 -6079 = 20 00:58 r1-00349874 127572-104222 7896 arc Out 4.1 = -4790 -4840 -6079 21 01:00 r1-00349875 = 127572-104222 7896 fla Out 25.1 -4790 -4840 = -6079 22 01:00 b1-00349875 127572-104222 7896 fla = Out 25.1 -4790 -4840 -6079 = 23 01:05 b1-00349876 -999--999 7897 sci Out 300.1 -4790 = -4840 -6079 24 01:05 r1-00349876 -999--999 = 7897 sci Out 300.1 -4790 -4840 -6079 25 01:13 r1-00349877 -999--999 7897 fla Out 25.1 = -4790 -4840 -6079 26 01:13 b1-00349877 = -999--999 7897 fla Out 25.1 -4790 -4840 = -6079 27 01:15 r1-00349878 -999--999 7897 fla Out = 25.1 -4790 -4840 -6079 28 = 01:15 b1-00349878 -999--999 7897 fla Out 25.1 -4790 = -4840 -6079 29 01:17 r1-00349879 -999--999 = 7897 fla Out 25.1 -4790 -4840 -6079 30 01:17 b1-00349879 -999--999 7897 fla Out 25.1 = -4790 -4840 -6079 31 01:49 b1-00349880 = 130833-104666 7898 fla Out 25.1 -4790 -4840 = -6079 32 01:49 r1-00349880 130833-104666 7898 fla = Out 25.1 -4790 -4840 -6079 = 33 01:51 r1-00349881 130833-104666 7898 arc Lef 4.1 = -4790 -4840 -6079 HART 34 01:51 = b1-00349881 130833-104666 7898 arc Lef 4.1 -4790 = -4840 -6079 HART 35 01:52 b1-00349882 = 130833-104666 7898 arc Rig 4.1 -4790 -4840 = -6079 HART 36 01:52 r1-00349882 130833-104666 7898 = arc Rig 4.1 -4790 -4840 -6079 = HART 37 01:53 r1-00349883 130833-104666 7898 arc Out = 4.1 -4754 -4803 -6048 = 38 01:53 b1-00349883 130833-104666 7898 arc Out 4.1 = -4754 -4803 -6048 39 02:14 r1-00349884 = 130833-104666 7898 sci Out 900.1 -4754 -4803 = -6048 40 02:14 b1-00349884 130833-104666 7898 sci = Out 900.1 -4754 -4803 -6048 41 = 02:31 r1-00349885 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048 42 02:31 b1-00349885 = 130834-104666 7899 sci Out 900.1 -4754 -4803 = -6048 43 02:52 b1-00349886 130834-104666 7899 sci = Out 900.1 -4754 -4803 -6048 44 = 02:52 r1-00349886 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048 45 03:08 b1-00349887 = 130834-104666 7899 sci Out 900.1 -4754 -4803 = -6048 46 03:08 r1-00349887 130834-104666 7899 sci = Out 900.1 -4754 -4803 -6048 47 = 03:25 r1-00349888 130834-104666 7899 sci Out 900.1 -4754 = -4803 -6048 48 03:25 b1-00349888 = 130834-104666 7899 sci Out 900.1 -4754 -4803 = -6048 49 04:11 b1-00349891 130835-104666 7900 sci = Out 900.1 -4754 -4803 -6048 50 = 04:11 r1-00349891 130835-104666 7900 sci Out 900.1 -4754 = -4803 -6048 ---- = BOSS Spectrographs Collimation ----Time = TSP1 R1 = B1 Move1 B1Resid=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D00:57:50 -0.9 1649 -10.3 = 2298 -3.101:52:40 -1.8 = 1319 0.8 36 = 0.9---- Time = Tracking ----*=3D*=3D*=3D Observatory: = APO*=3D*=3D*=3D MJD: 59928 = Fill night time in Hours below*=3D*=3D*=3D Total bright time = &n= bsp; : 7.4*=3D*=3D= *=3D Bright time, queued observations : = 0*=3D*=3D*=3D Bright time, commissioning/engineering: = 1.3*=3D*=3D*=3D Bright time lost to weather = : 6.1*=3D*=3D*=3D Bright time lost to technical issues : = 0*=3D*=3D*=3D Total dark time = &n= bsp; : 4.7*=3D*=3D*=3D Dark time, queued observations = : 2.7*=3D*=3D*=3D= Dark time, commissioning/engineering : 2.0*=3D*=3D*=3D= Dark time lost to weather = : = 0*=3D*=3D*=3D Dark time lost to technical issues = : 0 ____________________________________________________________Logs of Nightly APO Activities https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/b= rowser/private/sloan-25mlog/index.html__;!!K-Hz7m0Vt54!gjc17ylValeuSwPji4h= P8cx9zth7p4K8JYisv1elBG3jWAcbiWmwWLx_8I1rW-7GZ-ps84fgHthj6R4M3TnZZg$ = ____________________________________________________________Logs of Nightly APO Activities https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/b= rowser/private/sloan-25mlog/index.html__;!!K-Hz7m0Vt54!iuq18v8DKzwpLrZszVC= O_rjRVE6Pnb8bZBNwFchFyFL0xnjHC5Ok22qdMlV-xiGR4sp_DyU43jXoixo$ = --Apple-Mail=_F20BE176-3F3D-4F6D-B375-4CD4ADF12413--