Date: Fri, 16 Dec 2022 11:34:21 -0700 Subject: Re: [sloan-25mlog 6416] 2.5m obslog 20221215 (SJD 59929) From: "Audrey Oravetz" Ah, I think I know what happened. Everytime that the boss controllers are reloaded, the numerical sequencing gets mis-matched from the record of the exposure as it is started verus the exposure actually written to the disk. I got confused when I was keeping up with this in the log. So, in this case exposure 349938 is (sadly) the only exposure that made it through readout successfully. That is the one that was unguided. Thanks for catching this, Audrey > Hey Audrey, > > Looking at the logs from last night, I am trying to log the unguided > exposure for the pipeline and am slightly confused. You seemingly noted it > as 349933, but there does not seem to be an exposure with that number. > Additionally, there is an exposure 349937 that does not appear to be > logged. I am sure this confusion come from the craziness of the issues > last night, but can you clarify this situation so that I can note the > unguided exposure? > > Hope today runs more smoothly. > Sean > ________________________________ > From: sloan-25mlog on behalf of > asimmons@apo.nmsu.edu > Sent: Friday, December 16, 2022 6:06 AM > To: obs-report-25m@apo.nmsu.edu > Subject: [sloan-25mlog 6416] 2.5m obslog 20221215 (SJD 59929) > > > APACHE POINT OBSERVATORY SDSS 2.5M > > OBSERVING LOG > Thursday December 15, 2022 > > (SJD 59929) > > ---- Observing Team ---- > > Audrey Oravetz > > Jose Sanchez-Gallego (support) > > ---- Observing Plan ---- > > Thu 15 Dec 2022 59929 0.51 00:56 01:11 12:45 23:47 11:22 --- dark until > 06:48 ( 5.6 h), then bright ( 6.2 h, w/ 0.3 h for 12 deg design) > > > ---- Observing Summary ---- > > The first ~4 hours of the night went very smoothly, with no problems. > Around 03:45Z, we had our first sp1cam disconnection and M2 mirror > timeouts. At 04:20Z, they got too numerous that it was impossible to > observe. I was putting out fires left and right (logging out ports and > restarting boss-icc and reconnecting mirror controllers). Any BOSS > exposures that were started during this time failed to write to disk. > > Jose signed in at 05:15. We ran one exposure without the guider running > and we did not see any disconnects, so he moved cherno over to run on the > FPS machine as a test. Unfortunately it did not work as the next exposure > failed as well. > > 1.9 hrs were lost due to the technical issues. > > Closed at 06:15Z for high humidity/clouds, loss of 6.3hrs. > > Observed: > --------- > Time field-dsgn-conf Design type BOSS/APOGEE exp complete? > Comment > =========================================================================== > 00:25:33 104466-128616-7905. bright_time 1X900 + 1XAB DONE > 00:54:55 104637-129881-7906. dark_monit 2X900 + 2XAB DONE > 01:38:54 104637-129882-7907. dark_monit 2X900 + 2XAB DONE > 02:16:17 104637-129883-7908. dark_monit 2X900 + 2XAB DONE > 02:52:38 104637-129884-7909. dark_monit 5X900 + 5XAB not done; > sp1cam-controller problem > 04:56:40 104648-130158-7912 dark_monit Ongoing issues > > > ---- Software Issues ---- > > NOTES: > ====== > > Here is what we know and is repeatable whenever the event happens: > > First critical alert is spawned, and it is always > 'boss.camCheck.stale,critical... no change for XXX sec...' > (The messages that are posted at the time where there are obvious > disconnections are found in the problem section). > > boss ping: > 03:44:38 CmdStarted: APO.aoravetz 33 boss ping > 03:44:38 APO.aoravetz 33 boss i text="sp1daq: OK" > 03:44:40 APO.aoravetz 33 boss i text="Response parsed." > 03:44:40 APO.aoravetz 33 boss i text="sp1mech: OK" > 03:44:41 .alerts 0 alerts i activeAlerts=boss.camCheck.stale > *** There is no return on sp1cam *** > > Unfortunately, because this happened during the integration of a BOSS > exposure, the exposure was lost. It finished integrating, and then > proceeded to read out, hung at READING 0.0 for maybe 15 seconds and then > the following messages were sent. The exposure essentially timed out > before I had time to re-connect sp1cam and officially try to recover it. > > 03:47:25 APO.SurroundingMoose 41 hal e error="Command 'boss exposure > science itime=900.0' timed out." > > It seems like the best way to recover from these events are to restart the > bossICC with: > > sdss5@sdss5-boss-icc> systemctl --user restart bossICC > > Now all systems are back to being connected: > 03:47:19 CmdStarted: APO.aoravetz 35 boss ping > 03:47:19 APO.aoravetz 35 boss i text="sp1daq: OK" > 03:47:20 APO.aoravetz 35 boss i text="sp1cam: OK" > 03:47:20 APO.aoravetz 35 boss i text="Response parsed." > 03:47:20 APO.aoravetz 35 boss i text="sp1mech: OK" > 03:47:20 APO.aoravetz 35 boss : > 03:47:20 CmdDone: APO.aoravetz 35 boss ping > > But also the mechanicals get angry when the entire system is restarted, so > 'boss mechIack' is needed (and sometimes you need to log out of the port > first). > > The secondary mirror would get stuck occasionally as well (03:48:44 .tcc 0 > tcc w Text="Collimation Failure: Timed out"). Sometimes it would resolve > on its own, but other times it would be frozen and need to have the > controllers disconnected via the terminal process and the reconnected. > This behavior seemed to happen yesterday as well, and seems like both > issues with the BOSS camera and M2 are related. > > 06:00:13 .tcc 0 tcc W Text="Device Command A=5290688; B=5269132; > C=5354218; D=2977; E=-423; XQ #MOVE Failing: ?A=5290381; B=5268825; > C=5353911; D=2992; 005289750E=, 005268200-40, 0053533008; , > 000003000XQ , -000000400#MOVE" > 06:00:13 .tcc 0 tcc W Text="Device Command A=5290688; B=5269132; > C=5354218; D=2977; E=-423; XQ #MOVE Failing: ?XQ#STA commanded position" > 06:00:13 .tcc 0 tcc W Text="Device Command A=5290688; B=5269132; > C=5354218; D=2977; E=-423; XQ #MOVE Failing: ?A=5290688; B=5269132; > C=5354218; D=2977; E=-423; XQ #MOVE" > 06:00:13 .tcc 0 tcc W Text="Device Command A=5290688; B=5269132; > C=5354218; D=2977; E=-423; XQ #MOVE Failing: ? 005256690, 005256545, > 005256854, 000003000, -000000400 actual position" > 06:00:13 .tcc 0 tcc w Text="Collimation Failure: ? 005256690, 005256545, > 005256854, 000003000, -000000400 actual position" > > When we are closed, we hardly see any disconnections. It seems to be tied > to observations and seems to begin about 3-4 hours after data collection > begins. > > > > ---- Observing Log ---- > > Afternoon: > ========== > > Checked disk, mirrors, actors. Took APOGEE, BOSS, and FPS cals. > > 22:07:51 boss exposure 0.0, file=349899 > 22:09:36 boss exposure 20.0, file=349900 > 22:11:59 boss exposure 4.0, file=349901 > 22:12:44 boss exposure 4.0, file=349902 > 22:13:29 sp1.motor.move= -2250, -2249, -2247 > 22:13:29 Hartmann collimate output > sp1: offset: r= -2638 (Out of focus); b= 5.8 (Out of focus) > sp1: pred. move: spAverageMove= -2250 > sp1: pred. resid: r= -387, b= -1.2, txt= OK, spTemp = 1.5 > 22:13:59 boss exposure 4.0, file=349903 > 22:15:27 boss exposure 25.0, file=349904 > > 22:18:20 Config Loaded: -999-7904 > 22:20:30 boss exposure 300.0, file=349905 > > 22:18:23 APO.aoravetz 64 jaeger i > configuration_snapshot=/data/logs/jaeger/configuration_snapshots/59929/configuration_snapshot_59929_7904.pdf > 22:18:23 APO.aoravetz 64 jaeger : text="Configuration 7904 loaded." > 22:18:23 CmdDone: APO.aoravetz 64 jaeger configuration load > --from-positions > > 22:19:50 APO.aoravetz 65 fliswarm i > filename=fvc,fvc1n,/data/fcam/59929/fimg-fvc1n-0001.fits > 22:19:51 APO.aoravetz 65 fliswarm i > exposure_state=fvc,fvc1n,idle,NA,0.0,0,0,0 > 22:19:51 APO.aoravetz 65 fliswarm i > filename_bundle=/data/fcam/59929/fimg-fvc1n-0001.fits > 22:19:51 APO.aoravetz 65 fliswarm : > 22:19:51 CmdDone: APO.aoravetz 65 fliswarm talk -n fvc expose 50 > > 23:00Z - Telescope checkout went ok. Started ventilating. > > Night: > ====== > > 00:00Z - Opened the enclosure and started a fill. > > 00:08:51 apogee.expose=apRaw-43670011, 5, A, DomeFlat > 00:10:06 apogee.expose=apRaw-43670012, 5, A, DomeFlat > 00:11:20 apogee.expose=apRaw-43670013, 5, A, DomeFlat > > 00:12:33 boss exposure 25.0, file=349906 > 00:14:22 boss exposure 25.0, file=349907 > 00:16:13 boss exposure 25.0, file=349908 > > ******************************************************** > 00:25:33 Field-Design-Configuration: 104466-128616-7905 > ******************************************************** > 00:25:41 GOTO-FIELD > 00:28:42 FVC LOOP AND BOSS CALS > fvc_rms= 13.02 > fvc_deltarms= 27.78 > fvc_filename=/data/fcam/59929/fimg-fvc1n-0003.fits > fvc_perc_90= 10.00 > fvc_percent_reached= 50.80 > 00:29:12 boss exposure 25.0, file=349909 > 00:31:10 boss exposure 4.0, file=349910 > 00:31:51 boss exposure 4.0, file=349911 > 00:32:45 sp1.motor.move= 3118, 3117, 3118 > 00:32:46 Hartmann collimate output > sp1: offset: r= 2638 (Out of focus); b= -11.3 (Out of focus) > sp1: pred. move: spAverageMove= 3118 > sp1: pred. resid: r= -480, b= -1.5, txt= OK, spTemp = 0.4 > 00:33:05 boss exposure 4.0, file=349912 > 00:39:37 apogee.expose=apRaw-43670014, 43, A, Object > 00:39:44 boss exposure 900.0, file=349913 > 00:47:18 apogee.expose=apRaw-43670015, 43, B, Object > > ******************************************************** > 00:54:55 Field-Design-Configuration: 104637-129881-7906 > ******************************************************** > 00:55:48 GOTO-FIELD > 00:58:18 FVC LOOP AND BOSS CALS > fvc_rms= 30.76 > fvc_deltarms= 9.11 > fvc_filename=/data/fcam/59929/fimg-fvc1n-0005.fits > fvc_perc_90= 11.30 > fvc_percent_reached= 51.00 > 00:58:48 boss exposure 25.0, file=349914 > 01:00:46 boss exposure 4.0, file=349915 > 01:01:28 boss exposure 4.0, file=349916 > 01:02:14 sp1.motor.move= -124, -123, -124 > 01:02:15 Hartmann collimate output > sp1: offset: r= 329 (In Focus); b= 1.8 (In Focus) > sp1: pred. move: spAverageMove= -124 > sp1: pred. resid: r= 453, b= 1.4, txt= OK, spTemp = 0.0 > 01:02:37 boss exposure 4.0, file=349917 > 01:04:34 New-field overhead: 0:08:45 > 01:04:34 START SCIENCE EXPOSURES > 01:04:56 apogee.expose=apRaw-43670016, 43, B, Object > 01:05:03 boss exposure 900.0, file=349918 > 01:12:42 apogee.expose=apRaw-43670017, 43, A, Object > 01:21:29 apogee.expose=apRaw-43670018, 43, A, Object > 01:21:37 boss exposure 900.0, file=349919 > 01:29:10 apogee.expose=apRaw-43670019, 43, B, Object > > ******************************************************** > 01:38:54 Field-Design-Configuration: 104637-129882-7907 > ******************************************************** > 01:38:59 GOTO-FIELD > 01:42:07 FVC LOOP AND BOSS CALS > fvc_rms= 9.57 > fvc_deltarms= 29.21 > fvc_filename=/data/fcam/59929/fimg-fvc1n-0007.fits > fvc_perc_90= 10.70 > fvc_percent_reached= 46.80 > 01:44:08 Same-field overhead: 0:06:25 > 01:44:09 START SCIENCE EXPOSURES > 01:44:31 apogee.expose=apRaw-43670020, 43, B, Object > 01:44:38 boss exposure 900.0, file=349920 > 01:52:17 apogee.expose=apRaw-43670021, 43, A, Object > 02:01:00 apogee.expose=apRaw-43670022, 43, A, Object > 02:01:07 boss exposure 900.0, file=349921 > 02:08:41 apogee.expose=apRaw-43670023, 43, B, Object > > ******************************************************** > 02:16:17 Field-Design-Configuration: 104637-129883-7908 > ******************************************************** > 02:17:14 GOTO-FIELD > 02:19:31 FVC LOOP AND BOSS CALS > fvc_rms= 9.33 > fvc_deltarms= 28.8 > fvc_filename=/data/fcam/59929/fimg-fvc1n-0009.fits > fvc_perc_90= 9.90 > fvc_percent_reached= 50.30 > 02:20:33 Same-field overhead: 0:03:19 > 02:20:33 START SCIENCE EXPOSURES > 02:20:56 apogee.expose=apRaw-43670024, 43, B, Object > 02:21:02 boss exposure 900.0, file=349922 > 02:28:41 apogee.expose=apRaw-43670025, 43, A, Object > 02:37:22 apogee.expose=apRaw-43670026, 43, A, Object > 02:37:29 boss exposure 900.0, file=349923 > 02:45:03 apogee.expose=apRaw-43670027, 43, B, Object > > ******************************************************** > 02:52:38 Field-Design-Configuration: 104637-129884-7909 > ******************************************************** > 02:53:34 GOTO-FIELD > 02:55:49 FVC LOOP AND BOSS CALS > fvc_rms= 7.79 > fvc_deltarms= 29.95 > fvc_filename=/data/fcam/59929/fimg-fvc1n-0011.fits > fvc_perc_90= 9.10 > fvc_percent_reached= 56.90 > 02:57:03 Same-field overhead: 0:03:29 > 02:57:03 START SCIENCE EXPOSURES > > 02:57:25 apogee.expose=apRaw-43670028, 43, B, Object > 02:57:32 boss exposure 900.0, file=349924 > 03:05:11 apogee.expose=apRaw-43670029, 43, A, Object > 03:13:56 apogee.expose=apRaw-43670030, 43, A, Object > 03:14:04 boss exposure 900.0, file=349925 > 03:21:38 apogee.expose=apRaw-43670031, 43, B, Object > 03:30:25 apogee.expose=apRaw-43670032, 43, B, Object > 03:30:32 boss exposure 900.0, file=349926 > 03:38:11 apogee.expose=apRaw-43670033, 43, A, Object > 03:48:45 apogee.expose=apRaw-43670034, 43, A, Object > 03:48:57 boss exposure 900.0, file=349926 > 03:56:27 apogee.expose=apRaw-43670035, 43, B, Object > 04:06:18 apogee.expose=apRaw-43670036, 43, B, Object > 04:06:32 boss exposure 900.0, file=349928 > 04:14:04 apogee.expose=apRaw-43670037, 43, A, Object > > ... > > This time period I wrestled with the m2 controller and multiple sp1cam > disconnects. Sometimes I started the boss icc multiple times and got the > mechanics confused. I need to logout ports multiple times. Everytime I > would try to start an exposure, another system would go down. So this time > was lost to ongoing issues. Jose chimed in at the exact moment I needed > him the most. > > ******************************************************** > 04:56:40 Field-Design-Configuration: 104648-130158-7912 > ******************************************************** > 04:58:02 GOTO-FIELD > 05:00:19 FVC LOOP AND BOSS CALS > fvc_rms= 15.45 > fvc_deltarms= 24.4 > fvc_filename=/data/fcam/59929/fimg-fvc1n-0015.fits > fvc_perc_90= 10.20 > fvc_percent_reached= 49.60 > 05:00:52 boss exposure 25.0, file=349930 > 05:03:32 boss exposure 4.0, file=349932 > 05:08:50 sp1.motor.move= None, None, None > 05:10:08 sp1.motor.move= None, None, None > 05:11:38 New-field overhead: 0:49:01 > 05:12:00 apogee.expose=apRaw-43670038, 43, A, Object > 05:19:42 apogee.expose=apRaw-43670039, 43, B, Object > > 05:27:47 boss exposure 900.0, file=349933 > > Jose thought we could try to expose and not guide. Considering that sp1cam > disconnected every minute up to this point, and this exposure ran without > failing would I think be considered a success. > > 05:54:14 apogee.expose=apRaw-43670040, 43, B, Object > 05:54:25 boss exposure 900.0, file=349938 > > So, Jose moved cherno to run on the FPS machine instead of /data to see if > it would stop the sp1cam disconnect issues. Unfortunately, it failed > really quickly and the exposure was lost trying to recover from it. > > 06:02:00 apogee.expose=apRaw-43670041, 43, A, Object > 06:07:39 boss exposure 0.0, file=349938 > > 06:09:52 apogee.expose=apRaw-43670042, 5, A, DomeFlat > 06:11:06 apogee.expose=apRaw-43670043, 5, A, DomeFlat > 06:12:12 apogee.expose=apRaw-43670044, 5, A, DomeFlat > > 06:15Z - Closed the enclosure for high humidity. > > 08:48Z - Started morning cals (for closer ties to observations). Turned > off the FPS. > > > ---- Mirror Numbers ---- > > > PRIMARY: > -------- > Scale: 1.000000 > > MIGS TONIGHT NOMINAL > Axial A: 0.0070 0.0080 > Axial B: 0.0030 0.0040 > Axial C: -0.0310 -0.0290 > Trans D: 0.0280 0.0320 > Lateral E: 0.3315 0.3320 > Lateral F: 0.5360 0.5350 > > GALILS > Commanded: 12500 -1600 5300 47050 -8800 -8400 > Actual: 12500 -1600 5300 47050 -8800 -8400 > > SETMIR VALUES > PrimDesOrient: 0.0 -59.5 -9.15 -512.0 1906.3 > PrimOrient: 1.07 -59.51 -9.21 -512.13 1906.31 > > SECONDARY: > ---------- > > Focus: 0.0000 > Air temp: 92.83 > Altitude: 30.0 > > MIGS TONIGHT NOMINAL > Axial A: -0.0310 -0.0330 > Axial B: -0.0260 -0.0290 > Axial C: -0.0440 -0.0450 > Trans D: -0.2820 -0.3170 > > GALILS > Commanded: 4902200 4877050 4973250 -50 -3450 > Actual: 4859255 4858984 4859274 -50 -3450 > > SETMIR VALUES > SecDesOrient: 3857.0 0.0 0.0 0.0 42.22 > SecOrient: 3856.97 -0.07 -0.08 -0.05 42.91 > > > > ---- Telescope Status ---- > > 23:00Z Started ventilation > 00:00Z encl25m : open > 06:15Z encl25m : closed > > No interlocks bypassed > > Status at: > Telescope stowed at: 121.0, 30.0, 0.0 mount > Instrument mounted: FPS > Counterweights at: 685/710 > LN2 autofill systems: Connected and turned on > 180L LN2 dewar scales: SP1 126 lbs, 19 psi > > FPS Status > ========== > CP10 Loop = 7.15 L/min > GFA Loop = 1.63 L/min > Supply pressure = 42.2 psi > Return pressure = 29.4 psi > > Chiller Status > ============== > Set temperarature = 1.1 C > Flow = 5.3 Gpm > > > > ---- Software Used ---- > > Get Versions, 12/15/22, 22:09:32 > -- STUI software: > STUI : 1.7.18b1 > alerts : 2.3.0 > apo : 3.0.1 > boss : 6.0.2-alpha.0 > -boss.specMec : 3.0.4 > guider : FAILED > ecamera : FAILED > fliswarm : 0.3.1 > jaeger : 1.2.0b2 > kaiju : 1.2.4-alpha.0 > coordio : 1.5.0-beta.2 > fps_calibrati : 2022.09.11+c5432fa > -- done -- > (observers39) [observer@sdss5-hub: ~]$ spVersion > --------------Other Versions-------------- > sdss-obstools : 3.9.2 > TPM : 1.2-31-gd7eb > MCP : 2.1 > PLC : V35 > Fiducials : v1_148 > idlspec2d : v6_0_9 > Kronos : 1.5.0 > roboscheduler : 1.2.2 > > /dev/sdb1 2.0T 236G 1.8T 12% /home > /dev/sdd1 22T 2.6T 19T 13% /data > > > ---- APOGEE Data Summary ---- > > APOGEE data list, mjd=59929, all > raw_data: /data/apogee/utr_cdr/59929/apRaw-*.fits > archive: /data/apogee/archive/59929/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/59929/apq-*.fits > ------------------------------------------------------------------------------ > mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset > %Flux/FPI Comm > ------------------------------------------------------------------------------ > 59929 1 18:20 43670001 Dark 30 A 13.0 ---------- a-b-c - > - > 59929 2 22:04 43670002 Dark 3 A 13.0 ---------- a-b-c - > - > 59929 3 22:16 43670003 Dark 60 A 13.0 ---------- a-b-c - > - > 59929 4 22:18 43670004 QuarFlat 10 A 13.0 ---------- a-b-c - > - > 59929 5 22:20 43670005 Arc-Thar 12 B 13.5 ---------- a-b-c 1.68 > - > 59929 6 22:27 43670006 Arc-Une 40 B 13.5 ---------- a-b-c - > - > 59929 7 22:29 43670007 Arc-Thar 12 A 13.0 ---------- a-b-c 1.15 > - > 59929 8 22:37 43670008 Arc-Une 40 A 13.0 ---------- a-b-c - > - > 59929 9 22:48 43670009 Dark 60 A 13.0 ---------- a-b-c - > - > 59929 10 22:53 43670010 Arc---- 30 A 13.0 ---------- a-b-c - > 88 FPI > 59929 11 00:09 43670011 DomeFlat 5 A 13.0 -999-7904 a-b-c - > 145 > 59929 12 00:10 43670012 DomeFlat 5 A 13.0 -999-7904 a-b-c - > 145 > 59929 13 00:11 43670013 DomeFlat 5 A 13.0 -999-7904 a-b-c - > 147 > 59929 14 00:46 43670014 Object 43 A 13.0 128616-7905 a-b-c -0.14 > 1,1 > 59929 15 00:54 43670015 Object 43 B 13.5 128616-7905 a-b-c 0.45 > 1,1 > 59929 16 01:12 43670016 Object 43 B 13.5 129881-7906 a-b-c 0.71 > 1,1 > 59929 17 01:19 43670017 Object 43 A 13.0 129881-7906 a-b-c 0.07 > 1,1 > 59929 18 01:28 43670018 Object 43 A 13.0 129881-7906 a-b-c 0.08 > 1,1 > 59929 19 01:36 43670019 Object 43 B 13.5 129881-7906 a-b-c 0.72 > 1,1 > 59929 20 01:51 43670020 Object 43 B 13.5 129882-7907 a-b-c 0.60 > 1,1 > 59929 21 01:59 43670021 Object 43 A 13.0 129882-7907 a-b-c 0.04 > 1,1 > 59929 22 02:08 43670022 Object 43 A 13.0 129882-7907 a-b-c 0.07 > 1,1 > 59929 23 02:15 43670023 Object 43 B 13.5 129882-7907 a-b-c 0.64 > 1,1 > 59929 24 02:28 43670024 Object 43 B 13.5 129883-7908 a-b-c 0.84 > 1,1 > 59929 25 02:35 43670025 Object 43 A 13.0 129883-7908 a-b-c 0.15 > 1,1 > 59929 26 02:44 43670026 Object 43 A 13.0 129883-7908 a-b-c 0.11 > 1,1 > 59929 27 02:52 43670027 Object 43 B 13.5 129883-7908 a-b-c 0.78 > 1,1 > 59929 28 03:04 43670028 Object 43 B 13.5 129884-7909 a-b-c 0.68 > 1,1 > 59929 29 03:12 43670029 Object 43 A 13.0 129884-7909 a-b-c 0.10 > 1,1 > 59929 30 03:21 43670030 Object 43 A 13.0 129884-7909 a-b-c 0.09 > 1,1 > 59929 31 03:28 43670031 Object 43 B 13.5 129884-7909 a-b-c 0.70 > 1,1 > 59929 32 03:37 43670032 Object 43 B 13.5 129884-7909 a-b-c 0.70 > 1,1 > 59929 33 03:45 43670033 Object 43 A 13.0 129884-7909 a-b-c 0.08 > 1,1 > 59929 34 03:55 43670034 Object 43 A 13.0 129884-7909 a-b-c 0.10 > 1,1 > 59929 35 04:03 43670035 Object 43 B 13.5 129884-7909 a-b-c 0.73 > 1,1 > 59929 36 04:13 43670036 Object 43 B 13.5 129884-7909 a-b-c 0.67 > 1,1 > 59929 37 04:21 43670037 Object 43 A 13.0 129884-7909 a-b-c 0.12 > 1,1 > 59929 38 05:19 43670038 Object 43 A 13.0 130158-7912 a-b-c 0.11 > 1,1 > 59929 39 05:26 43670039 Object 43 B 13.5 130158-7912 a-b-c 0.62 > 1,1 > 59929 40 06:01 43670040 Object 43 B 13.5 130158-7912 a-b-c 0.62 > 1,1 > 59929 41 06:09 43670041 Object 43 A 13.0 130158-7912 a-b-c 0.12 > 1,1 > 59929 42 06:10 43670042 DomeFlat 5 A 13.0 130158-7912 a-b-c - > 134 > 59929 43 06:11 43670043 DomeFlat 5 A 13.0 130158-7912 a-b-c - > 138 > 59929 44 06:12 43670044 DomeFlat 5 A 13.0 130158-7912 a-b-c - > 136 > cannot read file : /data/apogee/utr_cdr/59929/apRaw-43670045.fits > ------------------------------------------------------------------------------ > > > > ---- BOSS Data Summary ---- > > Reading FITS from /data/spectro//59929 > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > n Time Filename design-field cfg flav shu Exp colA colB > colC Hrt > - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - > - - > 1 16:59 b1-00349894 ----------- ---- dar Clo 900.0 -4754 -4803 -6048 > 2 16:59 r1-00349894 ----------- ---- dar Clo 900.0 -4754 -4803 -6048 > 3 17:16 r1-00349895 ----------- ---- dar Clo 90.0 -4754 -4803 -6048 > 4 17:16 b1-00349895 ----------- ---- dar Clo 90.0 -4754 -4803 -6048 > 5 17:19 r1-00349896 ----------- ---- dar Clo 90.0 -4754 -4803 -6048 > 6 17:19 b1-00349896 ----------- ---- dar Clo 90.0 -4754 -4803 -6048 > 7 17:22 b1-00349897 ----------- ---- dar Clo 900.0 -4754 -4803 -6048 > 8 17:22 r1-00349897 ----------- ---- dar Clo 900.0 -4754 -4803 -6048 > 9 17:40 r1-00349898 ----------- ---- dar Clo 900.0 -4754 -4803 -6048 > 10 17:40 b1-00349898 ----------- ---- dar Clo 900.0 -4754 -4803 -6048 > 11 22:07 r1-00349899 ----------- ---- bia Clo 0.0 -4754 -4803 -6048 > 12 22:07 b1-00349899 ----------- ---- bia Clo 0.0 -4754 -4803 -6048 > 13 22:09 r1-00349900 ----------- ---- dar Clo 20.0 -4754 -4803 -6048 > 14 22:09 b1-00349900 ----------- ---- dar Clo 20.0 -4754 -4803 -6048 > 15 22:11 b1-00349901 126260-103721 7903 arc Lef 4.1 -4754 -4803 > -6048 HART > 16 22:11 r1-00349901 126260-103721 7903 arc Lef 4.1 -4754 -4803 > -6048 HART > 17 22:12 b1-00349902 126260-103721 7903 arc Rig 4.1 -4754 -4803 > -6048 HART > 18 22:12 r1-00349902 126260-103721 7903 arc Rig 4.1 -4754 -4803 > -6048 HART > 19 22:13 b1-00349903 126260-103721 7903 arc Out 4.1 -7004 -7052 > -8295 > 20 22:13 r1-00349903 126260-103721 7903 arc Out 4.1 -7004 -7052 > -8295 > 21 22:15 r1-00349904 126260-103721 7903 fla Out 25.1 -7004 -7052 > -8295 > 22 22:15 b1-00349904 126260-103721 7903 fla Out 25.1 -7004 -7052 > -8295 > 23 22:20 b1-00349905 -999--999 7904 sci Out 300.1 -7004 -7052 -8295 > 24 22:20 r1-00349905 -999--999 7904 sci Out 300.1 -7004 -7052 -8295 > 25 00:12 r1-00349906 -999--999 7904 fla Out 25.1 -7004 -7052 -8295 > 26 00:12 b1-00349906 -999--999 7904 fla Out 25.1 -7004 -7052 -8295 > 27 00:14 b1-00349907 -999--999 7904 fla Out 25.1 -7004 -7052 -8295 > 28 00:14 r1-00349907 -999--999 7904 fla Out 25.1 -7004 -7052 -8295 > 29 00:16 b1-00349908 -999--999 7904 fla Out 25.1 -7004 -7052 -8295 > 30 00:16 r1-00349908 -999--999 7904 fla Out 25.1 -7004 -7052 -8295 > 31 00:29 r1-00349909 128616-104466 7905 fla Out 25.1 -7004 -7052 > -8295 > 32 00:29 b1-00349909 128616-104466 7905 fla Out 25.1 -7004 -7052 > -8295 > 33 00:31 r1-00349910 128616-104466 7905 arc Lef 4.1 -7004 -7052 > -8295 HART > 34 00:31 b1-00349910 128616-104466 7905 arc Lef 4.1 -7004 -7052 > -8295 HART > 35 00:31 b1-00349911 128616-104466 7905 arc Rig 4.1 -7004 -7052 > -8295 HART > 36 00:31 r1-00349911 128616-104466 7905 arc Rig 4.1 -7004 -7052 > -8295 HART > 37 00:33 r1-00349912 128616-104466 7905 arc Out 4.1 -3886 -3935 > -5177 > 38 00:33 b1-00349912 128616-104466 7905 arc Out 4.1 -3886 -3935 > -5177 > 39 00:39 r1-00349913 128616-104466 7905 sci Out 900.1 -3886 -3935 > -5177 > 40 00:39 b1-00349913 128616-104466 7905 sci Out 900.1 -3886 -3935 > -5177 > 41 00:58 b1-00349914 129881-104637 7906 fla Out 25.1 -3886 -3935 > -5177 > 42 00:58 r1-00349914 129881-104637 7906 fla Out 25.1 -3886 -3935 > -5177 > 43 01:00 b1-00349915 129881-104637 7906 arc Lef 4.1 -3886 -3935 > -5177 HART > 44 01:00 r1-00349915 129881-104637 7906 arc Lef 4.1 -3886 -3935 > -5177 HART > 45 01:01 b1-00349916 129881-104637 7906 arc Rig 4.1 -3886 -3935 > -5177 HART > 46 01:01 r1-00349916 129881-104637 7906 arc Rig 4.1 -3886 -3935 > -5177 HART > 47 01:02 b1-00349917 129881-104637 7906 arc Out 4.1 -4010 -4058 > -5301 > 48 01:02 r1-00349917 129881-104637 7906 arc Out 4.1 -4010 -4058 > -5301 > 49 01:05 r1-00349918 129881-104637 7906 sci Out 900.1 -4010 -4058 > -5301 > 50 01:05 b1-00349918 129881-104637 7906 sci Out 900.1 -4010 -4058 > -5301 > 51 01:21 r1-00349919 129881-104637 7906 sci Out 900.1 -4010 -4058 > -5301 > 52 01:21 b1-00349919 129881-104637 7906 sci Out 900.1 -4010 -4058 > -5301 > 53 01:44 b1-00349920 129882-104637 7907 sci Out 900.1 -4010 -4058 > -5301 > 54 01:44 r1-00349920 129882-104637 7907 sci Out 900.1 -4010 -4058 > -5301 > 55 02:01 b1-00349921 129882-104637 7907 sci Out 900.1 -4010 -4058 > -5301 > 56 02:01 r1-00349921 129882-104637 7907 sci Out 900.1 -4010 -4058 > -5301 > 57 02:21 b1-00349922 129883-104637 7908 sci Out 900.1 -4010 -4058 > -5301 > 58 02:21 r1-00349922 129883-104637 7908 sci Out 900.1 -4010 -4058 > -5301 > 59 02:37 r1-00349923 129883-104637 7908 sci Out 900.1 -4010 -4058 > -5301 > 60 02:37 b1-00349923 129883-104637 7908 sci Out 900.1 -4010 -4058 > -5301 > 61 02:57 r1-00349924 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 62 02:57 b1-00349924 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 63 03:14 b1-00349925 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 64 03:14 r1-00349925 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 65 03:48 r1-00349927 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 66 03:48 b1-00349927 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 67 04:06 b1-00349928 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 68 04:06 r1-00349928 129884-104637 7909 sci Out 900.1 -4010 -4058 > -5301 > 69 04:25 r1-00349929 129997-104643 7910 fla Out 25.1 -4010 -4058 > -5301 > 70 04:25 b1-00349929 129997-104643 7910 fla Out 25.1 -4010 -4058 > -5301 > 71 05:00 r1-00349931 130158-104648 7912 fla Out 25.1 -4010 -4058 > -5301 > 72 05:00 b1-00349931 130158-104648 7912 fla Out 25.1 -4010 -4058 > -5301 > 73 05:03 r1-00349932 130158-104648 7912 arc Out 4.1 -4010 -4058 > -5301 > 74 05:03 b1-00349932 130158-104648 7912 arc Out 4.1 -4010 -4058 > -5301 > 75 05:27 b1-00349937 130158-104648 7912 sci Out 900.1 -4010 -4058 > -5301 > 76 05:27 r1-00349937 130158-104648 7912 sci Out 900.1 -4010 -4058 > -5301 > 77 06:07 b1-00349939 ----------- ---- bia Ope 0.0 -4010 -4058 -5301 > 78 06:07 r1-00349939 ----------- ---- bia Ope 0.0 -4010 -4058 -5301 > > > ---- BOSS Spectrographs Collimation ---- > > Time TSP1 R1 B1 Move1 B1Resid > ============================================= > 00:32:20 0.3 2638 -11.3 3118 -1.5 > 01:02:10 -0.0 329 1.8 -124 1.4 > > > ---- Weather Log ---- > > Wind 1 um DIMM IRSC > Time Temp C Dewp C MPH Dir Dust fwhm mean sigma > > 19:00 -2.7 -22.6 2.7 250.6 92 > 19:15 -2.6 -23.7 1.3 275.7 82 > 19:30 -2.8 -21.7 13.9 342.9 151 > 19:45 -2.8 -22.2 9.1 344.1 114 > 20:00 -2.6 -19.3 9.4 340.6 141 > 20:15 -2.4 -19.6 12.6 314.5 133 > 20:30 -2.2 -18.5 12.1 311.8 178 > 20:45 -2.4 -18.8 15.5 313.8 174 > 21:00 -2.4 -16.7 4.8 269.9 191 > 21:15 -2.2 -18.0 8.7 265.7 188 > 21:30 -2.4 -15.6 9.1 290.8 208 > 21:45 -2.5 -12.0 8.4 282.3 275 > 22:00 -2.3 -14.7 13.9 312.6 309 > 22:15 -2.4 -13.6 4.8 305.0 223 > 22:30 -2.1 -12.9 18.0 316.5 274 > 22:45 -1.7 -11.9 6.6 285.8 280 > 23:00 -1.4 -11.7 13.7 292.1 312 > 23:15 -2.4 -12.9 11.0 311.1 207 > 23:30 -2.6 -12.1 3.9 321.8 237 > 23:45 -2.2 -12.2 5.8 301.1 264 > 00:00 -2.1 -11.8 16.3 311.4 309 > 00:15 -2.6 -12.6 17.6 310.2 244 > 00:30 -2.8 -12.6 6.5 333.3 224 > 00:45 -3.3 -12.6 7.4 291.9 217 343 2.55 > 01:00 -3.8 -13.2 8.1 297.4 178 343 2.14 > 01:15 -4.3 -13.8 8.2 265.2 154 343 1.97 > 01:30 -4.9 -13.8 7.9 258.3 143 343 5.18 > 01:45 -5.3 -15.1 6.0 207.8 164 343 2.44 > 02:00 -5.6 -11.7 8.3 234.6 246 343 3.97 > 02:15 -5.3 -10.8 12.0 249.5 338 343 3.05 > 02:30 -5.3 -9.4 13.2 265.5 305 343 2.40 > 02:45 -5.1 -9.1 13.5 262.5 334 343 2.43 > 03:00 -5.2 -8.9 10.7 262.4 304 343 2.17 > 03:15 -4.8 -8.9 12.0 271.6 314 289 3.47 > 03:30 -4.7 -8.9 16.0 270.3 300 343 2.25 > 03:45 -4.7 -9.1 14.1 273.8 323 343 2.25 > 04:00 -4.7 -8.9 15.3 288.8 311 343 2.91 > 04:15 -4.6 -8.6 9.0 263.9 399 343 2.33 > 04:30 -4.7 -8.5 9.0 268.0 300 343 2.20 > 04:45 -4.7 -8.4 16.1 294.5 321 343 2.12 > 05:00 -4.8 -8.3 6.9 266.0 352 343 2.39 > 05:15 -4.8 -8.3 18.9 308.3 353 343 2.31 > 05:30 -4.7 -8.2 14.2 308.5 363 343 2.18 > 05:45 -5.1 -8.1 15.2 339.8 367 343 2.46 > 06:00 -5.2 -8.1 6.4 313.6 344 343 2.38 > 06:15 -5.3 -8.2 11.5 350.6 373 343 2.36 > 06:30 -5.6 -8.4 6.7 42.5 338 343 2.35 > 06:45 -5.8 -8.3 6.6 349.9 352 342 2.72 > 07:00 -5.9 -8.3 19.0 349.7 390 345 10.10 > 07:15 -5.9 -8.2 14.3 334.8 381 343 17.55 > 07:30 -5.1 -10.1 17.3 345.5 469 344 2.96 > 07:45 -5.7 -9.6 20.7 313.4 490 345 38.62 > 08:00 -5.3 -10.7 14.4 305.5 440 344 3.01 > 08:15 -5.7 -10.2 20.2 338.6 431 343 12.96 > 08:30 -5.8 -10.2 2.6 323.5 409 342 8.13 > 08:45 -5.8 -9.7 13.9 321.1 449 368 46.61 > > > ---- Time Tracking ---- > > > *=*=*= Observatory: APO > *=*=*= MJD: 59929 > Fill night time in Hours below > *=*=*= Total bright time : 6.5 > *=*=*= Bright time, queued observations : 0.3 > *=*=*= Bright time, commissioning/engineering: 0 > *=*=*= Bright time lost to weather : 6.2 > *=*=*= Bright time lost to technical issues : 0 > > *=*=*= Total dark time : 5.6 > *=*=*= Dark time, queued observations : 3.2 > *=*=*= Dark time, commissioning/engineering : 0 > *=*=*= Dark time lost to weather : 0.5 > *=*=*= Dark time lost to technical issues : 1.9 > > > ---- Problems in Detail ---- > > > 03:34:42 CmdStarted: apo.apo 677 tcc set weather sectrusstemp=-5.8, > airtemp=92.8, wspeed=13.3, wdir=278.7, pressure=72058.754, humid=1.000 > 03:34:42 apo.apo 677 tcc i AirTemp=92.80 > 03:34:42 apo.apo 677 tcc i SecTrussTemp=-5.80 > 03:34:42 apo.apo 677 tcc i PrimF_BFTemp=0.00,0.00 > 03:34:42 apo.apo 677 tcc i SecF_BFTemp=0.00,0.00 > 03:34:42 apo.apo 677 tcc i Pressure=72058.8 > 03:34:42 apo.apo 677 tcc i Humidity=1.00 > 03:34:42 apo.apo 677 tcc i TLapse=6.50 > 03:34:42 apo.apo 677 tcc i WindSpeed=13.3 > 03:34:42 apo.apo 677 tcc i WindDir=278.7 > 03:34:42 apo.apo 677 tcc i TimeStamp=5177878482.33 > 03:34:42 apo.apo 677 tcc : > 03:34:42 CmdDone: apo.apo 677 tcc set weather sectrusstemp=-5.8, > airtemp=92.8, wspeed=13.3, wdir=278.7, pressure=72058.754, humid=1.000 > 03:34:42 .tcc 0 tcc I > SecNetMountOffset=26211.54,5724.86,89327.95,18.95,18.91 > 03:34:42 .tcc 0 tcc I SecDesOrient=4296.17,0.0,0.0,0.0,-74.68,-13.0 > 03:34:42 .tcc 0 tcc I > SecCmdMount=5438765.77,5418244.83,5501853.85,4744.95,1344.92 > 03:34:42 .tcc 0 tcc I SecDesOrientAge=0.0 > 03:34:42 .tcc 0 tcc I > SecDesEncMount=5412550.42,5412516.19,5412527.39,4726.0,1326.0 > 03:34:42 .tcc 0 tcc I > SecModelMount=5412554.23,5412519.96,5412525.9,4726.01,1326.01 > 03:34:42 .tcc 0 tcc I SecMaxIter=3 > 03:34:42 .apo 1675 apo : pressure=21.279; windd=278.7; winds=13.3; > gustd=286.2; gusts=17.4; airtemp=-4.7; dewpoint=-9.1; dpErr=P; > humidity=71.6; windd25m=-999.0; winds25m=-999.0; encl25m=1; truss25m=-5.8; > airTempPT=92.8; dpTempPT=92.8; humidPT=100.0; irscmean=343; irscsd=2.28 > 03:34:43 .mcp 0 mcp i trace="Saw PLC version 35; expected "; > UNPARSEDTEXT="34\"\"" > 03:34:43 .fliswarm 0 fliswarm i status=gfa2,gfa2n,"MicroLine > ML4240",ML0122718,292,256,1,1,50,2,2098,2050,0,0,2048,2048,-20.0,5.5,3459,43.0 > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:43 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:43 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .tcc 0 tcc w Text="Collimation Failure: Timed out" > 03:34:44 .tcc 0 tcc I SecState=Moving,1,3,0.0,0.0 > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:44 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:44 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:45 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:45 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .tcc 0 tcc i TCCPos=-13.681490,56.109084,168.381413; > AxePos=-13.681844,56.109035,168.382360; > ObjInstAng=0.000008,0.000000,5177876158.3492498; > SpiderInstAng=-11.453388,-0.006111,5177878494.2927599 > 03:34:46 .tcc 0 tcc i AxisCmdState=Tracking,Tracking,Tracking; > AxisErrCode=OK,OK,OK; AxisNoTrack=FFF > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .apogeeql 0 apogeeql w text="guider cartridgeKey is not well > defined: KeyVar('guider', cartridgeLoaded=(None, None, None, None, None))" > 03:34:46 .apogeeql 0 apogeeql w text="failed to index KeyVar('guider', > version=(None,)) by None from version for v_guider: KeyVar('guider', > version=(None,)) is unknown" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > 03:34:46 .boss 2932 boss w text="this command has already been > finished!!!! (self.1 2932): camRead" > 03:34:46 .boss 2932 boss w text="Failed to extend fired timeout on sp1cam" > > ____________________________________________________________ > Logs of Nightly APO Activities > https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html__;!!DZ3fjg!6bsyLpT_lxQHElWCA1nlDi-eGnqOfzK98FAW5Mm73u7_X6ETNPU2L3-7vgueTggDpRPrAMYmJF2wzFnPoyuiT2M3$ >