From: =?UTF-8?B?Sm9zw6kgU8OhbmNoZXotR2FsbGVnbw==?= Date: Fri, 8 Sep 2023 11:48:13 -0500 Subject: Re: [sloan-25mlog 6802] 2.5m obslog 20230907 (SJD 60195) --0000000000001fa36c0604dbbe83 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Actually I took a few images with the ecam and they look fine. status works after that. A few logging errors in the stdout but I don't think those are worrying. I killed ecameraICC after the test so you'll need to start it when you need it. Jos=C3=A9 On Sep 8, 2023 at 09:40:44, Jos=C3=A9 S=C3=A1nchez-Gallego wrote: > Regarding the error > > 07:22:12 APO.doravetz 210 ecamera f text=3D"command failed: > AttributeError(\"'CameraCmd' object has no attribute 'dataDir'\",) at > /home/sdss5/software/gcameraICC/main/python/gcameraICC/Commands/ > CameraCmd.py:113 " > > I cannot claim to remember that code anymore, but a quick look seems to > indicate that the dataDir attribute is not set until you take the first > exposure, so it may all be normal. If you take an exposure and that fails= , > or after it the ecamera status still fails, then we should look into it. > > Jos=C3=A9 > > > On Sep 8, 2023 at 03:00:53, doravetz@apo.nmsu.edu wrote: > >> >> APACHE POINT OBSERVATORY SDSS 2.5M >> >> OBSERVING LOG >> Thursday September 07, 2023 >> >> (SJD 60195) >> >> ---- Observing Team ---- >> >> Dan Oravetz >> Kaike Pan (online) >> John Donor (alerts debugging) >> John Downey (support) >> Jose Sanchez-Gallego (sp1daq help) >> >> ---- Observing Plan ---- >> >> Shakedown >> >> ---- Observing Summary ---- >> >> Ventilated all night except for the very beginning when the dust was hig= h. >> >> Completed APOGEE quartz flats in SparsePak configuration: exp. #'s >> 46330010 - 46330029. >> >> Homed/relaxed mirrors. >> >> Problems >> =3D=3D=3D=3D=3D=3D=3D=3D >> >> (1) I couldn't get sp1daq connected for BOSS. Attempting to reload >> yielded a "No route to host" error. Verified the ethernet was properly >> seated at the spectrograph and at the copper-to-fiber converter in the R= OT >> wrap. Additional connections will be checked tomorrow. >> >> (2) We sorted out some new engCam connections with new hardware, but we >> weren't able to get a response from the engCam Galil after establishing = a >> telnet connection to the new IP/port. The telnet session would respond t= hat >> a connection was made to the target IP, however there were no >> status/connection/messages that we expect to see from the Galil when a >> connection is made. John Downey noted that there are some configuration >> settings that may need to be adjusted as was done with similar >> serial-to-ethernet hardware for the specMech. >> >> (3) We updated default.dat with the mechanical collimation constants and >> homed/relaxed the mirrors (the homing position errors were all acceptabl= e >> and now all axes report "Homed"), but after updating the default.dat fil= e >> and loading it, there were some troubling errors/warnings. First off, >> Collimate no longer works and just produces "Collimation Failure:" error= s. >> Secondly, there are some "violates mount limits" messages for M2: >> >> 196 1 f text=3D"Commanded orientation [ 6.57000000e-01 -1.80011320e-05 >> -3.53623099e-05 6.07870000e-02 >> 4.27040000e-01] violates mount limits" >> >> Perhaps unsurprisingly given the above issues, a 'tcc set foc' command >> does not complete and instead fails. We'll have to double-check what's >> going on with those new M2 default.dat coefficients tomorrow. >> >> (4) 'ecamera status' fails with the following error: >> >> 07:22:12 APO.doravetz 210 ecamera f text=3D"command failed: >> AttributeError(\"'CameraCmd' object has no attribute 'dataDir'\",) at >> /home/sdss5/software/gcameraICC/main/python/gcameraICC/Commands/CameraCm= d.py:113" >> >> (5) There were a bunch of 'alerts status' commands being spawned/execute= d >> in the background every minute for ~5s straight related to camCheck aler= ts. >> John Donor had a look and noted that systemctl status indicated somethin= g >> to do with camCheck, but five of them happening at the same time. He >> restarted alertsActor and that appeared to have resolved the problem, bu= t >> later on we experienced similar behavior following some camCheck alerts. >> John noted: "I believe the problem went away on its own. The obvious thi= ng >> that changed was camcheck, and I believe I did enough tests to determine >> that was the cause. I believe it may have been because there were multip= le >> camcheck alerts? Causing some kind of race condition or something? I'm >> going to look into a sleep(0.01) or something tomorrow." >> >> ---- Observing Log ---- >> >> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D >> AFTERNOON CHECKOUT >> >> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D >> >> Most systems alive and responsive, but no FPS right now because the >> engCam is mounted. I moved the gang connector back to the dense port for >> now just to get the APOGEE evening cals. >> >> Afternoon disk space: >> >> Filesystem Size Used Avail Use% Mounted on >> /dev/sdb1 2.0T 290G 1.8T 15% /home >> /dev/sdd1 22T 8.7T 13T 42% /data >> >> There were a bunch of 'alerts status' commands being spawned/executed in >> the background every minute for ~5s straight. John Donor had a look and >> noted that systemctl status indicated something to do with camCheck, but >> five of them happening at the same time. He restarted alertsActor and th= at >> resolved the problem. For reference, the 'instrumentState instrument=3Db= oss >> up' command was issued at 18:12Z after BOSS was mounted and turned on an= d >> prior to that no repetitive 'alerts status'. >> >> 01:15Z - Ventilation started. >> >> A 'boss ping' showed sp1daq was not connected. Checked the ethernet at >> the camera and it wasn't fully seated. After seating it, the green statu= s >> LEDs were lit, but still no sp1daq connection, just a "No route to host.= " >> Additional connections will be checked tomorrow. >> >> Another instance of multiple 'alerts status' commands being spawned: >> >> 04:12:45 APO.jdonor: I believe the problem went away on its own. The >> obvious thing that changed was camcheck, and I believe I did enough test= s >> to determine that was the cause. I believe it may have been because ther= e >> were multiple camcheck alerts? Causing some kind of race condition or >> something? I'm going to look into a sleep(0.01) or something tomorrow. >> >> Couldn't talk to engCam galil. After the telnet indicated a connection >> was made to the target IP, the session was just froze with no input/outp= ut >> in terms of typing commands, etc. John Downey noted that there could be >> some configuration settings that need to be adjusted. >> >> Updated default.dat with mechanical collimation coefficients. This let t= o >> some "Collimation Failure:" errors. Still, we were able to home/relax: >> >> ****************** >> Home/Relax mirrors >> ****************** >> >> M1 >> =3D=3D >> >> home E,F: position error", "999999999, 999999999, 999999999, 999999999, >> 000000000, 000000000" >> home D: position error", "999999999, 999999999, 999999999, -000000050, >> 999999999, 999999999" >> home A,B,C: position error", "000000000, 000000000, 000000000, 999999999= , >> 999999999, 999999999" >> >> At Prim des orient =3D 0,0,0,72.02,1175.46 the Galils are: 8600, -150, >> 7800, 34500, 10950, 10950 >> >> Completed 'relax': 06:34:38 .tcc 0 tcc : JobStatus=3Drelax,Done >> >> M2 >> =3D=3D >> >> home D: position error", "999999999, 999999999, 999999999, 000000000, >> 999999999" >> home E: position error", "999999999, 999999999, 999999999, 999999999, >> 000000000" >> home A,B,C: home: position error", "-000000209, 000000005, -000000127, >> 999999999, 999999999" >> >> Noticed there were M2 "Commanded orientation [...] violates mount limits= " >> errors when trying a slew. We'll have to double-check the default.dat >> updates tomorrow. >> >> Completed SparsePak quartz flats for APOGEE and returned gang connector >> to dense port, then started morning cals sequence. >> >> Called it a night. Stopped ventilation, BOSS filled not too long ago. >> >> ---- Mirror Numbers ---- >> >> After updating default.dat, we were getting "Collimation Failure:" error= s >> in TCC. We were able to home/relax the mirrors with acceptable values fo= r >> position errors. When slewing the telescope and looking at the M2 mirror >> output, there were errors such as: >> >> 196 1 f text=3D"Commanded orientation [ 6.57000000e-01 -1.80011320e-05 >> -3.53623099e-05 6.07870000e-02 >> 4.27040000e-01] violates mount limits" >> >> Trying to set the M2 focus to 0 also failed (not surprisingly, perhaps, >> given the problem with commanded orientation). Anyway, for the time bein= g >> here are values for mirror numbers. I DID NOT SAVE NEW NOMINALS BECAUSE = OF >> THE OMINOUS M2 ORIENTATION ERRORS. >> >> PRIMARY: >> -------- >> Scale: 1.000000 >> >> MIGS TONIGHT NOMINAL >> Axial A: 0.2880 -0.0010 >> Axial B: -0.0660 -0.0230 >> Axial C: -0.0290 -0.0250 >> Trans D: -0.5640 0.0060 >> Lateral E: 0.3545 0.3315 >> Lateral F: 0.5525 0.5345 >> >> GALILS >> Commanded: 8600 -150 7800 34500 10950 10950 >> Actual: 8600 -150 7800 34500 10950 10950 >> >> SETMIR VALUES >> PrimDesOrient: 0.0 0.0 0.0 72.02 1175.46 >> PrimOrient: 0.0 0.0 0.0 72.77 1175.46 >> >> SECONDARY: >> ---------- >> >> Focus: 0.0000 >> Air temp: 0.00 >> Altitude: 30.0 >> >> MIGS TONIGHT NOMINAL >> Axial A: 3.6470 0.0400 >> Axial B: 3.5250 0.0950 >> Axial C: 3.6300 -0.0620 >> Trans D: -0.0520 -0.1060 >> >> GALILS >> Commanded: 0 0 0 0 0 >> Actual: -26438 -24636 -40983 0 0 >> >> SETMIR VALUES >> SecDesOrient: NaN NaN NaN NaN NaN >> SecOrient: -24.37 5.43 -0.53 -38.5 -3.55 >> >> ---- Telescope Status ---- >> >> Status at: 09:32Z >> >> Telescope stowed at: 121.0, 30.0, 0.0 mount >> Instrument mounted: FPS >> Counterweights at: 468 468 470 469 >> LN2 autofill systems: Connected and turned on >> 180L LN2 dewar scales: SP1 85 lbs, 17 psi >> >> ---- Software Used ---- >> >> STUI : 1.7.19 >> alerts : 2.3.0 >> boss : 6.0.2-alpha.0 >> -boss.specMec : 3.0.4 >> apo : 3.0.1 >> fliswarm : 0.4.0 >> jaeger : 1.4.1a0 >> kaiju : 1.3.1 >> coordio : 1.7.1 >> fps_calibrati : 2023.07.27 >> cherno : 0.6.3 >> astrometrynet : 0.93 >> hal : 0.6.0b5 >> hartmann : 2.0.2 >> hub : 5.0.1a0 >> mcp : 2.1 >> -mcp.plc : 35 >> -mcp.azFiduci : v1_147 >> -mcp.altFiduc : v1_147 >> -mcp.rotFiduc : v1_147 >> tcc : 1.4.0 >> apogee : not available >> apogeecal : not available >> apogeeql : v1_7_0dev >> STUIAdditions : 3.1.20 >> Boson : 0.3.2 >> sdss-obstools : 3.9.3 >> TPM : 1.2-31-gd7eb >> MCP : 2.1-1-ge461 >> PLC : V35 >> Fiducials : v1_147 >> idlspec2d : v6_1_0 >> Kronos : 1.6.2a >> roboscheduler : 1.4.0 >> >> ---- APOGEE Data Summary ---- >> >> APOGEE data list, mjd=3D60195, all >> raw_data: /data/apogee/utr_cdr/60195/apRaw-*.fits >> archive: /data/apogee/archive/60195/apR-[a,b,c]-*.apz >> quickred: /data/apogee/quickred/60195/apq-*.fits >> ------------------------------------------------------------------------= ------ >> >> mjd i UT File/Exp Imtype Nread A/B Dth Design-Cfg Arch Offset >> %Flux/FPI Comm >> >> ------------------------------------------------------------------------= ------ >> 60195 1 00:55 46330001 Dark 3 A 13.0 ---------- a-b-c - >> - >> 60195 2 01:25 46330002 Dark 60 A 13.0 ---------- a-b-c - >> - >> 60195 3 01:27 46330003 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 60195 4 01:29 46330004 Arc-Thar 12 B 13.5 ---------- a-b-c 1.03 >> - >> 60195 5 01:36 46330005 Arc-Une 40 B 13.5 ---------- a-b-c - >> - >> 60195 6 01:39 46330006 Arc-Thar 12 A 13.0 ---------- a-b-c 0.46 >> - >> 60195 7 01:46 46330007 Arc-Une 40 A 13.0 ---------- a-b-c - >> - >> 60195 8 01:57 46330008 Dark 60 A 13.0 ---------- a-b-c - >> - >> 60195 9 02:02 46330009 Arc---- 30 A 13.0 ---------- a-b-c - >> 89 FPI >> 60195 10 03:59 46330010 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 11 04:22 46330011 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 12 04:33 46330012 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 13 04:43 46330013 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 14 05:19 46330014 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 15 06:01 46330015 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 16 06:07 46330016 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 17 06:24 46330017 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 18 06:29 46330018 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 19 06:34 46330019 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 20 06:40 46330020 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 21 06:45 46330021 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 22 06:50 46330022 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 23 06:56 46330023 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 24 07:01 46330024 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 25 07:06 46330025 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 26 07:12 46330026 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 27 07:17 46330027 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 28 07:22 46330028 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 29 07:28 46330029 QuarFlat 30 A 13.0 ---------- a-b-c - >> - SparsePak >> 60195 30 08:07 46330030 Dark 60 A 13.0 ---------- a-b-c - >> - >> 60195 31 08:17 46330031 Dark 60 A 13.0 ---------- a-b-c - >> - >> 60195 32 08:28 46330032 Dark 60 A 13.0 ---------- a-b-c - >> - >> 60195 33 08:30 46330033 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 60195 34 08:32 46330034 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 60195 35 08:33 46330035 QuarFlat 10 A 13.0 ---------- a-b-c - >> - >> 60195 36 08:36 46330036 Arc-Thar 12 B 13.5 ---------- a-b-c 1.05 >> - >> 60195 37 08:43 46330037 Arc-Une 40 B 13.5 ---------- a-b-c - >> - >> 60195 38 08:45 46330038 Arc-Thar 12 A 13.0 ---------- a-b-c 0.47 >> - >> 60195 39 08:52 46330039 Arc-Une 40 A 13.0 ---------- a-b-c - >> - >> 60195 40 08:58 46330040 Dark 30 A 13.0 ---------- a-b-c - >> - >> 60195 41 09:03 46330041 InteFlat 30 A 13.0 ---------- a-b-c - >> - >> 60195 42 09:09 46330042 InteFlat 30 A 13.0 ---------- a-b-c - >> - >> 60195 43 09:14 46330043 InteFlat 30 A 13.0 ---------- a-b-c - >> - >> 60195 44 09:19 46330044 Dark 30 A 13.0 ---------- a-b-c - >> - >> 60195 45 09:25 46330045 Arc---- 30 A 13.0 ---------- a-b-c - >> 87 FPI >> >> ------------------------------------------------------------------------= ------ >> >> ---- Time Tracking ---- >> >> *=3D*=3D*=3D Observatory: APO >> *=3D*=3D*=3D MJD: 60195 >> Fill night time in Hours below >> *=3D*=3D*=3D Total bright time : 5.2 >> *=3D*=3D*=3D Bright time, queued observations : 0 >> *=3D*=3D*=3D Bright time, commissioning/engineering: 5.2 >> *=3D*=3D*=3D Bright time lost to weather : 0 >> *=3D*=3D*=3D Bright time lost to technical issues : 0 >> >> *=3D*=3D*=3D Total dark time : 4.1 >> *=3D*=3D*=3D Dark time, queued observations : 0 >> *=3D*=3D*=3D Dark time, commissioning/engineering : 4.1 >> *=3D*=3D*=3D Dark time lost to weather : 0 >> *=3D*=3D*=3D Dark time lost to technical issues : 0 >> >> ____________________________________________________________ >> Logs of Nightly APO Activities >> >> https://urldefense.com/v3/__https://trac.sdss.org/tracmailman/browser/pr= ivate/sloan-25mlog/index.html__;!!K-Hz7m0Vt54!nKtqiVvmgUGVD0sjqmoNeLTmws_sC= NX3_Yi4YIm4pNuUCT_frMne05xAZ_Fzpj_HjcbLWkNRIByGoRBtUxwoxg$ >> > --0000000000001fa36c0604dbbe83 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Actually I took a few images with the ecam and they look fine. status w= orks after that. A few logging errors in the stdout but I don't think t= hose are worrying.I kille= d ecameraICC after the test so you'll need to start it when you need it= .Jos=C3=A9 On Sep 8, 2023 at 09:40:44, Jos= =C3=A9 S=C3=A1nchez-Gallego wrote: Regarding the error07= :22:12 APO.doravetz 210 ecamera f text=3D"command failed: AttributeErr= or(\"'CameraCmd' object has no attribute 'dataDir'\&qu= ot;,) at /home/sdss5/software/gcameraICC/main/python/gcameraICC/Commands/C= ameraCmd.py:113"= I cannot claim to remember that code anymore, but a quick look seems to ind= icate that the dataDir attribute is not set until you take the first exposu= re, so it may all be normal. If you take an exposure and that fails, or aft= er it the ecamera status still fails, then we should look into it.Jos=C3=A9= On Sep 8, 2023 at 03:00:53, doravetz@apo.nmsu.edu wrote: =20 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0APACHE PO= INT OBSERVATORY SDSS 2.5M =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0OBSERVING LOG =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0Thursday September 07, 2023 =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0(SJD 60195)---- =C2=A0Observing Team = ----Dan OravetzKaike Pan (online)John Donor (alerts debuggi= ng)John Downey (support)Jose Sanchez-Gallego (sp1daq help)-= --- Observing Plan ----Shakedown---- Observing Summary ----= Ventilated all night except for the very beginning when the dust wa= s high.Completed APOGEE quartz flats in SparsePak configuration: ex= p. #'s 46330010 - 46330029.Homed/relaxed mirrors.Proble= ms=3D=3D=3D=3D=3D=3D=3D=3D(1) I couldn't get sp1daq connect= ed for BOSS. Attempting to reload yielded a "No route to host" er= ror. Verified the ethernet was properly seated at the spectrograph and at t= he copper-to-fiber converter in the ROT wrap. Additional connections will b= e checked tomorrow.(2) We sorted out some new engCam connections wi= th new hardware, but we weren't able to get a response from the engCam = Galil after establishing a telnet connection to the new IP/port. The telnet= session would respond that a connection was made to the target IP, however= there were no status/connection/messages that we expect to see from the Ga= lil when a connection is made. John Downey noted that there are some config= uration settings that may need to be adjusted as was done with similar seri= al-to-ethernet hardware for the specMech.(3) We updated default.dat= with the mechanical collimation constants and homed/relaxed the mirrors (t= he homing position errors were all acceptable and now all axes report "= ;Homed"), but after updating the default.dat file and loading it, ther= e were some troubling errors/warnings. First off, Collimate no longer works= and just produces "Collimation Failure:" errors. Secondly, there= are some "violates mount limits" messages for M2:196 1 f= text=3D"Commanded orientation [ =C2=A06.57000000e-01 =C2=A0-1.8001132= 0e-05 =C2=A0-3.53623099e-05 =C2=A0=C2=A06.07870000e-02 =C2=A0=C2=A04.27= 040000e-01] violates mount limits"Perhaps unsurprisingly given= the above issues, a 'tcc set foc' command does not complete and in= stead fails. We'll have to double-check what's going on with those = new M2 default.dat coefficients tomorrow.(4) 'ecamera status= 9; fails with the following error:07:22:12 APO.doravetz 210 ecamera= f text=3D"command failed: AttributeError(\"'CameraCmd' o= bject has no attribute 'dataDir'\",) at /home/sdss5/software/g= cameraICC/main/python/gcameraICC/Commands/CameraCmd.py:113"(5)= There were a bunch of 'alerts status' commands being spawned/execu= ted in the background every minute for ~5s straight related to camCheck ale= rts. John Donor had a look and noted that systemctl status indicated someth= ing to do with camCheck, but five of them happening at the same time. He re= started alertsActor and that appeared to have resolved the problem, but lat= er on we experienced similar behavior following some camCheck alerts. John = noted: "I believe the problem went away on its own. The obvious thing = that changed was camcheck, and I believe I did enough tests to determine th= at was the cause. I believe it may have been because there were multiple ca= mcheck alerts? Causing some kind of race condition or something? I'm go= ing to look into a sleep(0.01) or something tomorrow."---- Obs= erving Log ----=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0AFTERNOON CH= ECKOUT=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3DMost systems alive and responsive, but no FPS right = now because the engCam is mounted. I moved the gang connector back to the d= ense port for now just to get the APOGEE evening cals.Afternoon dis= k space:Filesystem =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Size =C2=A0Used Avail Use% Mounted on/dev/sdb1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A02.0T =C2=A0290G =C2=A01.8T =C2=A015% /home/d= ev/sdd1 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A022T =C2=A08.7T =C2=A0=C2=A013T =C2=A042% /dat= aThere were a bunch of 'alerts status' commands being spawn= ed/executed in the background every minute for ~5s straight. John Donor had= a look and noted that systemctl status indicated something to do with camC= heck, but five of them happening at the same time. He restarted alertsActor= and that resolved the problem. For reference, the 'instrumentState ins= trument=3Dboss up' command was issued at 18:12Z after BOSS was mounted = and turned on and prior to that no repetitive 'alerts status'.01:15Z - Ventilation started.A 'boss ping' showed sp1daq= was not connected. Checked the ethernet at the camera and it wasn't fu= lly seated. After seating it, the green status LEDs were lit, but still no = sp1daq connection, just a "No route to host." Additional connecti= ons will be checked tomorrow.Another instance of multiple 'aler= ts status' commands being spawned:04:12:45 APO.jdonor: I believ= e the problem went away on its own. The obvious thing that changed was camc= heck, and I believe I did enough tests to determine that was the cause. I b= elieve it may have been because there were multiple camcheck alerts? Causin= g some kind of race condition or something? I'm going to look into a sl= eep(0.01) or something tomorrow.Couldn't talk to engCam galil. = After the telnet indicated a connection was made to the target IP, the sess= ion was just froze with no input/output in terms of typing commands, etc. J= ohn Downey noted that there could be some configuration settings that need = to be adjusted.Updated default.dat with mechanical collimation coef= ficients. This let to some "Collimation Failure:" errors. Still, = we were able to home/relax:******************Home/Relax mirrors= ******************M1=3D=3Dhome E,F: position error&= quot;, "999999999, 999999999, 999999999, 999999999, 000000000, 0000000= 00"home D: position error", "999999999, 999999999, 99999= 9999, -000000050, 999999999, 999999999"home A,B,C: position error&= quot;, "000000000, 000000000, 000000000, 999999999, 999999999, 9999999= 99"At Prim des orient =3D 0,0,0,72.02,1175.46 the Galils are: = 8600, -150, 7800, 34500, 10950, 10950Completed 'relax': 06:= 34:38 .tcc 0 tcc : JobStatus=3Drelax,DoneM2=3D=3Dhome D= : position error", "999999999, 999999999, 999999999, 000000000, 9= 99999999"home E: position error", "999999999, 999999999,= 999999999, 999999999, 000000000"home A,B,C: home: position error&= quot;, "-000000209, 000000005, -000000127, 999999999, 999999999"Noticed there were M2 "Commanded orientation [...] violates mou= nt limits" errors when trying a slew. We'll have to double-check t= he default.dat updates tomorrow.Completed SparsePak quartz flats fo= r APOGEE and returned gang connector to dense port, then started morning ca= ls sequence.Called it a night. Stopped ventilation, BOSS filled not= too long ago.---- Mirror Numbers ----After updating defaul= t.dat, we were getting "Collimation Failure:" errors in TCC. We w= ere able to home/relax the mirrors with acceptable values for position erro= rs. When slewing the telescope and looking at the M2 mirror output, there w= ere errors such as:196 1 f text=3D"Commanded orientation [ =C2= =A06.57000000e-01 =C2=A0-1.80011320e-05 =C2=A0-3.53623099e-05 =C2=A0=C2=A06= .07870000e-02 =C2=A0=C2=A04.27040000e-01] violates mount limits"Trying to set the M2 focus to 0 also failed (not surprisingly, perhap= s, given the problem with commanded orientation). Anyway, for the time bein= g here are values for mirror numbers. I DID NOT SAVE NEW NOMINALS BECAUSE O= F THE OMINOUS M2 ORIENTATION ERRORS.PRIMARY:--------Scale: = 1.000000MIGS TONIGHT NOMINALAxial A: 0.2880 -0.0010Axial = B: -0.0660 -0.0230Axial C: -0.0290 -0.0250Trans D: -0.5640 0.0060Lateral E: 0.3545 0.3315Lateral F: 0.5525 0.5345GALILSCommanded: =C2=A08600 -150 7800 34500 10950 10950Actual: =C2=A0=C2=A0= =C2=A0=C2=A08600 -150 7800 34500 10950 10950SETMIR VALUESPrimDe= sOrient: 0.0 0.0 0.0 72.02 1175.46PrimOrient: =C2=A0=C2=A0=C2=A00.0 0.0= 0.0 72.77 1175.46SECONDARY:----------Focus: =C2=A0=C2= =A0=C2=A0=C2=A0=C2=A00.0000Air temp: =C2=A0=C2=A00.00Altitude: =C2= =A0=C2=A030.0MIGS TONIGHT NOMINALAxial A: 3.6470 0.0400A= xial B: 3.5250 0.0950Axial C: 3.6300 -0.0620Trans D: -0.0520 -0.= 1060GALILSCommanded: 0 0 0 0 0Actual: =C2=A0=C2=A0=C2=A0-26= 438 -24636 -40983 0 0SETMIR VALUESSecDesOrient: NaN NaN NaN NaN= NaNSecOrient: =C2=A0=C2=A0=C2=A0-24.37 5.43 -0.53 -38.5 -3.55-= --- Telescope Status ----Status at: 09:32ZTelescope stowed = at: =C2=A0=C2=A0=C2=A0121.0, 30.0, 0.0 mountInstrument mounted: =C2=A0= =C2=A0=C2=A0=C2=A0FPSCounterweights at: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A04= 68 468 470 469LN2 autofill systems: =C2=A0=C2=A0Connected and turned on= 180L LN2 dewar scales: =C2=A0SP1 85 lbs, 17 psi---- Software Us= ed ----STUI =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0:= 1.7.19alerts =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 2.3.0boss= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 6.0.2-alpha.0-= boss.specMec : 3.0.4apo =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0: 3.0.1fliswarm =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 0.4.0jaeger =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 1.4.1a0kaiju =C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 1.3.1coordio =C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0: 1.7.1fps_calibrati : 2023.07.27cherno =C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 0.6.3astrometrynet : 0.93h= al =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 0.6.0b5hartmann =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 2.0.2hub =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 5.0.1a0mcp =C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 2.1-mcp.plc =C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0: 35-mcp.azFiduci : v1_147-mcp.altFiduc : v1_147-mcp.rotFiduc : v1_147tcc =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0: 1.4.0apogee =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0: not availableapogeecal =C2=A0=C2=A0=C2=A0=C2=A0: not available= apogeeql =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: v1_7_0devSTUIAdditions : 3.1.2= 0Boson =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 0.3.2sdss-= obstools : 3.9.3 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0TPM =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0: 1.2-31-gd7eb =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0MCP =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 2.= 1-1-ge461 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0PLC =C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: V35 =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0Fiducials =C2=A0=C2=A0=C2=A0=C2=A0: v1_147 =C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0idlspec2d =C2=A0= =C2=A0=C2=A0=C2=A0: v6_1_0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Kronos =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0: 1.6.2a =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0roboscheduler : 1.4.0---- APOGEE Data Summ= ary ----APOGEE data list, =C2=A0=C2=A0mjd=3D60195, =C2=A0allraw= _data: =C2=A0/data/apogee/utr_cdr/60195/apRaw-*.fitsarchive: =C2=A0/dat= a/apogee/archive/60195/apR-[a,b,c]-*.apzquickred: =C2=A0/data/apogee/qu= ickred/60195/apq-*.fits------------------------------------------------= ------------------------------ mjd =C2=A0=C2=A0i =C2=A0=C2=A0UT =C2=A0= =C2=A0File/Exp =C2=A0Imtype Nread A/B Dth =C2=A0Design-Cfg =C2=A0Arch Offse= t %Flux/FPI Comm -----------------------------------------------------= -------------------------60195 =C2=A0=C2=A01 00:55 =C2=A046330001 =C2= =A0=C2=A0=C2=A0Dark =C2=A0=C2=A0=C2=A03 =C2=A0A =C2=A013.0 =C2=A0----------= =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2=A0=C2=A0= 2 01:25 =C2=A046330002 =C2=A0=C2=A0=C2=A0Dark =C2=A0=C2=A060 =C2=A0A =C2=A0= 13.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2=A0=C2=A03 01:27 =C2=A046330003 =C2=A0QuarFlat 10 =C2=A0A =C2= =A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2= =A060195 =C2=A0=C2=A04 01:29 =C2=A046330004 =C2=A0Arc-Thar 12 =C2=A0B = =C2=A013.5 =C2=A0---------- =C2=A0a-b-c =C2=A01.03 =C2=A0- =C2=A060195 = =C2=A0=C2=A05 01:36 =C2=A046330005 =C2=A0Arc-Une =C2=A040 =C2=A0B =C2=A013.= 5 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0= 60195 =C2=A0=C2=A06 01:39 =C2=A046330006 =C2=A0Arc-Thar 12 =C2=A0A =C2=A013= .0 =C2=A0---------- =C2=A0a-b-c =C2=A00.46 =C2=A0- =C2=A060195 =C2=A0= =C2=A07 01:46 =C2=A046330007 =C2=A0Arc-Une =C2=A040 =C2=A0A =C2=A013.0 =C2= =A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195= =C2=A0=C2=A08 01:57 =C2=A046330008 =C2=A0=C2=A0=C2=A0Dark =C2=A0=C2=A060 = =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0= =C2=A0- =C2=A060195 =C2=A0=C2=A09 02:02 =C2=A046330009 =C2=A0Arc---- = =C2=A030 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0= - =C2=A0=C2=A089 =C2=A0FPI60195 =C2=A010 03:59 =C2=A046330010 =C2=A0Qua= rFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0= - =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A011 04:22 =C2=A04633001= 1 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0= =C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A012 04:33 = =C2=A046330012 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0= a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2= =A013 04:43 =C2=A046330013 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0-----= ----- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A014 05:19 =C2=A046330014 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 = =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0= SparsePak60195 =C2=A015 06:01 =C2=A046330015 =C2=A0QuarFlat 30 =C2=A0A = =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- = =C2=A0=C2=A0SparsePak60195 =C2=A016 06:07 =C2=A046330016 =C2=A0QuarFlat= 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2= =A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A017 06:24 =C2=A046330017 =C2= =A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0= =C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A018 06:29 =C2=A04= 6330018 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c = =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A019 0= 6:34 =C2=A046330019 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- = =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak6019= 5 =C2=A020 06:40 =C2=A046330020 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0= ---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0Sparse= Pak60195 =C2=A021 06:45 =C2=A046330021 =C2=A0QuarFlat 30 =C2=A0A =C2=A0= 13.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0= =C2=A0SparsePak60195 =C2=A022 06:50 =C2=A046330022 =C2=A0QuarFlat 30 = =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0= =C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A023 06:56 =C2=A046330023 =C2=A0= QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2= =A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A024 07:01 =C2=A04633= 0024 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2= =A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A025 07:0= 6 =C2=A046330025 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2= =A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 = =C2=A026 07:12 =C2=A046330026 =C2=A0QuarFlat 30 =C2=A0A =C2=A013.0 =C2=A0--= -------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2=A0SparsePa= k60195 =C2=A027 07:17 =C2=A046330027 =C2=A0QuarFlat 30 =C2=A0A =C2=A013= .0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A0=C2= =A0SparsePak60195 =C2=A028 07:22 =C2=A046330028 =C2=A0QuarFlat 30 =C2= =A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2= =A0- =C2=A0=C2=A0SparsePak60195 =C2=A029 07:28 =C2=A046330029 =C2=A0Qua= rFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0= - =C2=A0=C2=A0- =C2=A0=C2=A0SparsePak60195 =C2=A030 08:07 =C2=A04633003= 0 =C2=A0=C2=A0=C2=A0Dark =C2=A0=C2=A060 =C2=A0A =C2=A013.0 =C2=A0----------= =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2=A031 08:= 17 =C2=A046330031 =C2=A0=C2=A0=C2=A0Dark =C2=A0=C2=A060 =C2=A0A =C2=A013.0 = =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060= 195 =C2=A032 08:28 =C2=A046330032 =C2=A0=C2=A0=C2=A0Dark =C2=A0=C2=A060 =C2= =A0A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2= =A0- =C2=A060195 =C2=A033 08:30 =C2=A046330033 =C2=A0QuarFlat 10 =C2=A0= A =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0-= =C2=A060195 =C2=A034 08:32 =C2=A046330034 =C2=A0QuarFlat 10 =C2=A0A = =C2=A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- = =C2=A060195 =C2=A035 08:33 =C2=A046330035 =C2=A0QuarFlat 10 =C2=A0A =C2= =A013.0 =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2= =A060195 =C2=A036 08:36 =C2=A046330036 =C2=A0Arc-Thar 12 =C2=A0B =C2=A0= 13.5 =C2=A0---------- =C2=A0a-b-c =C2=A01.05 =C2=A0- =C2=A060195 =C2=A0= 37 08:43 =C2=A046330037 =C2=A0Arc-Une =C2=A040 =C2=A0B =C2=A013.5 =C2=A0---= ------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2= =A038 08:45 =C2=A046330038 =C2=A0Arc-Thar 12 =C2=A0A =C2=A013.0 =C2=A0-----= ----- =C2=A0a-b-c =C2=A00.47 =C2=A0- =C2=A060195 =C2=A039 08:52 =C2=A04= 6330039 =C2=A0Arc-Une =C2=A040 =C2=A0A =C2=A013.0 =C2=A0---------- =C2=A0a-= b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2=A040 08:58 =C2=A0= 46330040 =C2=A0=C2=A0=C2=A0Dark =C2=A0=C2=A030 =C2=A0A =C2=A013.0 =C2=A0---= ------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2= =A041 09:03 =C2=A046330041 =C2=A0InteFlat 30 =C2=A0A =C2=A013.0 =C2=A0-----= ----- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2=A04= 2 09:09 =C2=A046330042 =C2=A0InteFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------= - =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2=A043 09= :14 =C2=A046330043 =C2=A0InteFlat 30 =C2=A0A =C2=A013.0 =C2=A0---------- = =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060195 =C2=A044 09:1= 9 =C2=A046330044 =C2=A0=C2=A0=C2=A0Dark =C2=A0=C2=A030 =C2=A0A =C2=A013.0 = =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A0- =C2=A060= 195 =C2=A045 09:25 =C2=A046330045 =C2=A0Arc---- =C2=A030 =C2=A0A =C2=A013.0= =C2=A0---------- =C2=A0a-b-c =C2=A0=C2=A0=C2=A0- =C2=A0=C2=A087 =C2=A0FPI------------------------------------------------------------------------= ---------- Time Tracking ----*=3D*=3D*=3D Observatory: APO*=3D*=3D*=3D MJD: 60195 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0Fill nig= ht time in Hours below*=3D*=3D*=3D Total bright time =C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0: 5.2*=3D*=3D*=3D Bright time, queued observ= ations =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 0*=3D*=3D*=3D Bright time, commi= ssioning/engineering: 5.2*=3D*=3D*=3D Bright time lost to weather =C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 0*=3D*=3D*= =3D Bright time lost to technical issues =C2=A0: 0*=3D*=3D*=3D Tota= l dark time =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 4.1*=3D*=3D*=3D Dark time, queued observations =C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0: 0*=3D*=3D*=3D Dark time, commissioning/engineering =C2= =A0: 4.1*=3D*=3D*=3D Dark time lost to weather =C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0: 0*=3D*=3D*=3D Dark ti= me lost to technical issues =C2=A0=C2=A0=C2=A0: 0__________________= __________________________________________Logs of Nightly APO Activitie= s