From: Renbin Yan Date: Sat, 15 Feb 2020 15:23:43 -0500 Subject: Re: [sloan-25mlog 4986] Re: 2.5m obslog 20200214 (SJD 58894) --000000000000e28f51059ea31984 Content-Type: text/plain; charset="UTF-8" Hi Kaike, We re-reduced those exposures this morning after fixing a bug in DOS that will only be triggered when there are missing holes in the CMM files. The DOS reruns finished quickly. The CMM files for a bunch of plates in this drill run (2019.12.c.manga-apogee2) are missing the MaNGA holes so that is why we have so many "No hole" warnings. We will probably ship those plates back to UW for re-measurements after they are observed. best, -renbin On Sat, Feb 15, 2020 at 12:29 PM Kaike Pan wrote: > > > On Feb 15, 2020, at 6:09 AM, asimmons@apo.nmsu.edu wrote: > > > APACHE POINT OBSERVATORY SDSS 2.5M > > OBSERVING LOG > Friday February 14, 2020 > > (SJD 58894) > > ---- Observing Team ---- > > Karen Kinemuchi > Audrey Oravetz > > > ---- Observing Plan ---- > > > Fri 14 FEB 2020 58894.0 0.63 01:57 12:53 04:32 15:29 > ---- MaNGA until 07:06 ( 5.1 h), then APOGEE-2 ( 5.8 h) > > ---- Observing Summary ---- > > The night started out clear, but then quickly became mostly cloudy for > MaNGA-led > science. The sky mostly cleared back up for APOGEE-led science. > > **MaNGA-led** > ============= > > Cart 6, M&Ap 10min plate 12045 IC342_MGA; 2XNSE + 6XAB > Missing fibers: [34, 181, 211, 273] > Faint fibers: ['1 - 24', '26 - 28', '30 - 33', '35 - 60', 105, 121, 123, > 133, > 136, 148, '151 - 180'] > ----------------- > > Cart 2, M&Ap plate 8240, 177+34_MGA; 2XNSE + 6XAB > Missing fibers: [38, 57, 59, 198, 211, 259, 273] > Faint fibers: ['61 - 73', '75 - 90', 137, 173, '213 - 217', '219 - 226', > '228 > - 240'] > ----------------- > > Cart 1, M&Ap plate 12511, MJ135.11479+1.27731; 2XNSE + 6XAB > Missing fibers: [211, 273] > Faint fibers: ['1 - 30', '61 - 90', '121 - 180', 227] > ----------------- > --> ERROR REDUCING SCIENCE EXPOSURES WITH DOS FOR THIS PLATE > > **APOGEE-led** > ============== > > Cart 8, Ap&M plate 11781, NGC2632_btx; 2XDAB > Missing fibers: [35, 184, 211, 227, 273] > Faint fibers: ['121 - 180', '271 - 272', '274 - 276', '279 - 300'] > ----------------- > > Cart 5, Ap&M plate 12396, K2_C1_264+52_btx; 2XDAB + 4XC - DONE > Missing fibers: [61, 211, 273] > Faint fibers: ['31 - 60', '121 - 173', '175 - 183', '186 - 197', 199, > 202, > 204, '208 - 210', '212 - 213', 216, 222, '225 - 226', '229 - 231', 234, > '236 - > 247', '251 - 270'] > ----------------- > > Cart 3, Ap&M plate 11774, K2EB_C06_314+48_btx; 4XAB + 4XC > Missing fibers: [211, 273] > Faint fibers: ['121 - 192', '194 - 197', 199, 202, 204, '207 - 209', > '271 - > 272', '274 - 276', '278 - 300'] > ----------------- > > Cart 4, Ap&M plate 11777, K2EB_C06_327+54_btx; 5XAB + 5XC > Missing fibers: [20, 39, 151, 186, 193, 211, 241, 273, 278] > Faint fibers: ['1 - 19', '21 - 24', '26 - 37', '40 - 42', '45 - 56', 58, > 60, > 121, 123, '125 - 128', 134, 137, 148, 152, 162, '179 - 182', 185, '187 - > 188', > 190, 192, '194 - 196', 199, '201 - 202', 204] > ----------------- > > Notes: > ====== > > Note to observers: the TestActors script returns a hung process for the > alertsActor, but it is running (with verification of 'alerts ping' and > response > 'I am not dead yet!'). > > We witnessed an unexpected movement in altitude during afternoon checkout > at a > time-stamp of 00:28Z (and again at closing 12:56Z). After returning the > telescope to stow from instrument change, an axis stop was issued, then a > stop > button, but then we saw the altitude change +20 arcmin. See log body for > the > chain of events. The telescope moves slightly AFTER a stop button is put > in. > > DOS began misbehaving with cart 1. It gave the output for exposures 23, 24 > (the > arc and flat), but failed to give any results for the science exposures. > runHubWatch was running fine, but runDos produced an error message that > read > 'Failed to reduce data. Error inside IDL.' > > > Not sure whether someone re-reduce the science exposures on the plate. > When I checked > all science exposures (25 - 30) have been reduced at 10:20 AM. Also there > are warning messages > on the plate. Here is one of them: > > sdR-b2-00308725.fit.gz: *WARNING*: No hole at xfocal= -81.9097 yfocal= 35.0158 found in CMM data. > > Three days ago, I found that VM MaNGA is running very slow, the top > returns 99% st on CPU usage. Shane was informed, he > will invest why CPU is stolen on this VM. Is it possible that the > slowness contribute to the failure? > > > Last night and tonight, we noticed that cart 5 has a secondary latch that > has a > hard time disengaging (SE). > > Cart 5 and 4 are consistently showing a lot of faint fibers. > > > > ____________________________________________________________ > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html > --000000000000e28f51059ea31984 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi Kaike,We re-reduced thos= e exposures this morning after fixing a bug in DOS that will only be trigge= red when there are missing holes in the CMM files. The DOS reruns finished = quickly.=C2=A0=C2=A0 The CMM files for a bunch of plates in this drill run = (2019.12.c.manga-apogee2) are missing the MaNGA holes so that is why we hav= e so many "No hole" warnings. We will probably ship those plates = back to UW for re-measurements after they are observed.best,-renbinOn Sat, Feb 15, 2020 at 12:29 PM= Kaike Pan ERROR REDUCING SCIENCE EXPOSURES WITH DOS FOR THIS P= LATE**APOGEE-led**=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3DCart 8, Ap&M plate 11781, NGC2632_btx; 2XDAB Missing fibers: [= 35, 184, 211, 227, 273]Faint fibers: =C2=A0=C2=A0['121 - 180', = '271 - 272', '274 - 276', '279 - 300']---------= --------Cart 5, Ap&M plate 12396, K2_C1_264+52_btx; 2XDAB + 4XC= - DONEMissing fibers: [61, 211, 273]Faint fibers: =C2=A0=C2=A0[= 9;31 - 60', '121 - 173', '175 - 183', '186 - 197= 9;, 199, 202, 204, '208 - 210', '212 - 213', 216, 222, = '225 - 226', '229 - 231', 234, '236 - 247', = 9;251 - 270']-----------------Cart 3, Ap&M plate 11774,= K2EB_C06_314+48_btx; 4XAB + 4XC Missing fibers: [211, 273]Faint fi= bers: =C2=A0=C2=A0['121 - 192', '194 - 197', 199, 202, 204,= '207 - 209', '271 - 272', '274 - 276', '27= 8 - 300']-----------------Cart 4, Ap&M plate 11777, K2E= B_C06_327+54_btx; 5XAB + 5XC Missing fibers: [20, 39, 151, 186, 193, 21= 1, 241, 273, 278]Faint fibers: =C2=A0=C2=A0['1 - 19', '21 -= 24', '26 - 37', '40 - 42', '45 - 56', 58, 60, = 121, 123, '125 - 128', 134, 137, 148, 152, 162, '179 - 182&= #39;, 185, '187 - 188', 190, 192, '194 - 196', 199, = 9;201 - 202', 204]-----------------Notes:=3D=3D=3D=3D= =3D=3DNote to observers: the TestActors script returns a hung proce= ss for the alertsActor, but it is running (with verification of 'al= erts ping' and response 'I am not dead yet!').We wi= tnessed an unexpected movement in altitude during afternoon checkout at a time-stamp of 00:28Z (and again at closing 12:56Z). After returning the = telescope to stow from instrument change, an axis stop was issued, then= a stop button, but then we saw the altitude change +20 arcmin. See log= body for the chain of events. The telescope moves slightly AFTER a sto= p button is put in. DOS began misbehaving with cart 1. It gave the = output for exposures 23, 24 (the arc and flat), but failed to give any = results for the science exposures. runHubWatch was running fine, but ru= nDos produced an error message that read 'Failed to reduce data. Er= ror inside IDL.'Not su= re whether someone re-reduce the science exposures on the plate. When I che= ckedall science exposures (25 - 30) have been reduced at 10:20 A= M. Also there are warning messageson the plate. Here is one of t= hem:sdR-b2-003= 08725.fit.gz: WARNING: No hole at xfo= cal=3D -81.9097 yfocal=3D 35.0158 found in CMM data.Three days ago, I found that VM MaNGA is running very slow, the top return= s 99% st on CPU usage. Shane was informed, hewill invest why CPU= is stolen on this VM.=C2=A0 Is it possible that the slowness contribute to= the failure?=C2=A0= Last night and tonight, we noticed that cart 5 has a secondary latch that h= as ahard time disengaging (SE). Cart 5 and 4 are consistently s= howing a lot of faint fibers.____________________________________________________________ Logs of Nightly APO Activities https://trac.sdss.org/tracm= ailman/browser/private/sloan-25mlog/index.html --000000000000e28f51059ea31984--