Date: Wed, 24 Feb 2016 10:29:12 -0800 From: =?utf-8?Q?Jos=C3=A9_S=C3=A1nchez-Gallego?= Subject: Re: [sloan-25mlog 2851] 2.5m obslog 20160223 (SJD 57442) --56cdf678_30a11ea2_17bd4 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline Hi John, I think the observers issued a few of these, but I=E2=80=99m not complete= ly sure. In any case I have added that to the S&S page under tron, so tha= t we remember it next time. Thanks, Jos=C3=A9 -- Jos=C3=A9 S=C3=A1nchez-Gallego Research Associate Astronomy Department University of Washington On =46ebruary 24, 2016 at 10:24:04, John K. Parejko (parejkoj=40uw.edu) w= rote: I think the only thing you *may* have to do is: =20 hub startNubs boss =20 hub startNubs apogee =20 hub startNubs apogeeCal =20 hub startNubs mcp =20 hub startNubs tcc =20 Though I'm not sure whether any of these are actually *necessary*. But if= they are necessary, they *should* be sufficient. If not, there's a bug i= n either the actor models or in tron. =20 John =20 On 24=46eb 2016, at 10:04, Jos=C3=A9 S=C3=A1nchez-Gallego wrote: =20 > Hi Kaike, =20 > =20 > The source of the problem last night was almost for sure the restart we= did of the hub (both of the product and of the sdss4-hub server) in the = afternoon to test =232405. After restarting sdss4-hub all actor living th= ere were automatically, I didn=E2=80=99t restart anyone, apart from creat= ing a new screen for runHubWatch. =20 > =20 > If this problem of actors living outside sdss-hub having problems to co= mmunicate after a hub reboot is known, I agree it should be documented in= the servers and services page. I have added a quick note about the probl= em with the hartmanns here (https://trac.sdss.org/wiki/APO/Observatory/sd= ssProcedures/RestartActors=23a2.bossICC). =20 > =20 > It would be good to have a complete checklist of things that need to be= done (restart runHubWatch, restart bossICC, check apogeeActor, etc) afte= r a full restart of the hub server. =20 > =20 > Thanks, =20 > =20 > Jos=C3=A9 =20 > -- =20 > Jos=C3=A9 S=C3=A1nchez-Gallego =20 > Research Associate =20 > Astronomy Department =20 > University of Washington =20 > =20 > On =46ebruary 24, 2016 at 06:38:32, Kaike Pan (kpan=40apo.nmsu.edu) wro= te: =20 > =20 >> Patrick. the problems on the first APOGEE cart was due to software tes= ts. =20 >> We are in engineering time, and this is what the engineering time for.= =20 >> =46or time tracking, we don=E2=80=99t need to track this time =E2=80=9C= lost=E2=80=9D as operation =20 >> time lose. =20 >> =20 >> Observers, maybe, we should differentiate this kind of test time lose = =20 >> from real operation time lose in the night logs. e.g. We can describe = it as: =20 >> we are in engineering time, ran some software tests. After the test, =20 >> we have some issue on goto=46ield etc which took us =7E42 minutes to f= ix. =20 >> =20 >> Jon, Jose, what actors were restarted after your test on tron and hub-= rebooting. =20 >> I think that actors that don=E2=80=99t live on hub, i.e. BOSS-icc, har= tmann, APOGEE =20 >> related actors (APOGEE-ics,APOGEEql, apogeecal), sos, dos, platedb, ne= ed =20 >> to be checked. tcc needs to be reconnected. Checks include take test =20 >> exposures that involve those software. Sometimes, a actor is alive but= don=E2=80=99t =20 >> communicate properly with other actors, e.g. BOSS-icc could ping ok, a= nd actually =20 >> take exposures but the fits header don=E2=80=99t need correct lamp inf= ormation (this may be =20 >> the case last night) since it doesn=E2=80=99t communicate with mcp pro= perly. I will update =20 >> our procedure on this. =20 >> =20 >> Jon and Jose, do you have more information to include in the procedure= =3F Thanks, =20 >> =20 >> -kaike =20 >> =20 >> =20 >> > On =46eb 24, 2016, at 6:26 AM, kinemuchi=40apo.nmsu.edu wrote: =20 >> > =20 >> > =20 >> > APACHE POINT OBSERVATORY SDSS 2.5M =20 >> > =20 >> > OBSERVING LOG =20 >> > Tuesday =46ebruary 23, 2016 =20 >> > =20 >> > (SJD 57442) =20 >> > =20 >> > ---- Observing Team ---- =20 >> > =20 >> > Patrick Gaulme =20 >> > Karen Kinemuchi =20 >> > Elena Malanushenko (afternoon) =20 >> > Jose Sanchez-Gallego (online for Ticket =232415) =20 >> > Jon Brinkmann (phone) =20 >> > Kaike Pan (phone) =20 >> > Nathan De Lee (phone/online) =20 >> > =20 >> > ---- Observing Summary ---- =20 >> > =20 >> > Closed due to high humidity and blowing snow until 7:30Z. =20 >> > =20 >> > APOGEE lost the first 5 hours due to weather. =20 >> > APOGEE lost time due to software issues: =7E42 minutes see log for d= etails. =20 >> > =20 >> > APOGEE OBSERVING: =20 >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D =20 >> > Cart 1 (Plate 8913; =46ield=3D120+45) - no guide stars; offset +10=22= ; 1xAB + MaNGA 1xC =20 >> > =20 >> > Cart 4 (Plate 8923; =46ield=3DPAL5-1) - GSOGT=46; 1xDAB + MaNGA 2xC = =20 >> > =20 >> > Cart 6 (Plate 8634; =46ield=3DPAL5-2) - no guide stars; offset -10=22= ; 3xDAB + MaNGA 6xC. =20 >> > Missing fibers: =5B62, 105, 181=5D =20 >> > =46aint fibers: =5B15, 34, 99, 110, 133, 144, 173, 211, 217, 248, 27= 7, 297=5D =20 >> > Notes to day staff and developers: =20 >> > =20 >> > * APOGEE data from Cart 1 and 4 are sitting on apogee-ics and not ap= ogee-ql. =20 >> > Nathan will contact Holtzman and Neville to fix this. Thus, no missi= ng/faint =20 >> > fiber information this evening for these APOGEE carts. =20 >> > =20 >> > * Jose will update instructions on what actors need to be restarted = when hub is =20 >> > rebooted. =20 >> =20 >> =20 >> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =20 >> Logs of Nightly APO Activities =20 >> https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.h= tml =20 > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F =20 > Logs of Nightly APO Activities =20 > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.ht= ml =20 -- =20 ************************* =20 John Parejko =20 parejkoj=40uw.edu =20 http://staff.washington.edu/parejkoj/ =20 Department of Physics and Astronomy =20 University of Washington =20 Seattle, WA =20 ************************** =20 --56cdf678_30a11ea2_17bd4 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline body=7Bfont-family:Helvetica,Arial;font-size:13px=7DHi John,I think the observers issued a few of these, but I=E2=80=99m not co= mpletely sure. In any case I have added that to the S&S page under tr= on, so that we remember it next time.Thanks,Jos=C3=A9 --Jos=C3=A9 S=C3=A1nchez-GallegoResearch AssociateAstronomy DepartmentUni= versity of Washington On =46ebruary 24, 2016 at 10:24:04, John K. Parejko (parejkoj=40uw.edu) wrote: I think th= e only thing you *may* have to do is: hub startNubs boss hub startNubs apogee hub startNubs apogeeCal hub startNubs mcp hub startNubs tcc Though I'm not sure whether any of these are actually *necessary*. Bu= t if they are necessary, they *should* be sufficient. If not, there's a b= ug in either the actor models or in tron. John On 24=46eb 2016, at 10:04, Jos=C3=A9 S=C3=A1nchez-Gallego wrote: > Hi Kaike, > =20 > The source of the problem last night was almost for sure the res= tart we did of the hub (both of the product and of the sdss4-hub server) = in the afternoon to test =232405. After restarting sdss4-hub all actor li= ving there were automatically, I didn=E2=80=99t restart anyone, apart fro= m creating a new screen for runHubWatch. > =20 > If this problem of actors living outside sdss-hub having problem= s to communicate after a hub reboot is known, I agree it should be docume= nted in the servers and services page. I have added a quick note about th= e problem with the hartmanns here (https://trac.sdss.org/wiki/APO/Observa= tory/sdssProcedures/RestartActors=23a2.bossICC). > =20 > It would be good to have a complete checklist of things that nee= d to be done (restart runHubWatch, restart bossICC, check apogeeActor, et= c) after a full restart of the hub server. > =20 > Thanks, > =20 > Jos=C3=A9 > -- > Jos=C3=A9 S=C3=A1nchez-Gallego > Research Associate > Astronomy Department > University of Washington > =20 > On =46ebruary 24, 2016 at 06:38:32, Kaike Pan (kpan=40apo.nmsu.e= du) wrote: > =20 >> Patrick. the problems on the first APOGEE cart was due to so= ftware tests. >> We are in engineering time, and this is what the engineering= time for. >> =46or time tracking, we don=E2=80=99t need to track this tim= e =E2=80=9Clost=E2=80=9D as operation >> time lose. =20 >> =20 >> Observers, maybe, we should differentiate this kind of test = time lose =20 >> from real operation time lose in the night logs. e.g. We can= describe it as: >> we are in engineering time, ran some software tests. After t= he test, >> we have some issue on goto=46ield etc which took us =7E42 mi= nutes to fix. =20 >> =20 >> Jon, Jose, what actors were restarted after your test on tro= n and hub-rebooting. >> I think that actors that don=E2=80=99t live on hub, i.e. BOS= S-icc, hartmann, APOGEE =20 >> related actors (APOGEE-ics,APOGEEql, apogeecal), sos, dos, p= latedb, need >> to be checked. tcc needs to be reconnected. Checks include t= ake test =20 >> exposures that involve those software. Sometimes, a actor is= alive but don=E2=80=99t =20 >> communicate properly with other actors, e.g. BOSS-icc could = ping ok, and actually >> take exposures but the fits header don=E2=80=99t need correc= t lamp information (this may be >> the case last night) since it doesn=E2=80=99t communicate wi= th mcp properly. I will update >> our procedure on this. =20 >> =20 >> Jon and Jose, do you have more information to include in the= procedure=3F Thanks, >> =20 >> -kaike >> =20 >> =20 >> > On =46eb 24, 2016, at 6:26 AM, kinemuchi=40apo.nmsu.edu= wrote: >> > =20 >> > =20 >> > APACHE POINT OBSERVATORY SDSS 2.5M >> > =20 >> > OBSERVING LOG >> > Tuesday =46ebruary 23, 2016 >> > =20 >> > (SJD 57442) >> > =20 >> > ---- Observing Team ---- >> > =20 >> > Patrick Gaulme >> > Karen Kinemuchi >> > Elena Malanushenko (afternoon) >> > Jose Sanchez-Gallego (online for Ticket =232415) >> > Jon Brinkmann (phone) >> > Kaike Pan (phone) >> > Nathan De Lee (phone/online) >> > =20 >> > ---- Observing Summary ---- >> > =20 >> > Closed due to high humidity and blowing snow until 7:30= Z. >> > =20 >> > APOGEE lost the first 5 hours due to weather. >> > APOGEE lost time due to software issues: =7E42 minutes = see log for details. >> > =20 >> > APOGEE OBSERVING: >> > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D >> > Cart 1 (Plate 8913; =46ield=3D120+45) - no guide stars;= offset +10=22; 1xAB + MaNGA 1xC >> > =20 >> > Cart 4 (Plate 8923; =46ield=3DPAL5-1) - GSOGT=46; 1xDAB= + MaNGA 2xC >> > =20 >> > Cart 6 (Plate 8634; =46ield=3DPAL5-2) - no guide stars;= offset -10=22; 3xDAB + MaNGA 6xC. >> > Missing fibers: =5B62, 105, 181=5D >> > =46aint fibers: =5B15, 34, 99, 110, 133, 144, 173, 211,= 217, 248, 277, 297=5D >> > Notes to day staff and developers: >> > =20 >> > * APOGEE data from Cart 1 and 4 are sitting on apogee-i= cs and not apogee-ql. =20 >> > Nathan will contact Holtzman and Neville to fix this. T= hus, no missing/faint >> > fiber information this evening for these APOGEE carts. >> > =20 >> > * Jose will update instructions on what actors need to = be restarted when hub is >> > rebooted. =20 >> =20 >> =20 >> =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F >> Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25ml= og/index.html > =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F= =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F > Logs of Nightly APO Activities > https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/i= ndex.html -- ************************* John Parejko parejkoj=40uw.edu http://staff.washington.edu/parejkoj/ Department of Physics and Astronomy University of Washington Seattle, WA ************************** --56cdf678_30a11ea2_17bd4--