From: Renbin Yan Date: Sun, 6 Oct 2019 10:56:57 -0400 Subject: Re: [sloan-25mlog 4682] 2.5m obslog 20191005 (SJD 58762) --0000000000003d96b605943f267e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable HI Kaike, Thanks. Looks like the r1 electronic light contamination issue is back again. best, -renbin On Sun, Oct 6, 2019 at 10:44 AM Kaike Pan wrote: > > > > On Oct 6, 2019, at 6:32 AM, dgatlin@apo.nmsu.edu wrote: > > > > > > APACHE POINT OBSERVATORY SDSS 2.5M > > > > OBSERVING LOG > > Saturday October 05, 2019 > > > > (SJD 58762) > > > > ---- Observing Team ---- > > > > Elena Malanushenko > > Dan Oravetz > > > > Dylan Gatlin (training) > > > > ---- Observing Plan ---- > > > > > > Sat 5 OCT 2019 58762.0 0.51 01:37 11:53 19:31 05:50 > ---- > > APOGEE-2 until 06:30 ( 4.9 h), then MaNGA ( 5.4 h) > > > > > > ---- Observing Summary ---- > > > > It was a very clear night with excellent seeing. > > > > APOGEE lost 1.08h due to software related to MCP. > > > > ### Nighttime > > > > We had difficulties with MCP, TCC and semaphore. We called Jeff and wer= e > able to > > resolve the issue. We restarted MCP, TCC, and alertsActor, and rolled > MCP back > > to a previous version, but this version does not have a version number. > > Unfortunately, by the time we were back on field it was too > > late for cart 2, so we moved on to cart 5. See cart 2 in log for detail= s. > > > > After going to field for cart 2, we restarted the guider > > > > John Downey requested that we test the SOP slew scripts like Goto Gang > Change > > and Goto Instrument Change, but since we rolled back the MCP, we decide= d > to not > > do those test and continued to use tcc track commands instead. > > > > Cart 5, 11266, K19_076+07_btx, 2xDAB, 4xC, done > > Missing fibers: [211, 273] > > Faint fibers: ['122 - 123', '126 - 127', 133, 137, 139, 147, 154] > > > > Cart 8, 11595, K2_C12_082-60_btx, 2xAB, done > > Missing fibers: [35, 184, 211, 227, 273] > > Faint fibers: [123, 127, 148] > > > > Cart 9, 11424, NGC188_btx, 3xAB, done > > Missing fibers: [36, 211, 245, 273] > > Faint fibers: [123, 127, 136] > > > > Cart 1, 8075, MJ359.55450+14.94322, 2xNSE, done > > > > Cart 4, 10224, MJ35.68307-0.00000, 2xNSE, 6xAB, not done > > Missing fibers: [20, 39, 151, 186, 193, 211, 241, 273, 278] > > Faint fibers: [48, '121 - 128', 140, 142, 145, 243, 284] > > > > Cart 3, 10480, MJ55.91016+68.43894, 2xNSE, 6xAB, not done > > Missing fibers: [211, 273] > > Faint fibers: [123, '125 - 126'] > > > > ### Notes for developers and day staff > > > > We had another fast slew issue on GotoField, see Cart 3 log for details= . > > MaNGA team: here is a DOS warn on cart 5 ; Observers, please have the > warns in the Summary section. > > sdR-r1-00296247.fit.gz: WARNING: pixel bias rms for camera r1 is 4.79727x > above > nominal limit- is there an electronics problem?? > sdR-r1-00296247.fit.gz: WARNING: r1 blowtorch median 208.32152 > (electronics > failure??) > sdR-r1-00296248.fit.gz: WARNING: r1 blowtorch median 156.96501 > (electronics > failure??) > > > > > > > ### There was a problem with the MCP where it wouldn't send the > semaphore to the > > TCC. As a result, all SOP commands failed, as well as axis init. When w= e > went to > > 60,60,60, it failed, saying that semaphore owner was None. We called > Jeff to > > address the issue. We rebooted TCC which did not fix the issue, we > rebooted MCP, > > we rolled back MCP version. We also restarted the alerts actor. > > > > ### We believe that there might be some confusion on the version we > rolled back > > to, and it's possible we did not actually roll back to a previous > version. The > > previous version that we have used was 1.1|2019-09-27|13:31. A few > nights ago we > > updated to 1.1-14-g00bc|2019-10-03|11:22. The current version is > > |2019-10-03|11:09 (ie. no version number, but it was from the same date > as > > 1.1-14, with a slightly different timestamp. > > > > 01:17:59 OBSERVER.telescope 98 sop e text=3D"Cannot axis init: Semaphor= e > is owned > > by None:0:0" > > 01:17:59 OBSERVER.telescope 98 sop e text=3D"If you are the owner (e.g.= , > via MCP > > Menu), release it and try again." > > 01:17:59 OBSERVER.telescope 98 sop e text=3D"If you are not the owner, > confirm > > that you can steal it from them, then issue: mcp sem.steal" > > 01:17:59 OBSERVER.telescope 98 sop f text=3D"Failed to slew to position > az=3D60, > > alt=3D60, rot=3D60" > > 01:17:59 CmdDone: OBSERVER.telescope 98 sop gotoAll60 > > > > ### mcp sem_give ran, but it didn't change anything. The same is true > with mcp > > sem_Steal > > > > 01:30:50 CmdStarted: OBSERVER.telescope 108 mcp sem_Steal > > 01:30:50 .mcp 0 mcp i mcpVersion=3D"1.1-14-g00bc|2019-10-03|11:22" > > 01:30:50 .mcp 0 mcp i haveSemaphore=3Dfalse > > 01:30:50 .mcp 0 mcp i semaphoreOwner=3D"None:0:0" > > 01:30:50 OBSERVER.telescope 108 mcp i haveSemaphore=3Dtrue > > 01:30:50 OBSERVER.telescope 108 mcp i semaphoreOwner=3D"hub:hub:38641" > > 01:30:50 OBSERVER.telescope 108 mcp : command=3D"sem_steal" > > 01:30:50 CmdDone: OBSERVER.telescope 108 mcp sem_Steal > > > > 01:39:21 CmdStarted: OBSERVER.elenam 54 mcp sem_give > > 01:39:21 OBSERVER.elenam 54 mcp i haveSemaphore=3Dfalse > > 01:39:21 OBSERVER.elenam 54 mcp i semaphoreOwner=3D"None:0:0" > > 01:39:21 OBSERVER.elenam 54 mcp : command=3D"sem_give" > > 01:39:21 CmdDone: OBSERVER.elenam 54 mcp sem_give > > > > ### All SOP commands check the semaphore owner before running the > command. If > > MCP doesn't hold the semaphore, then the TCC should grab it, and we > should never > > see the semaphore owner as None. For some reason, TCC wasn't grabbing t= he > > semaphore. Jeff believes he found this bug, but we decided to stop > debugging as > > soon as possible and go to normal observing by rolling back MCP > versions. Jeff > > believes this issue arose from a discrepancy between the version he was > testing > > with and the version we were using. > > > > ### When going to field, it failed because a guider flat was not taken. > The > > guider flat was not taken because the guiderActor was hung. We kill -9'= d > a hung > > guider process and restarted the actor. > > > > 02:28:53 boss exposure 4.0, file=3D296237, S > > 02:29:32 boss exposure 4.0, file=3D296238, S > > 02:30:49 sp1.motor.move=3D 519, 518, 518 > > 02:30:52 sp2.motor.move=3D -787, -794, -786 > > 02:30:52 Hartmann collimate output on cart #2 > > sp1: offset: r=3D 329 (In Focus); b=3D -2.2 (In Focus) > > sp2: offset: r=3D -315 (In Focus); b=3D 3.9 (In Focus) > > sp1: pred. move: spAverageMove=3D 518 > > sp2: pred. move: spAverageMove=3D -787 > > sp1: pred. resid: r=3D -189, b=3D -0.6, txt=3D OK, spTemp =3D 7.9 > > sp2: pred. resid: r=3D 472, b=3D 1.5, txt=3D OK, spTemp =3D 12.1 > > 02:31:11 boss exposure 4.0, file=3D296239, S > > 02:32:38 boss exposure 25.0, file=3D296240, S > > 02:34:08 sop.gotoField =3D failed; > > 02:35:41 sop.gotoField =3D running; > > 02:38:02 boss exposure 4.0, file=3D296241, S > > 02:39:30 boss exposure 25.0, file=3D296242, S > > 02:40:59 sop.gotoField =3D failed; > > > > ### Due to the MCP issues, there were large fiducial errors, so we did > not have > > GSOGTF. By the time we identified the fiducials as the source of the > issue, it > > was too late to expose cart 2. > > ### Cartridge 5 > > > > > Dan, Elena, and Dylan, what is the version number you rolled back to? > > > > > > > 10:27:45 loadCart: Cart 3, Plate 10480; > > > 10:27:45 survey: APOGEE-2&MaNGA; lead: MaNGA 10min > > > 10:29:38 sop.gotoField =3D running; > > > 10:30:20 sop.gotoField =3D failed; > > > > ### When slewing to the field, alt got close to the target and behaved > like a > > damped oscillator. Az sped up faster than it should go, and would have > overshot, > > except a stop button was pressed quickly. After the stop button was > taken back > > out, GotoField completed successfully. > > > > 10:29:37 CmdStarted: OBSERVER.dgatlin 121 sop gotoField arcTime=3D= 4.0 > > flatTime=3D25.0 guiderFlatTime=3D0.5 guiderTime=3D5.0 > > > > 10:29:38 .tcc 0 tcc w text=3D'STATUS failed for axis: az' > > 10:29:38 .tcc 0 tcc w text=3D'STATUS failed for axis: alt' > > 10:29:38 .tcc 0 tcc w text=3D'STATUS failed for axis: rot1' > > > > ### Below is from right after the button was pushed > > > > 10:30:20 OBSERVER.dgatlin.sop 323 tcc f text=3D"Slew failed: 'TCC25mAct= or' > object > > has no attribute 'log'" > > 10:30:20 CmdDone: OBSERVER.dgatlin.sop 323 tcc track 55.910200, > 68.438900 icrs > > /rottype=3Dobject/rotang=3D0/rotwrap=3Dmid > > 10:30:20 OBSERVER.dgatlin 121 sop e text=3D"tcc track command failed wi= th > axis > > states: Halting,Halting,Halting and error codes: > > HaltRequested,HaltRequested,HaltRequested" > > 10:30:20 OBSERVER.dgatlin 121 sop e text=3D"Failed to complete slew: se= e > TCC > > messages for details." > > 10:30:20 OBSERVER.dgatlin 121 sop f text=3D"Failed to close screens, wa= rm > up > > lamps, and slew to field" > > 10:30:20 CmdDone: OBSERVER.dgatlin 121 sop gotoField arcTime=3D4.0 > > flatTime=3D25.0 guiderFlatTime=3D0.5 guiderTime=3D5.0 > > > > ### We believe this was the only slew where we went straight from field > to gang > > change, without first doing an axis stop and an axis init, and that thi= s > was the > > cause of the anomalous slew. > > I am a little confused here. Can you please give more information on the > slew sequence? > If we missed =E2=80=9Ctcc axis stop=E2=80=9D from the previous field=E2= =80=9D to the gangChange on > the previous > cartridge, we should have one when on gotoInstrument Change. Do you mean > from > the gangChange to the field? > > We had the issue on the gotoField on Cart 3. So we the sequence should b= e: > > tracking the previous field > take a APOGEE dome flat > tcc axis stop > goto gang change > goto instrument change > tcc axis stop > eSTOP, pin the telescope, swap cartridges > goto gangChange and plug > tcc axis stop > gotoField on Cart 3 > > Where do you think =E2=80=9Ctcc axis stop=E2=80=9D is missed? > Thanks, > > --0000000000003d96b605943f267e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable HI Kaike, Thanks. Looks= like the r1 electronic light contamination issue is back again. = best,-renbinOn Sun, Oct 6, 201= 9 at 10:44 AM Kaike Pan wrote: > On Oct 6, 2019, at 6:32 AM, dgatlin@apo.nmsu.edu wrote: > > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0APACHE POINT OBSERVATORY SDSS 2.5M > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0OBSERVING LOG >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 Saturday October 05, 2019 > >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 (SJD 58762) > > ----=C2=A0 Observing Team ---- > > Elena Malanushenko > Dan Oravetz > > Dylan Gatlin (training) > > ---- Observing Plan ---- > > > Sat=C2=A0 5 OCT 2019=C2=A0 =C2=A0 58762.0=C2=A0 =C2=A00.51=C2=A0 =C2= =A0 01:37=C2=A0 11:53=C2=A0 =C2=A0 19:31=C2=A0 05:50=C2=A0 =C2=A0 =C2=A0---= -=C2=A0 =C2=A0 > APOGEE-2 until 06:30 ( 4.9 h), then MaNGA ( 5.4 h) > > > ---- Observing Summary ---- > > It was a very clear night with excellent seeing. > > APOGEE lost 1.08h due to software related to MCP. > > ### Nighttime > > We had difficulties with MCP, TCC and semaphore. We called Jeff and we= re able to > resolve the issue. We restarted MCP, TCC, and alertsActor, and rolled = MCP back > to a previous version, but this version does not have a version number= . > Unfortunately, by the time we were back on field it was too > late for cart 2, so we moved on to cart 5. See cart 2 in log for detai= ls. > > After going to field for cart 2, we restarted the guider > > John Downey requested that we test the SOP slew scripts like Goto Gang= Change > and Goto Instrument Change, but since we rolled back the MCP, we decid= ed to not > do those test and continued to use tcc track commands instead. > > Cart 5, 11266, K19_076+07_btx, 2xDAB, 4xC, done > Missing fibers: [211, 273] > Faint fibers:=C2=A0 =C2=A0['122 - 123', '126 - 127', 1= 33, 137, 139, 147, 154] > > Cart 8, 11595, K2_C12_082-60_btx, 2xAB, done > Missing fibers: [35, 184, 211, 227, 273] > Faint fibers:=C2=A0 =C2=A0[123, 127, 148] > > Cart 9, 11424, NGC188_btx, 3xAB, done > Missing fibers: [36, 211, 245, 273] > Faint fibers:=C2=A0 =C2=A0[123, 127, 136] > > Cart 1, 8075, MJ359.55450+14.94322, 2xNSE, done > > Cart 4, 10224, MJ35.68307-0.00000, 2xNSE, 6xAB, not done > Missing fibers: [20, 39, 151, 186, 193, 211, 241, 273, 278] > Faint fibers:=C2=A0 =C2=A0[48, '121 - 128', 140, 142, 145, 243= , 284] > > Cart 3, 10480, MJ55.91016+68.43894, 2xNSE, 6xAB, not done > Missing fibers: [211, 273] > Faint fibers:=C2=A0 =C2=A0[123, '125 - 126'] > > ### Notes for developers and day staff > > We had another fast slew issue on GotoField, see Cart 3 log for detail= s. MaNGA team: here is a DOS warn on cart 5 ; Observers, please have the warns= in the Summary section. sdR-r1-00296247.fit.gz: WARNING: pixel bias rms for camera r1 is 4.79727x a= bove nominal limit- is there an electronics problem?? sdR-r1-00296247.fit.gz: WARNING: r1 blowtorch median=C2=A0 208.32152 (elect= ronics failure??) sdR-r1-00296248.fit.gz: WARNING: r1 blowtorch median=C2=A0 156.96501 (elect= ronics failure??) > > ### There was a problem with the MCP where it wouldn't send the se= maphore to the > TCC. As a result, all SOP commands failed, as well as axis init. When = we went to > 60,60,60, it failed, saying that semaphore owner was None. We called J= eff to > address the issue. We rebooted TCC which did not fix the issue, we reb= ooted MCP, > we rolled back MCP version. We also restarted the alerts actor. > > ### We believe that there might be some confusion on the version we ro= lled back > to, and it's possible we did not actually roll back to a previous = version. The > previous version that we have used was 1.1|2019-09-27|13:31. A few nig= hts ago we > updated to 1.1-14-g00bc|2019-10-03|11:22. The current version is > |2019-10-03|11:09 (ie. no version number, but it was from the same dat= e as > 1.1-14, with a slightly different timestamp. > > 01:17:59 OBSERVER.telescope 98 sop e text=3D"Cannot axis init: Se= maphore is owned > by None:0:0" > 01:17:59 OBSERVER.telescope 98 sop e text=3D"If you are the owner= (e.g., via MCP > Menu), release it and try again." > 01:17:59 OBSERVER.telescope 98 sop e text=3D"If you are not the o= wner, confirm > that you can steal it from them, then issue: mcp sem.steal" > 01:17:59 OBSERVER.telescope 98 sop f text=3D"Failed to slew to po= sition az=3D60, > alt=3D60, rot=3D60" > 01:17:59 CmdDone: OBSERVER.telescope 98 sop gotoAll60 > > ### mcp sem_give ran, but it didn't change anything. The same is t= rue with mcp > sem_Steal > > 01:30:50 CmdStarted: OBSERVER.telescope 108 mcp sem_Steal > 01:30:50 .mcp 0 mcp i mcpVersion=3D"1.1-14-g00bc|2019-10-03|11:22= " > 01:30:50 .mcp 0 mcp i haveSemaphore=3Dfalse > 01:30:50 .mcp 0 mcp i semaphoreOwner=3D"None:0:0" > 01:30:50 OBSERVER.telescope 108 mcp i haveSemaphore=3Dtrue > 01:30:50 OBSERVER.telescope 108 mcp i semaphoreOwner=3D"hub:hub:3= 8641" > 01:30:50 OBSERVER.telescope 108 mcp : command=3D"sem_steal" > 01:30:50 CmdDone: OBSERVER.telescope 108 mcp sem_Steal > > 01:39:21 CmdStarted: OBSERVER.elenam 54 mcp sem_give > 01:39:21 OBSERVER.elenam 54 mcp i haveSemaphore=3Dfalse > 01:39:21 OBSERVER.elenam 54 mcp i semaphoreOwner=3D"None:0:0"= ; > 01:39:21 OBSERVER.elenam 54 mcp : command=3D"sem_give" > 01:39:21 CmdDone: OBSERVER.elenam 54 mcp sem_give > > ### All SOP commands check the semaphore owner before running the comm= and. If > MCP doesn't hold the semaphore, then the TCC should grab it, and w= e should never > see the semaphore owner as None. For some reason, TCC wasn't grabb= ing the > semaphore. Jeff believes he found this bug, but we decided to stop deb= ugging as > soon as possible and go to normal observing by rolling back MCP versio= ns. Jeff > believes this issue arose from a discrepancy between the version he wa= s testing > with and the version we were using. > > ### When going to field, it failed because a guider flat was not taken= . The > guider flat was not taken because the guiderActor was hung. We kill -9= 'd a hung > guider process and restarted the actor. > > 02:28:53=C2=A0 boss exposure=C2=A0 =C2=A0 4.0, file=3D296237, S > 02:29:32=C2=A0 boss exposure=C2=A0 =C2=A0 4.0, file=3D296238, S > 02:30:49=C2=A0 sp1.motor.move=3D 519, 518, 518 > 02:30:52=C2=A0 sp2.motor.move=3D -787, -794, -786 > 02:30:52 Hartmann collimate output on cart #2 >=C2=A0 =C2=A0sp1: offset: r=3D 329 (In Focus);=C2=A0 b=3D -2.2 (In Focu= s) >=C2=A0 =C2=A0sp2: offset: r=3D -315 (In Focus);=C2=A0 b=3D 3.9 (In Focu= s) >=C2=A0 =C2=A0sp1: pred. move: spAverageMove=3D 518 >=C2=A0 =C2=A0sp2: pred. move: spAverageMove=3D -787 >=C2=A0 =C2=A0sp1: pred. resid: r=3D -189, b=3D -0.6, txt=3D OK, spTemp = =3D 7.9 >=C2=A0 =C2=A0sp2: pred. resid: r=3D 472, b=3D 1.5, txt=3D OK, spTemp = =3D 12.1 > 02:31:11=C2=A0 boss exposure=C2=A0 =C2=A0 4.0, file=3D296239, S > 02:32:38=C2=A0 boss exposure=C2=A0 =C2=A025.0, file=3D296240, S > 02:34:08=C2=A0 sop.gotoField =3D failed;=C2=A0 > 02:35:41=C2=A0 sop.gotoField =3D running;=C2=A0 > 02:38:02=C2=A0 boss exposure=C2=A0 =C2=A0 4.0, file=3D296241, S > 02:39:30=C2=A0 boss exposure=C2=A0 =C2=A025.0, file=3D296242, S > 02:40:59=C2=A0 sop.gotoField =3D failed;=C2=A0 > > ### Due to the MCP issues, there were large fiducial errors, so we did= not have > GSOGTF. By the time we identified the fiducials as the source of the i= ssue, it > was too late to expose cart 2. > ### Cartridge 5=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 > Dan, Elena, and Dylan,=C2=A0 what is the version number you rolled back to?= >=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > > 10:27:45=C2=A0 loadCart: Cart 3, Plate 10480;=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > 10:27:45 survey: APOGEE-2&MaNGA;=C2=A0 lead: MaNGA 10min=C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 > 10:29:38=C2=A0 sop.gotoField =3D running;=C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 > 10:30:20=C2=A0 sop.gotoField =3D failed; > > ### When slewing to the field, alt got close to the target and behaved= like a > damped oscillator. Az sped up faster than it should go, and would have= overshot, > except a stop button was pressed quickly. After the stop button was ta= ken back > out, GotoField completed successfully. > > 10:29:37 CmdStarted: OBSERVER.dgatlin 121 sop gotoField=C2=A0 =C2=A0 = =C2=A0 arcTime=3D4.0 > flatTime=3D25.0 guiderFlatTime=3D0.5 guiderTime=3D5.0 > > 10:29:38 .tcc 0 tcc w text=3D'STATUS failed for axis: az' > 10:29:38 .tcc 0 tcc w text=3D'STATUS failed for axis: alt' > 10:29:38 .tcc 0 tcc w text=3D'STATUS failed for axis: rot1' > > ### Below is from right after the button was pushed > > 10:30:20 OBSERVER.dgatlin.sop 323 tcc f text=3D"Slew failed: '= ;TCC25mActor' object > has no attribute 'log'" > 10:30:20 CmdDone: OBSERVER.dgatlin.sop 323 tcc track 55.910200, 68.438= 900 icrs > /rottype=3Dobject/rotang=3D0/rotwrap=3Dmid > 10:30:20 OBSERVER.dgatlin 121 sop e text=3D"tcc track command fai= led with axis > states: Halting,Halting,Halting and error codes: > HaltRequested,HaltRequested,HaltRequested" > 10:30:20 OBSERVER.dgatlin 121 sop e text=3D"Failed to complete sl= ew: see TCC > messages for details." > 10:30:20 OBSERVER.dgatlin 121 sop f text=3D"Failed to close scree= ns, warm up > lamps, and slew to field" > 10:30:20 CmdDone: OBSERVER.dgatlin 121 sop gotoField=C2=A0 =C2=A0 =C2= =A0 arcTime=3D4.0 > flatTime=3D25.0 guiderFlatTime=3D0.5 guiderTime=3D5.0 > > ### We believe this was the only slew where we went straight from fiel= d to gang > change, without first doing an axis stop and an axis init, and that th= is was the > cause of the anomalous slew. I am a little confused here. Can you please give more information on the sl= ew sequence? If we missed =E2=80=9Ctcc axis stop=E2=80=9D from the previous field=E2=80= =9D to the gangChange on the previous cartridge, we should have one when on gotoInstrument Change. Do you mean fr= om the gangChange to the field? We had the issue on the gotoField on Cart 3.=C2=A0 So we the sequence shoul= d be: tracking the previous field take a APOGEE dome flat tcc axis stop goto gang change goto instrument change tcc axis stop eSTOP, pin the telescope, swap cartridges goto gangChange and plug tcc axis stop gotoField on Cart 3 Where do you think =E2=80=9Ctcc axis stop=E2=80=9D is missed? Thanks, --0000000000003d96b605943f267e--