Subject: Re: [sloan-25mlog 4621] 2.5m obslog 20190906 (SJD 58733) From: Conor Sayres Date: Mon, 9 Sep 2019 15:31:37 -0700 Hi Guys, I=E2=80=99m looking into the M2 =E2=80=98hanging=E2=80=99 issue, it = looks pretty mysterious so far. Could you keep noting the times in the log that you notice it? Next time you see it, would you try a =E2=80=99tcc mirror init sec=E2=80=99= , to see if that fixes it before restarting the controller? That will = be a useful datapoint. Conor > On Sep 7, 2019, at 4:27 AM, asimmons@apo.nmsu.edu wrote: >=20 >=20 > APACHE POINT OBSERVATORY SDSS 2.5M >=20 > OBSERVING LOG > Friday September 06, 2019 >=20 > (SJD 58733) >=20 > ---- Observing Team ---- >=20 > Elena Malanushenko > Audrey Oravetz > Dylan Gatlin (training) > Kaike Pan (support) >=20 > ---- Observing Plan ---- >=20 > Fri 6 SEP 2019 58733.0 0.57 01:57 11:33 17:58 03:34 = ---- =20 > APOGEE-2 until 06:55 ( 5.0 h), then MaNGA ( 4.6 h)=20 >=20 > ---- Observing Summary ---- >=20 > Clear night. Good seeing.=20 >=20 > Cart 9, AP plate 11268, K19_076+07_btx; 2XDAB > Missing fibers: [36, 211, 245, 273] > Faint fibers: [123, '126 - 127'] > ----------------- >=20 > Cart 8, Ap plate 11463, NGC6791_btx; 4XAB > Missing fibers: [35, 184, 211, 227, 273] > Faint fibers: [123, 127] >=20 > Cart 6, Ap plate 11421, N6819-RV_btx; 4XAB > Missing fibers: [181, 211, 273] > Faint fibers: [15, 34, 105, 121, 123, 125, 127, 289] >=20 > Cart 3, Ap plate 11466, NGC7789_btx; 4XAB > Missing fibers: [211, 273] > Faint fibers: [123, 125, 127, '133 - 134'] >=20 > Cart 7, Ap plate 11213, 135-24_btx; 4XAB, done > Missing fibers: [211, 223, 273] > Faint fibers: [123, 127, 148, 173, 202] >=20 > Cart 1, Ap plate 11602, W34_btx; 2XAB > Missing fibers: [95, 211, 273] > Faint fibers: ['1 - 30', 148, '151 - 180', 227] >=20 >=20 > Notes: > =3D=3D=3D=3D=3D=3D >=20 > We operated the crane for cart changes.=20 >=20 > APOGEE lost 38 minutes due to a software disconnect. One APOGEE = exposure hung > mid-way through integration, which severed the connection with any = apogee or > apogeecal actor. Kaike helped remotely by restarting the apogee-ics = machine. >=20 > Same error as yesterday with gotoGangChange: 'INIT'; STATUS: Cancelled = on queue >=20 > 05:54Z - M2 hung once and we followed Conor's steps to restart the = mirror > controller.=20 >=20 > Cart 1 seems to have many faint fibers, please check. >=20 > 1.1 hrs APOGEE lost due to weather (high humidity) >=20 >=20 > ---- Observing Log ---- >=20 > Afternoon > ---------- > Telescope has been checked, all system are working as expected.=20 > eBoss cameras are not mounted on telescope.=20 > Cartridge buy manipulator is not operated.=20 > gcamera temperature was low, set it to -35 degrees.=20 > Mirror numbers are not available.=20 > Apogee evening calibrations has been done.=20 > Telescope has been checked, movement, fiducial reading, all ok.=20 > 01:00 started ventilation.=20 >=20 > Night > =3D=3D=3D=3D=3D=3D >=20 > 02:08:58 encl25m : open; =20 >=20 > **************** >=20 > 01:44:44 loadCart: Cart 9, Plate 11268; K19_076+07_btx > 02:13:42 sop.gotoField =3D running; =20 > 02:20:59 apogee.expose=3DapRaw-31710010, 94, A, Object > 02:37:41 apogee.expose=3DapRaw-31710011, 94, B, Object > This exposure hung on 81 reads of 93, and with this STUI message, we = were unable > to start the next exposure sequence. The apogee actor/ics machine = failed to > respond to any restart/initialization. We do not have permission to = access > apogee-ics. We called Kaike. We was able to log into the apogee-ics = machine to > restart the process. Afterwards, hub startNubs worked and the actors = began to > respond again.=20 >=20 > 02:54:02 .tcc 0 tcc w Text=3D"Collimation Failure: Cancelled while = queued by new > command '949 move -835.9427,8.7039,10.8271,-40.2729,-133.1170'" > 02:54:09 .boss 3893 boss w text=3D"camcheck found missing or = unresponsive cameras: > set(['sp1cam'])" > 02:54:16 .hub 0 cmds d > = CmdQueued=3D1299083,1567824855.82,"OBSERVER.elenam.guider",376,"gcamera",3= 343,"expose > time=3D5.0 stack=3D3" > 02:54:24 OBSERVER.elenam 57 sop e text=3D"failed to start object = exposure" > 02:54:24 OBSERVER.elenam 57 sop f text=3D"Failed to take an object = exposure" > 02:54:24 CmdDone: OBSERVER.elenam 57 sop doApogeeScience = ditherPairs=3D2=20 > expTime=3D\"1000.0\" >=20 > 03:16:42 apogee.expose=3DapRaw-31710012, 3, A, Dark. After a reset of = the machine.=20 > 03:21:48 apogee.expose=3DapRaw-31710013, 94, B, Object > 03:54 apogee.expose=3DapRaw-31710014, 94, A, Object > 04:03:41 apogee.expose=3DapRaw-31710015, 94, B, Object > 04:14:06 apogee.expose=3DapRaw-31710016, 94, A, Object > 04:15:27 apogee.expose=3DapRaw-31710017, 5, A, DomeFlat >=20 > Missing fibers: [36, 211, 245, 273] > Faint fibers: [123, '126 - 127'] >=20 > **************** >=20 > Cart 8, Ap plate 11463, NGC6791_btx >=20 > 04:29:14 apogee.expose=3DapRaw-31710020, 47, A, Object > 04:37:38 apogee.expose=3DapRaw-31710021, 47, B, Object > 04:45:58 apogee.expose=3DapRaw-31710022, 47, B, Object > 04:54:25 apogee.expose=3DapRaw-31710023, 47, A, Object > 05:02:45 apogee.expose=3DapRaw-31710024, 47, A, Object > 05:11:08 apogee.expose=3DapRaw-31710025, 47, B, Object > 05:19:28 apogee.expose=3DapRaw-31710026, 47, B, Object > 05:27:56 apogee.expose=3DapRaw-31710027, 47, A, Object > 05:36:48 apogee.expose=3DapRaw-31710028, 5, A, DomeFlat > 05:40:58 apogee.expose=3DapRaw-31710029, 10, A, Dark > 05:42:45 apogee.expose=3DapRaw-31710030, 10, A, Dark >=20 > Missing fibers: [35, 184, 211, 227, 273] > Faint fibers: [123, 127] >=20 > *************** >=20 > Cart 6, Ap plate 11421, N6819-RV_btx >=20 > 06:11:50 apogee.expose=3DapRaw-31710031, 47, A, Object > 06:20:14 apogee.expose=3DapRaw-31710032, 47, B, Object > 06:28:33 apogee.expose=3DapRaw-31710033, 47, B, Object > 06:37:01 apogee.expose=3DapRaw-31710034, 47, A, Object > 06:45:21 apogee.expose=3DapRaw-31710035, 47, A, Object > 06:53:44 apogee.expose=3DapRaw-31710036, 47, B, Object > 07:02:04 apogee.expose=3DapRaw-31710037, 47, B, Object > 07:10:32 apogee.expose=3DapRaw-31710038, 47, A, Object > 07:19:29 apogee.expose=3DapRaw-31710039, 5, A, DomeFlat > 07:23:36 apogee.expose=3DapRaw-31710040, 10, A, Dark > 07:25:24 apogee.expose=3DapRaw-31710041, 10, A, Dark >=20 > Missing fibers: [181, 211, 273] > Faint fibers: [15, 34, 105, 121, 123, 125, 127, 289] >=20 > ************** >=20 >=20 > Cart 3, Ap plate 11466, NGC7789_btx >=20 > 07:37:06 apogee.expose=3DapRaw-31710042, 47, A, Object > 07:45:29 apogee.expose=3DapRaw-31710043, 47, B, Object > 07:53:49 apogee.expose=3DapRaw-31710044, 47, B, Object > 08:02:16 apogee.expose=3DapRaw-31710045, 47, A, Object > 08:10:36 apogee.expose=3DapRaw-31710046, 47, A, Object > 08:18:59 apogee.expose=3DapRaw-31710047, 47, B, Object > 08:27:19 apogee.expose=3DapRaw-31710048, 47, B, Object > 08:35:47 apogee.expose=3DapRaw-31710049, 47, A, Object > 08:44:35 apogee.expose=3DapRaw-31710050, 5, A, DomeFlat > 08:48:18 apogee.expose=3DapRaw-31710051, 10, A, Dark > 08:50:06 apogee.expose=3DapRaw-31710052, 10, A, Dark >=20 > Missing fibers: [211, 273] > Faint fibers: [123, 125, 127, '133 - 134'] >=20 > *************** >=20 > Cart 7, Ap plate 11213, 135-24_btx >=20 > 09:05:01 apogee.expose=3DapRaw-31710053, 47, A, Object > 09:13:25 apogee.expose=3DapRaw-31710054, 47, B, Object > 09:21:45 apogee.expose=3DapRaw-31710055, 47, B, Object > 09:30:13 apogee.expose=3DapRaw-31710056, 47, A, Object > 09:38:32 apogee.expose=3DapRaw-31710057, 47, A, Object > 09:46:56 apogee.expose=3DapRaw-31710058, 47, B, Object > 09:55:16 apogee.expose=3DapRaw-31710059, 47, B, Object > 10:03:43 apogee.expose=3DapRaw-31710060, 47, A, Object > 10:12:23 apogee.expose=3DapRaw-31710061, 5, A, DomeFlat > 10:15:06 apogee.expose=3DapRaw-31710062, 10, A, Dark > 10:16:54 apogee.expose=3DapRaw-31710063, 10, A, Dark > 10:20:45 apogee.expose=3DapRaw-31710064, 5, A, DomeFlat -- Please = ignore >=20 > Missing fibers: [211, 223, 273] > Faint fibers: [123, 127, 148, 173, 202] >=20 > ************** >=20 > Cart 1, Ap plate 11602, W34_btx >=20 > 10:27:55 apogee.expose=3DapRaw-31710065, 47, A, Object > 10:36:19 apogee.expose=3DapRaw-31710066, 47, B, Object > 10:44:38 apogee.expose=3DapRaw-31710067, 47, B, Object > 10:53:06 apogee.expose=3DapRaw-31710068, 47, A, Object > 11:01:42 apogee.expose=3DapRaw-31710069, 5, A, DomeFlat > 11:12:21 apogee.expose=3DapRaw-31710070, 60, A, Dark >=20 > Missing fibers: [95, 211, 273] > Faint fibers: ['1 - 30', 148, '151 - 180', 227] >=20 >=20 > 11:22Z Closed enclosure, stopped ventilating, closed early due to high = humidity, > started APOGEE morning cals. >=20 >=20 > ---- Mirror Numbers ---- >=20 >=20 > PRIMARY: > -------- > Scale: 1.000000; >=20 > MIGS TONIGHT NOMINAL > Axial A: tpm_MIG1A --- Invalid channel name tpm_MIG1A --- = Invalid channel name > Axial B: tpm_MIG1B --- Invalid channel name tpm_MIG1B --- = Invalid channel name > Axial C: tpm_MIG1C --- Invalid channel name tpm_MIG1C --- = Invalid channel name > Trans D: tpm_MIG1DT --- Invalid channel name tpm_MIG1DT --- = Invalid channel name > Lateral E: tpm_MIG1EL --- Invalid channel name 9.3345 > Lateral F: tpm_MIG1FL --- Invalid channel name 14.3637 >=20 > GALILS > Commanded: 11200 -1850 6850 29650 21250 21050 > Actual: 11200 -1850 6850 29650 21250 21050 >=20 > SETMIR VALUES > PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 > PrimOrient: 0.0 -11.4 6.58 375.38 620.77 >=20 > SECONDARY: > ---------- >=20 > Focus: 0.0000 > Air temp: 17.3 > Altitude: 30.0 >=20 > MIGS TONIGHT NOMINAL > Axial A: -6.665 -6.668 > Axial B: -0.036 tpm_MIG2B --- Invalid channel name > Axial C: -6.67 tpm_MIG2C --- Invalid channel name > Trans D: -0.08 -0.074 >=20 > GALILS > Commanded: 1058750 1101250 1079500 -7227 -6550 > Actual: 1058687 1101628 1080123 -7250 -6550 >=20 > SETMIR VALUES > SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 > SecOrient: 857.14 2.61 12.85 -38.93 157.36 >=20 >=20 >=20 > PRIMARY: > -------- > Scale: 1.000000; >=20 > MIGS TONIGHT NOMINAL > Axial A: -5.33 tpm_MIG1A --- Invalid channel name > Axial B: -5.052 tpm_MIG1B --- Invalid channel name > Axial C: tpm_MIG1C --- Invalid channel name tpm_MIG1C --- = Invalid channel name > Trans D: -0.576 tpm_MIG1DT --- Invalid channel name > Lateral E: 9.3472 9.3345 > Lateral F: 14.4399 14.3637 >=20 > GALILS > Commanded: 11200 -1850 6850 29650 21250 21050 > Actual: 11200 -1850 6850 29650 21250 21050 >=20 > SETMIR VALUES > PrimDesOrient: 0.0 -11.37 6.87 375.47 621.12 > PrimOrient: 0.0 -11.4 6.58 375.38 620.77 >=20 > SECONDARY: > ---------- >=20 > Focus: 0.0000 > Air temp: 17.2 > Altitude: 30.0 >=20 > MIGS TONIGHT NOMINAL > Axial A: -6.665 -6.668 > Axial B: -0.036 tpm_MIG2B --- Invalid channel name > Axial C: -6.67 tpm_MIG2C --- Invalid channel name > Trans D: -0.08 -0.074 >=20 > GALILS > Commanded: 1058750 1101400 1079700 -7250 -6550 > Actual: 1058762 1101477 1079814 -7250 -6550 >=20 > SETMIR VALUES > SecDesOrient: 857.0 2.7 12.75 -38.35 156.51 > SecOrient: 857.06 2.66 12.78 -38.41 156.75 >=20 >=20 >=20 > ---- Telescope Offsets and Scale ---- >=20 >=20 > --- Offsets --- (arcsec)=20 > Time Inst Az Alt Rot objArcOff guideRot calibOff = =20 > = --------------------------------------------------------------------------= ------ > 04:03 9-11268A 194.2 81.5 15.6 ( -4.1, -8.0) (-371.3) ( 0.0, = 0.0, 0.0) apog=20 > 04:03 9-11268A 194.3 81.4 15.8 ( -4.1, -8.0) (-371.3) ( 0.0, = 0.0, 0.0) =20 > 04:13 9-11268A 205.9 80.7 28.9 ( -1.4, -7.3) (-372.0) ( 0.0, = 0.0, 0.0) apog=20 > 04:28 8-11463A 246.4 75.6 75.8 ( 2.3, -3.9) (-285.0) ( 0.0, = 0.0, 0.0) apog=20 > 04:37 8-11463A 247.8 74.0 78.5 ( 2.7, -3.6) (-287.5) ( 0.0, = 0.0, 0.0) apog=20 > 04:45 8-11463A 248.8 72.3 80.7 ( 2.1, -3.2) (-281.7) ( 0.0, = 0.0, 0.0) apog=20 > 04:53 8-11463A 249.5 70.7 82.7 ( 2.2, -2.6) (-280.8) ( 0.0, = 0.0, 0.0) apog=20 > 05:02 8-11463A 250.0 69.0 84.4 ( 3.2, -1.9) (-282.6) ( 0.0, = 0.0, 0.0) apog=20 > 05:10 8-11463A 250.4 67.4 86.1 ( 3.1, -1.8) (-281.9) ( 0.0, = 0.0, 0.0) apog=20 > 05:18 8-11463A 250.6 65.7 87.5 ( 3.0, -1.5) (-281.1) ( 0.0, = 0.0, 0.0) apog=20 > 05:27 8-11463A 250.7 64.1 88.9 ( 3.1, -1.1) (-282.4) ( 0.0, = 0.0, 0.0) apog=20 > 06:11 6-11421A 246.8 58.9 89.6 ( 4.8, -1.9) (-274.9) ( 0.0, = 0.0, 0.0) apog=20 > 06:19 6-11421A 246.8 57.3 90.9 ( 3.4, -1.8) (-280.3) ( 0.0, = 0.0, 0.0) apog=20 > 06:27 6-11421A 246.7 55.6 92.2 ( 3.6, -1.6) (-278.9) ( 0.0, = 0.0, 0.0) apog=20 > 06:36 6-11421A 246.6 54.0 93.4 ( 3.7, -1.6) (-276.9) ( 0.0, = 0.0, 0.0) apog=20 > 06:44 6-11421A 246.4 52.4 94.6 ( 3.7, -1.6) (-278.5) ( 0.0, = 0.0, 0.0) apog=20 > 06:53 6-11421A 246.1 50.8 95.7 ( 4.0, -1.7) (-277.7) ( 0.0, = 0.0, 0.0) apog=20 > 07:01 6-11421A 245.9 49.2 96.8 ( 3.9, -1.8) (-277.9) ( 0.0, = 0.0, 0.0) apog=20 > 07:09 6-11421A 245.5 47.5 97.9 ( 4.1, -1.6) (-279.9) ( 0.0, = 0.0, 0.0) apog=20 > 07:36 3-11466A 173.0 65.7 -10.8 ( -7.1, -6.4) (-381.3) ( 0.0, = 0.0, 0.0) apog=20 > 07:44 3-11466A 175.8 65.9 -6.5 ( -6.9, -6.8) (-353.1) ( 0.0, = 0.0, 0.0) apog=20 > 07:53 3-11466A 178.6 66.0 -2.2 ( -6.2, -7.2) (-349.6) ( 0.0, = 0.0, 0.0) apog=20 > 08:01 3-11466A 181.4 66.0 2.1 ( -5.6, -7.5) (-348.0) ( 0.0, = 0.0, 0.0) apog=20 > 08:09 3-11466A 184.2 65.9 6.4 ( -4.9, -7.3) (-349.9) ( 0.0, = 0.0, 0.0) apog=20 > 08:18 3-11466A 187.0 65.7 10.7 ( -4.3, -7.7) (-349.5) ( 0.0, = 0.0, 0.0) apog=20 > 08:26 3-11466A 189.7 65.5 14.9 ( -3.8, -7.6) (-343.0) ( 0.0, = 0.0, 0.0) apog=20 > 08:35 3-11466A 192.3 65.1 19.1 ( -3.1, -7.5) (-340.7) ( 0.0, = 0.0, 0.0) apog=20 > 09:04 7-11213A 117.8 80.5 -68.3 ( -3.6, -3.4) (-338.4) ( 0.0, = 0.0, 0.0) apog=20 > 09:12 7-11213A 122.4 82.1 -62.4 ( -2.8, -3.4) (-326.7) ( 0.0, = 0.0, 0.0) apog=20 > 09:21 7-11213A 129.4 83.5 -54.1 ( -1.6, -3.4) (-327.3) ( 0.0, = 0.0, 0.0) apog=20 > 09:29 7-11213A 140.5 84.8 -41.9 ( 0.0, -3.0) (-327.5) ( 0.0, = 0.0, 0.0) apog=20 > 09:37 7-11213A 157.5 85.7 -23.7 ( 1.7, -1.5) (-324.6) ( 0.0, = 0.0, 0.0) apog=20 > 09:46 7-11213A 180.5 86.0 0.5 ( 2.5, 0.4) (-316.2) ( 0.0, = 0.0, 0.0) apog=20 > 09:54 7-11213A 203.1 85.7 24.3 ( 3.2, 2.3) (-312.7) ( 0.0, = 0.0, 0.0) apog=20 > 10:03 7-11213A 220.0 84.7 42.4 ( 3.1, 4.2) (-313.3) ( 0.0, = 0.0, 0.0) apog=20 > 10:27 1-11602A 179.6 61.3 -0.6 ( -7.6, -7.2) (-352.1) ( 0.0, = 0.0, 0.0) apog=20 > 10:35 1-11602A 181.7 61.3 3.1 ( -7.5, -8.3) (-340.5) ( 0.0, = 0.0, 0.0) apog=20 > 10:44 1-11602A 183.8 61.2 6.8 ( -7.1, -8.7) (-342.8) ( 0.0, = 0.0, 0.0) apog=20 > 10:52 1-11602A 185.9 61.1 10.4 ( -6.6, -8.9) (-343.3) ( 0.0, = 0.0, 0.0) apog=20 >=20 >=20 > ---- Focus Log ---- >=20 >=20 > --- Focus --- > Time Inst Scale M1 M2 Focus Az Alt Temp Wind Dir = > = --------------------------------------------------------------------------= ------ > 04:03 9-11268A 1.000170 -1678 -790 -115 194.2 81.5 15.2 14 113 = nan apog > 04:03 9-11268A 1.000170 -1678 -788 -115 194.3 81.4 15.2 14 113 = nan=20 > 04:13 9-11268A 1.000170 -1678 -736 -65 205.9 80.7 14.9 14 112 = nan apog > 04:28 8-11463A 1.000170 -1678 -682 -15 246.4 75.6 14.4 15 108 = nan apog > 04:37 8-11463A 1.000170 -1678 -662 2 247.8 74.0 14.6 18 109 = nan apog > 04:45 8-11463A 1.000170 -1678 -651 10 248.8 72.3 14.5 15 112 = nan apog > 04:53 8-11463A 1.000170 -1678 -637 22 249.5 70.7 14.3 16 112 = nan apog > 05:02 8-11463A 1.000200 -1975 -871 30 250.0 69.0 14.1 17 121 = nan apog > 05:10 8-11463A 1.000200 -1975 -886 11 250.4 67.4 14.2 17 115 = nan apog > 05:18 8-11463A 1.000200 -1975 -908 -12 250.6 65.7 14.6 15 125 = nan apog > 05:27 8-11463A 1.000200 -1975 -911 -40 250.7 64.1 14.4 13 116 = nan apog > 06:11 6-11421A 1.000218 -2153 -1142 -115 246.8 58.9 14.2 17 123 = nan apog > 06:19 6-11421A 1.000128 -1264 -402 -115 246.8 57.3 14.2 14 124 = nan apog > 06:27 6-11421A 1.000128 -1264 -402 -119 246.7 55.6 14.1 12 112 = nan apog > 06:36 6-11421A 1.000128 -1264 -400 -122 246.6 54.0 14.0 17 126 = nan apog > 06:44 6-11421A 1.000128 -1264 -395 -122 246.4 52.4 14.0 16 123 = nan apog > 06:53 6-11421A 1.000128 -1264 -392 -124 246.1 50.8 13.9 18 124 = nan apog > 07:01 6-11421A 1.000128 -1264 -389 -129 245.9 49.2 13.9 11 123 = nan apog > 07:09 6-11421A 1.000158 -1264 -392 -135 245.5 47.5 13.8 17 127 = nan apog > 07:36 3-11466A 1.000158 -1560 -706 -153 173.0 65.7 13.8 15 128 = nan apog > 07:44 3-11466A 1.000158 -1560 -717 -165 175.8 65.9 13.9 8 111 = nan apog > 07:53 3-11466A 1.000158 -1560 -715 -162 178.6 66.0 13.9 12 126 = nan apog > 08:01 3-11466A 1.000158 -1560 -724 -172 181.4 66.0 13.9 15 132 = nan apog > 08:09 3-11466A 1.000158 -1560 -736 -184 184.2 65.9 13.9 11 134 = nan apog > 08:18 3-11466A 1.000158 -1560 -742 -190 187.0 65.7 13.9 14 132 = nan apog > 08:26 3-11466A 1.000158 -1560 -745 -191 189.7 65.5 13.8 11 126 = nan apog > 08:35 3-11466A 1.000158 -1560 -737 -187 192.3 65.1 13.7 8 119 = nan apog > 09:04 7-11213A 1.000158 -1560 -765 -191 117.8 80.5 13.3 6 142 = nan apog > 09:12 7-11213A 1.000188 -1857 -1015 -196 122.4 82.1 13.4 7 114 = nan apog > 09:21 7-11213A 1.000188 -1857 -1012 -192 129.4 83.5 13.5 14 138 = nan apog > 09:29 7-11213A 1.000188 -1857 -1020 -199 140.5 84.8 13.5 15 134 = nan apog > 09:37 7-11213A 1.000188 -1857 -1025 -203 157.5 85.7 13.4 14 135 = nan apog > 09:46 7-11213A 1.000188 -1857 -1028 -206 180.5 86.0 13.5 14 135 = nan apog > 09:54 7-11213A 1.000218 -2153 -1265 -198 203.1 85.7 13.3 13 137 = nan apog > 10:03 7-11213A 1.000218 -2153 -1251 -184 220.0 84.7 13.2 10 134 = nan apog > 10:27 1-11602A 1.000218 -2153 -1215 -182 179.6 61.3 12.8 9 152 = nan apog > 10:35 1-11602A 1.000218 -2153 -1232 -199 181.7 61.3 12.8 6 166 = nan apog > 10:44 1-11602A 1.000218 -2153 -1247 -214 183.8 61.2 12.6 5 179 = nan apog > 10:52 1-11602A 1.000218 -2153 -1242 -207 185.9 61.1 12.4 3 134 = nan apog >=20 >=20 > ---- Telescope Status ---- >=20 > 00:30Z Started ventilation > 01:30Z encl25m : open > 11:22Z encl25m : closed >=20 > Status at:11:23Z > Telescope stowed at: 121.0, 30.0, 0.0 mount > Instrument mounted: cartridge =3D 1 > Counterweights at: 270 > LN2 autofill systems: Connected and turned on >=20 >=20 >=20 >=20 > ---- Software Used ---- >=20 > Get Versions, 09/07/19, 00:56:53 > -- STUI software:=20 > STUI : 1.7.8 > alerts : trunk > apo : trunk > boss : v4_0 > -boss.specMec : 3.0.4 > gcamera : v2_2 > guider : 3.8.1 > hartmann : 1.6.0 > hub : Tag: v4.2 > mcp : Linux > -mcp.plc : 34 > -mcp.azFiduci : v1_140 > -mcp.altFiduc : v1_140 > -mcp.rotFiduc : v1_140 > platedb : 2.8.0 > sop : 3.12.3 > tcc : 1.3.1 > apogee : not availble > apogeecal : not availble > apogeeql : v1_6_1 > APO-local : not availble > -- Other versions ---- > idlspec2d : v5_7_12 > plate-mapper3 : NOCVS:v6_0_12 > petunia : 1.4.2 > autoscheduler : 1.3.0 > sdss_python_module: trunk > mangadrp : trunk exported >=20 > ---- APOGEE Data Summary ---- >=20 > APOGEE data list, mjd=3D58733, all > raw_data: /data/apogee/utr_cdr/58733/apRaw-*.fits > archive: /data/apogee/archive/58733/apR-[a,b,c]-*.apz > quickred: /data/apogee/quickred/58733/ap2D-a-*.fits.fz > = --------------------------------------------------------------------------= ------=20 > mjd i UT File/Exp Imtype Nread A/B Dth Ct-Plate Arch Offset = %Flux Comm=20 > = --------------------------------------------------------------------------= ------ > 58733 1 23:04 31710001 Dark 60 A 13.0 6-10351 a-b-c - = - =20 > 58733 2 23:15 31710002 QuarFlat 10 A 13.0 6-10351 a-b-c - = 50=20 > 58733 3 23:17 31710003 Arc-Thar 12 A 13.0 6-10351 a-b-c -3.30 = - =20 > 58733 4 23:19 31710004 Arc-Une 40 A 13.0 6-10351 a-b-c - = - =20 > 58733 5 23:26 31710005 Arc-Thar 12 B 13.5 6-10351 a-b-c -2.80 = - =20 > 58733 6 23:28 31710006 Arc-Une 40 B 13.5 6-10351 a-b-c - = - =20 > 58733 7 23:36 31710007 Dark 60 A 13.0 6-10351 a-b-c - = - =20 > 58733 8 02:08 31710008 Dark 10 A 13.0 9-11268 a-b-c - = - =20 > 58733 9 02:09 31710009 Dark 10 A 13.0 9-11268 a-b-c - = - =20 > 58733 10 02:20 31710010 Object 94 A 13.0 9-11268 a-b-c - = - =20 > 58733 11 02:51 31710011 Object 94 B 13.5 ------- x-x-x - = - =20 > 58733 12 03:16 31710012 Dark 3 A 13.0 9-11268 a-b-c - = - =20 > 58733 13 03:21 31710013 Object 94 B 13.5 9-11268 a-b-c - = - =20 > 58733 14 03:38 31710014 Object 94 A 13.0 9-11268 a-b-c - = - =20 > 58733 15 03:57 31710015 Object 94 B 13.5 9-11268 a-b-c - = - =20 > 58733 16 04:13 31710016 Object 94 A 13.0 9-11268 a-b-c - = - =20 > 58733 17 04:15 31710017 DomeFlat 5 A 13.0 9-11268 a-b-c - = 91=20 > 58733 18 04:18 31710018 Dark 10 A 13.0 9-11268 a-b-c - = - =20 > 58733 19 04:20 31710019 Dark 10 A 13.0 9-11268 a-b-c - = - =20 > 58733 20 04:29 31710020 Object 47 A 13.0 8-11463 a-b-c - = - =20 > 58733 21 04:37 31710021 Object 47 B 13.5 8-11463 a-b-c - = - =20 > 58733 22 04:45 31710022 Object 47 B 13.5 8-11463 a-b-c - = - =20 > 58733 23 04:54 31710023 Object 47 A 13.0 8-11463 a-b-c - = - =20 > 58733 24 05:02 31710024 Object 47 A 13.0 8-11463 a-b-c - = - =20 > 58733 25 05:10 31710025 Object 47 B 13.5 8-11463 a-b-c - = - =20 > 58733 26 05:19 31710026 Object 47 B 13.5 8-11463 a-b-c - = - =20 > 58733 27 05:27 31710027 Object 47 A 13.0 8-11463 a-b-c - = - =20 > 58733 28 05:36 31710028 DomeFlat 5 A 13.0 8-11463 a-b-c - = 80=20 > 58733 29 05:40 31710029 Dark 10 A 13.0 8-11463 a-b-c - = - =20 > 58733 30 05:42 31710030 Dark 10 A 13.0 8-11463 a-b-c - = - =20 > 58733 31 06:11 31710031 Object 47 A 13.0 6-11421 a-b-c - = - =20 > 58733 32 06:20 31710032 Object 47 B 13.5 6-11421 a-b-c - = - =20 > 58733 33 06:28 31710033 Object 47 B 13.5 6-11421 a-b-c - = - =20 > 58733 34 06:36 31710034 Object 47 A 13.0 6-11421 a-b-c - = - =20 > 58733 35 06:45 31710035 Object 47 A 13.0 6-11421 a-b-c - = - =20 > 58733 36 06:53 31710036 Object 47 B 13.5 6-11421 a-b-c - = - =20 > 58733 37 07:01 31710037 Object 47 B 13.5 6-11421 a-b-c - = - =20 > 58733 38 07:10 31710038 Object 47 A 13.0 6-11421 a-b-c - = - =20 > 58733 39 07:19 31710039 DomeFlat 5 A 13.0 6-11421 a-b-c - = 77=20 > 58733 40 07:23 31710040 Dark 10 A 13.0 6-11421 a-b-c - = - =20 > 58733 41 07:25 31710041 Dark 10 A 13.0 6-11421 a-b-c - = - =20 > 58733 42 07:36 31710042 Object 47 A 13.0 3-11466 a-b-c - = - =20 > 58733 43 07:45 31710043 Object 47 B 13.5 3-11466 a-b-c - = - =20 > 58733 44 07:53 31710044 Object 47 B 13.5 3-11466 a-b-c - = - =20 > 58733 45 08:02 31710045 Object 47 A 13.0 3-11466 a-b-c - = - =20 > 58733 46 08:10 31710046 Object 47 A 13.0 3-11466 a-b-c - = - =20 > 58733 47 08:18 31710047 Object 47 B 13.5 3-11466 a-b-c - = - =20 > 58733 48 08:27 31710048 Object 47 B 13.5 3-11466 a-b-c - = - =20 > 58733 49 08:35 31710049 Object 47 A 13.0 3-11466 a-b-c - = - =20 > 58733 50 08:44 31710050 DomeFlat 5 A 13.0 3-11466 a-b-c - = 88=20 > 58733 51 08:48 31710051 Dark 10 A 13.0 3-11466 a-b-c - = - =20 > 58733 52 08:49 31710052 Dark 10 A 13.0 3-11466 a-b-c - = - =20 > 58733 53 09:04 31710053 Object 47 A 13.0 7-11213 a-b-c - = - =20 > 58733 54 09:13 31710054 Object 47 B 13.5 7-11213 a-b-c - = - =20 > 58733 55 09:21 31710055 Object 47 B 13.5 7-11213 a-b-c - = - =20 > 58733 56 09:30 31710056 Object 47 A 13.0 7-11213 a-b-c - = - =20 > 58733 57 09:38 31710057 Object 47 A 13.0 7-11213 a-b-c - = - =20 > 58733 58 09:46 31710058 Object 47 B 13.5 7-11213 a-b-c - = - =20 > 58733 59 09:55 31710059 Object 47 B 13.5 7-11213 a-b-c - = - =20 > 58733 60 10:03 31710060 Object 47 A 13.0 7-11213 a-b-c - = - =20 > 58733 61 10:12 31710061 DomeFlat 5 A 13.0 7-11213 a-b-c - = 80=20 > 58733 62 10:14 31710062 Dark 10 A 13.0 7-11213 a-b-c - = - =20 > 58733 63 10:16 31710063 Dark 10 A 13.0 7-11213 a-b-c - = - =20 > 58733 64 10:20 31710064 DomeFlat 5 A 13.0 7-11213 a-b-c - = 0=20 > 58733 65 10:27 31710065 Object 47 A 13.0 1-11602 a-b-c - = - =20 > 58733 66 10:36 31710066 Object 47 B 13.5 1-11602 a-b-c - = - =20 > 58733 67 10:44 31710067 Object 47 B 13.5 1-11602 a-b-c - = - =20 > 58733 68 10:52 31710068 Object 47 A 13.0 1-11602 a-b-c - = - =20 > 58733 69 11:01 31710069 DomeFlat 5 A 13.0 1-11602 a-b-c - = 64=20 > 58733 70 11:22 31710070 Dark 60 A 13.0 ------- x-x-x - = - =20 > 58733 71 11:25 31710071 Dark 60 A 13.0 ------- x-x-x - = - =20 > = --------------------------------------------------------------------------= ------=20 >=20 >=20 >=20 > ---- Weather Log ---- >=20 > --- Weather --- > Time Inst Temp DP Dif Humid Wind Dir Dust,1um IRSC IRSCm = > = --------------------------------------------------------------------------= ------ > 04:03 9-11268A 15.2 9.3 5.9 67 14 113 347 3.3 332 = nan apog=20 > 04:03 9-11268A 15.2 9.3 5.9 67 14 113 347 3.3 332 = nan =20 > 04:13 9-11268A 14.9 9.9 5.0 71 14 112 486 2.6 331 = nan apog=20 > 04:28 8-11463A 14.4 9.9 4.5 74 15 108 477 2.8 332 = nan apog=20 > 04:37 8-11463A 14.6 10.0 4.6 73 18 109 487 2.4 332 = nan apog=20 > 04:45 8-11463A 14.5 9.9 4.6 74 15 112 479 2.8 331 = nan apog=20 > 04:53 8-11463A 14.3 9.9 4.4 75 16 112 468 3.6 322 = nan apog=20 > 05:02 8-11463A 14.1 9.9 4.2 75 17 121 445 2.1 327 = nan apog=20 > 05:10 8-11463A 14.2 9.9 4.3 75 17 115 437 2.8 332 = nan apog=20 > 05:18 8-11463A 14.6 9.9 4.7 73 15 125 428 2.6 332 = nan apog=20 > 05:27 8-11463A 14.4 9.9 4.5 74 13 116 432 2.3 332 = nan apog=20 > 06:11 6-11421A 14.2 10.0 4.2 75 17 123 463 3.2 332 = nan apog=20 > 06:19 6-11421A 14.2 9.9 4.3 75 14 124 365 4.1 332 = nan apog=20 > 06:27 6-11421A 14.1 10.0 4.1 76 12 112 424 3.6 332 = nan apog=20 > 06:36 6-11421A 14.0 9.9 4.1 76 17 126 371 3.0 318 = nan apog=20 > 06:44 6-11421A 14.0 10.0 4.0 76 16 123 401 2.4 332 = nan apog=20 > 06:53 6-11421A 13.9 10.0 3.9 77 18 124 420 2.6 332 = nan apog=20 > 07:01 6-11421A 13.9 9.9 4.0 76 11 123 409 2.2 332 = nan apog=20 > 07:09 6-11421A 13.8 9.9 3.9 77 17 127 408 2.4 332 = nan apog=20 > 07:36 3-11466A 13.8 9.9 3.9 77 15 128 422 2.2 332 = nan apog=20 > 07:44 3-11466A 13.9 9.8 4.1 76 8 111 344 2.0 332 = nan apog=20 > 07:53 3-11466A 13.9 9.8 4.1 76 12 126 357 2.4 332 = nan apog=20 > 08:01 3-11466A 13.9 9.8 4.1 76 15 132 359 2.6 332 = nan apog=20 > 08:09 3-11466A 13.9 9.8 4.1 76 11 134 321 2.3 332 = nan apog=20 > 08:18 3-11466A 13.9 9.8 4.1 76 14 132 359 2.1 332 = nan apog=20 > 08:26 3-11466A 13.8 9.8 4.0 77 11 126 350 1.9 332 = nan apog=20 > 08:35 3-11466A 13.7 9.8 3.9 77 8 119 344 2.9 331 = nan apog=20 > 09:04 7-11213A 13.3 9.6 3.7 78 6 142 365 2.8 332 = nan apog=20 > 09:12 7-11213A 13.4 9.6 3.8 77 7 114 332 2.9 332 = nan apog=20 > 09:21 7-11213A 13.5 9.7 3.8 77 14 138 321 2.0 332 = nan apog=20 > 09:29 7-11213A 13.5 9.8 3.7 78 15 134 361 2.9 332 = nan apog=20 > 09:37 7-11213A 13.4 9.8 3.6 78 14 135 347 3.7 318 = nan apog=20 > 09:46 7-11213A 13.5 9.9 3.6 78 14 135 322 3.0 332 = nan apog=20 > 09:54 7-11213A 13.3 9.9 3.4 79 13 137 343 2.8 332 = nan apog=20 > 10:03 7-11213A 13.2 9.9 3.3 80 10 134 320 2.7 332 = nan apog=20 > 10:27 1-11602A 12.8 9.9 2.9 82 9 152 325 2.0 332 = nan apog=20 > 10:35 1-11602A 12.8 9.9 2.9 82 6 166 303 3.4 332 = nan apog=20 > 10:44 1-11602A 12.6 10.1 2.5 84 5 179 309 2.2 332 = nan apog=20 > 10:52 1-11602A 12.4 10.3 2.1 87 3 134 264 1.9 332 = nan apog=20 >=20 >=20 > ---- Problems in Detail ---- >=20 > Same TCC/MCP error after every gotoGangChange command tonight: >=20 > 04:16:36 .tcc 0 tcc w text=3D'STATUS failed for axis: rot1' > 04:16:36 .alerts 0 alerts w text=3D"Error processing rules for = tcc.tccPos: > KeyVar('tcc', tccPos=3D(None, None, None)) is unknown" > 04:16:42 .tcc 0 tcc w text=3D'STATUS failed for axis: az' > 04:16:42 .tcc 0 tcc w text=3D'STATUS failed for axis: alt' > 04:16:42 .tcc 0 tcc w text=3D'STATUS failed for axis: rot1' > 04:16:42 .alerts 0 alerts w text=3D"Error processing rules for = tcc.tccPos: > KeyVar('tcc', tccPos=3D(None, None, None)) is unknown" > 04:16:44 .tcc 0 tcc W Text=3D"number keys does not match number of = values" > 04:16:45 .hub 0 cmds d > = CmdQueued=3D1300277,1567829805.64,"OBSERVER.elenam",140,"sop",168,"gotoGan= gChange > nodomeFlat alt=3D30.0" > 04:16:45 .hub 0 cmds d > = CmdQueued=3D1300278,1567829805.68,"OBSERVER.elenam.sop",884,"tcc",14562,"a= xis status" > 04:16:48 .tcc 0 tcc w text=3D'STATUS failed for axis: az' > 04:16:48 .tcc 0 tcc w text=3D'STATUS failed for axis: az' > 04:16:48 .tcc 0 tcc w text=3D'STATUS failed for axis: alt' > 04:16:48 .tcc 0 tcc w text=3D'STATUS failed for axis: alt' > 04:16:48 .tcc 0 tcc w text=3D'STATUS failed for axis: rot1' > 04:16:48 .tcc 0 tcc w text=3D'STATUS failed for axis: rot1' > 04:16:48 OBSERVER.elenam.sop 884 tcc f text=3D"Sub-command(s) failed: = STATUS: > Cancelled on queue by immediate priority command 'INIT'; STATUS: = Cancelled on > queue by immediate priority command 'INIT'; STATUS: Cancelled on queue = by > immediate priority command 'INIT'" > 04:16:48 CmdDone: OBSERVER.elenam.sop 884 tcc axis status > 04:16:48 .alerts 0 alerts w text=3D"Error processing rules for = tcc.tccPos: > KeyVar('tcc', tccPos=3D(None, None, None)) is unknown" > 04:16:48 OBSERVER.elenam 140 sop e text=3D"Cannot check axis status. = Something is > very wrong!" > 04:16:48 OBSERVER.elenam 140 sop e text=3D"Is the TCC in a responsive = state?" > 04:16:48 OBSERVER.elenam 140 sop f text=3D"Failed to slew to gang = change" > 04:16:48 CmdDone: OBSERVER.elenam 140 sop gotoGangChange nodomeFlat = alt=3D30.0 > 04:16:48 .mcp 0 mcp ! command=3D"init" > 04:16:54 .tcc 0 tcc w text=3D'STATUS failed for axis: alt' > 04:16:54 .alerts 0 alerts w text=3D"Error processing rules for = tcc.tccPos: > KeyVar('tcc', tccPos=3D(None, None, None)) is unknown" > 04:17:00 .tcc 0 tcc w text=3D'STATUS failed for axis: alt' > 04:17:00 .tcc 0 tcc w text=3D'STATUS failed for axis: rot1' > 04:17:00 .alerts 0 alerts w text=3D"Error processing rules for = tcc.tccPos: > KeyVar('tcc', tccPos=3D(None, None, None)) is unknown" > 04:17:02 .hub 0 cmds d > = CmdQueued=3D1300282,1567829822.91,"OBSERVER.elenam",141,"sop",169,"gotoGan= gChange > nodomeFlat alt=3D30.0" > 04:17:02 .hub 0 cmds d > = CmdQueued=3D1300283,1567829822.94,"OBSERVER.elenam.sop",885,"tcc",14564,"a= xis status" > 04:17:06 .alerts 0 alerts w text=3D"Error processing rules for = tcc.tccPos: > KeyVar('tcc', tccPos=3D(None, None, None)) is unknown" > 04:17:10 .boss 4064 boss w text=3D"camcheck found missing or = unresponsive cameras: > set(['sp1cam'])" >=20 > ____________________________________________________________ > Logs of Nightly APO Activities > = https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html