Subject: Re: [sloan-25mlog 3471] Re: 2.5m obslog 20170419 (SJD 57863)
From: Kai Zhang
Date: Thu, 20 Apr 2017 11:26:48 -0400
Hi Kaike,
    Thank you for the information. This makes a lot of sense.=20
    242280 and 242281 do not exist, and other exposures seem fine. I =
checked against the guider movie and confirm when the guider header says =
the dithering position is =A1=AFN=A1=AF, the telescope is pointing at =
=A1=AFN=A1=AF indeed.=20
    Best~
                                                Kai
> =D4=DA 2017=C4=EA4=D4=C220=C8=D5=A3=AC=C9=CF=CE=E710:42=A3=ACKaike Pan =
 =D0=B4=B5=C0=A3=BA
>=20
> Hello All,
>=20
> Here is what I think happened last night.=20
>=20
> Patrick is right, there was no exposure to recover. Looks like =
BOSS-icc
> dead in flush stage instead of reading out stage. i.e. BOSS-icc =
crashed at the very
> beginning of =A1=B0E=A1=B1 dither when it flushed for the data taking. =
So, this is not Netburner
> problem.
>=20
>=20
> Lessons learned:
> 1) BOSS-icc could fail at flush stage while prepare for a new exposure =
or at the reading out stage.
> One should able to tell from BOSS instrument GUI or from existed files =
in /data/spectro/mjd. If
> it fails at the flush stage, what we need to do is: 1) reload BOSS =
controller, boss ping,=20
> boss clearExposure (may not needed actually, but don=A1=AFt hut), and =
boss exposure bias.
> 2) This failure will mess file frame number output by logFun script =
(last night, we , but disk files are
> fines. Last night logFun outputted two files with number of 242280:
>=20
> 07:03:02  apogee.expose=3DapRaw-23010047, 42, B, Object
> 07:03:09  boss exposure  900.0, file=3D242280, N
> 07:06:36  apogee.expose=3DapRaw-23010048, 42, B, Object
>=20
> 07:11:02  apogee.expose=3DapRaw-23010049, 42, B, Object
> 07:11:09  boss exposure  900.0, file=3D242280, E
> 07:18:38  apogee.expose=3DapRaw-23010050, 42, A, Object
>=20
> while disk files is: 38 07:11 b1-00242282  6  9037 sci Out E  900.1  =
26365  26217  22702    =20
> Looks like that logFun outputted a non-existed =A1=B0N=A1=B1 dither =
exposure, messed the frame number
> on the next E dither exposure. It should be 242282 while logFun =
outputted it as
> 242280.
>=20
> MaNGA team, please double check and make sure dither position for =
exposures on cart 6 are all
> correct (I think they are), and let us know. I would like to update =
our BOSS trouble shooting based
> on your check results and what I found here.
>=20
> Thanks,
>=20
> -kaike
>=20
>=20
>>=20
>> Good weather, good seeing.
>> Time loss: 30 min (netburner issue, 20 operator, 10 software), see =
log cart 6.
>>=20
>>=20
>>=20
>> ****************************************************
>> 06:00:59  loadCart: Cart 6, Plate 9037;
>> 06:00:59 survey: APOGEE-2&MaNGA;  lead: MaNGA dither
>> ****************************************************
>> 06:05:28 Hartmann collimate output on cart #6
>>  sp1: offset: r=3D 1979 (Out of focus);  b=3D -7.3 (Out of focus)=20
>>  sp2: offset: r=3D 1893 (Out of focus);  b=3D -7.9 (Out of focus)=20
>>  sp1: pred. move: spAverageMove=3D 2146
>>  sp2: pred. move: spAverageMove=3D 2207
>>  sp1: pred. resid: r=3D -167, b=3D -0.5, txt=3D OK, spTemp =3D 10.8
>>  sp2: pred. resid: r=3D -314, b=3D -1.0, txt=3D OK, spTemp =3D 10.1
>>=20
>> NoGSOGTF (-20RA), clear, seeing 1.1
>>=20
>> %... Field exposures
>> 06:13:12  apogee.expose=3DapRaw-23010040, 42, A, Object
>> 06:13:19  boss exposure  900.0, file=3D242278, N
>> 06:20:43  apogee.expose=3DapRaw-23010041, 42, B, Object
>>=20
>> 06:29:35  apogee.expose=3DapRaw-23010042, 42, B, Object
>> 06:29:43  boss exposure  900.0, file=3D242279, S
>> 06:37:12  apogee.expose=3DapRaw-23010043, 42, A, Object
>>=20
>> 06:45:59  apogee.expose=3DapRaw-23010044, 42, A, Object
>> 06:53:31  apogee.expose=3DapRaw-23010045, 42, B, Object
>> 07:01:00  apogee.expose=3DapRaw-23010046, 42, B, Object
>>=20
>> 07:03:02  apogee.expose=3DapRaw-23010047, 42, B, Object
>> 07:03:09  boss exposure  900.0, file=3D242280, N
>> 07:06:36  apogee.expose=3DapRaw-23010048, 42, B, Object
>>=20
>> 07:11:02  apogee.expose=3DapRaw-23010049, 42, B, Object
>> 07:11:09  boss exposure  900.0, file=3D242280, E
>> 07:18:38  apogee.expose=3DapRaw-23010050, 42, A, Object
>>=20
>> 07:27:26  apogee.expose=3DapRaw-23010051, 42, A, Object
>> 07:27:33  boss exposure  900.0, file=3D242283, N
>> 07:34:56  apogee.expose=3DapRaw-23010052, 42, B, Object
>>=20
>> 07:43:49  apogee.expose=3DapRaw-23010053, 42, B, Object
>> 07:43:56  boss exposure  900.0, file=3D242284, S
>> 07:51:24  apogee.expose=3DapRaw-23010054, 42, A, Object
>>=20
>> 08:00:13  apogee.expose=3DapRaw-23010055, 42, A, Object
>> 08:00:20  boss exposure  900.0, file=3D242285, E
>> 08:07:43  apogee.expose=3DapRaw-23010056, 42, B, Object
>>=20
>> 08:16:39  apogee.expose=3DapRaw-23010057, 42, B, Object
>> 08:16:46  boss exposure  900.0, file=3D242286, N
>> 08:24:14  apogee.expose=3DapRaw-23010058, 42, A, Object
>>=20
>> 08:33:03  apogee.expose=3DapRaw-23010059, 42, A, Object
>> 08:33:11  boss exposure  900.0, file=3D242287, S
>> 08:40:34  apogee.expose=3DapRaw-23010060, 42, B, Object
>>=20
>> %... DOS
>> SCIENCE-00242278 	(S/N)^2 	2.2 	6.3 	2.5 	5.6 	=
900.1
>> SCIENCE-00242279 	(S/N)^2 	2.2 	6.3 	2.7 	5.8 	=
900.1=20
>> SCIENCE-00242282 	(S/N)^2 	2.3 	6.2 	2.7 	5.5 	=
900.1
>> SCIENCE-00242283 	(S/N)^2 	2.4 	6.3 	2.7 	5.5 	=
900.1=20
>> SCIENCE-00242284 	(S/N)^2 	2.3 	6.2 	2.8 	5.6 	=
900.1=20
>> SCIENCE-00242285 	(S/N)^2 	2.4 	5.9 	2.8 	5.0 	=
900.1
>> SCIENCE-00242286 	(S/N)^2 	2.5 	5.9 	2.8 	4.8 	=
900.1
>> SCIENCE-00242287 	(S/N)^2 	2.3 	5.8 	2.7 	4.5 	=
900.1=20
>> TOTAL- 	TOTAL (S/N)^2 	18.6 	48.9 	21.7 	42.3 =09
>>=20
>> *********** NETBURNER ISSUE *************
>> We had a netburner issue during the first E (after NS). Everything =
else was
>> doing fine (guiding, apogee etc). When we realized the boss icc was =
dead, we
>> double-checked the procedure and the DOS. I (Patrick) got confused =
when
>> scrolling down the DOS page, despite I checked twice if the last E =
was read and
>> analyzed. Twice I thought it was the case, and I cleared the exposure =
(after
>> reload controllers). I probably got confused with cart 5 which was =
right after 6
>> in the DOS. We thus lost the E exposure. I am very sorry for that. =
Then we
>> restarted the cycle. Unfortunately, because of the break, we could =
not patch the
>> first two exposures, so we'll need to trade time with boss to get the =
missing
>> exposure done.
>>=20
>> 35 06:07 b1-00242277  6  9037 fla Out C   25.1  26365  26217  22702   =
 =20
>>   06:07 b2-00242277  6  9037 fla Out C   25.1  23396  24115  23334    =
=20
>>   06:07 r1-00242277  6  9037 fla Out C   25.1  26365  26217  22702    =
=20
>>   06:07 r2-00242277  6  9037 fla Out C   25.1  23396  24115  23334    =
=20
>> 36 06:13 b1-00242278  6  9037 sci Out N  900.1  26365  26217  22702   =
 =20
>>   06:13 b2-00242278  6  9037 sci Out N  900.1  23396  24115  23334    =
=20
>>   06:13 r1-00242278  6  9037 sci Out N  900.1  26365  26217  22702    =
=20
>>   06:13 r2-00242278  6  9037 sci Out N  900.1  23396  24115  23334    =
=20
>> 37 06:29 b1-00242279  6  9037 sci Out S  900.1  26365  26217  22702   =
 =20
>>   06:29 b2-00242279  6  9037 sci Out S  900.2  23396  24115  23334    =
=20
>>   06:29 r1-00242279  6  9037 sci Out S  900.1  26365  26217  22702    =
=20
>>   06:29 r2-00242279  6  9037 sci Out S  900.2  23396  24115  23334    =
=20
>> 38 07:11 b1-00242282  6  9037 sci Out E  900.1  26365  26217  22702   =
 =20
>>   07:11 b2-00242282  6  9037 sci Out E  900.2  23396  24115  23334    =
=20
>>   07:11 r1-00242282  6  9037 sci Out E  900.1  26365  26217  22702    =
=20
>>   07:11 r2-00242282  6  9037 sci Out E  900.2  23396  24115  23334    =
=20
>> 39 07:27 b1-00242283  6  9037 sci Out N  900.1  26365  26217  22702   =
 =20
>>   07:27 b2-00242283  6  9037 sci Out N  900.1  23396  24115  23334    =
=20
>>   07:27 r1-00242283  6  9037 sci Out N  900.1  26365  26217  22702    =
=20
>>   07:27 r2-00242283  6  9037 sci Out N  900.1  23396  24115  23334    =
=20
>> 40 07:43 b1-00242284  6  9037 sci Out S  900.1  26365  26217  22702   =
 =20
>>   07:43 b2-00242284  6  9037 sci Out S  900.2  23396  24115  23334    =
=20
>>   07:43 r1-00242284  6  9037 sci Out S  900.1  26365  26217  22702    =
=20
>>   07:43 r2-00242284  6  9037 sci Out S  900.2  23396  24115  23334    =
=20
>> 41 08:00 b1-00242285  6  9037 sci Out E  900.1  26365  26217  22702   =
 =20
>>   08:00 b2-00242285  6  9037 sci Out E  900.1  23396  24115  23334    =
=20
>>   08:00 r1-00242285  6  9037 sci Out E  900.1  26365  26217  22702    =
=20
>>   08:00 r2-00242285  6  9037 sci Out E  900.1  23396  24115  23334    =
=20
>> 42 08:16 b1-00242286  6  9037 sci Out N  900.1  26365  26217  22702   =
 =20
>>   08:16 b2-00242286  6  9037 sci Out N  900.2  23396  24115  23334    =
=20
>>   08:16 r1-00242286  6  9037 sci Out N  900.1  26365  26217  22702    =
=20
>>   08:16 r2-00242286  6  9037 sci Out N  900.2  23396  24115  23334    =
=20
>> 43 08:33 b1-00242287  6  9037 sci Out S  900.1  26365  26217  22702   =
 =20
>>   08:33 b2-00242287  6  9037 sci Out S  900.2  23396  24115  23334    =
=20
>>   08:33 r1-00242287  6  9037 sci Out S  900.1  26365  26217  22702    =
=20
>>   08:33 r2-00242287  6  9037 sci Out S  900.2  23396  24115  23334    =
=20
>>=20
>=20
> ____________________________________________________________
> Logs of Nightly APO Activities
> =
https://trac.sdss.org/tracmailman/browser/private/sloan-25mlog/index.html