Date: Thu, 2 Oct 2014 14:49:25 -0700 Subject: Re: SOS problems on SJD 56932 From: Stephen Bailey --089e015382221723b805047797e9 Content-Type: text/plain; charset=UTF-8 On Thu, Oct 2, 2014 at 2:38 PM, Daniel Oravetz wrote: > As eboss@sdss4-eboss, the command syntax used was 'sos_aporedo -g -z > exp#'. > Thanks for the clarification. That is the correct command (note that -m MJD is needed if you are processing stuff for a prior night, but it defaults to the current night). My guess is that the daemons were simultaneously processing another exposure that had been taken in the meantime, resulting in an out-of-memory problem killing off the redo processes. If you encounter this again, please re-retry sos_aporedo when you know that the SOS daemons are not processing another exposure (e.g. the previous exposure has finished posting to the sos webpage http://sdss-sos.apo.nmsu.edu/logfile-current.html . Regards, Stephen > > > > From the SJD 56932 observing log: > > > > On Thu, Oct 2, 2014 at 4:51 AM, wrote: > > > >> > >> APACHE POINT OBSERVATORY SDSS 2.5M > >> > >> OBSERVING LOG > >> Wednesday October 01, 2014 > >> > >> (SJD 56932) > >> > >> ---- Observing Team ---- > >> > >> Karen Kinemuchi > >> Dan Oravetz > >> > >> [...] > > > > > >> SOS no information for cart. 12 exposure > >> ---------------------------------------- > >> For cart. 12, 7157 we had an arc on disk (exp. # 186106), a flat (exp. # > >> 186108), and the lone science exposure (exp. # 186110), but only had SOS > >> output > >> for the flat. Below are the associated comments SOS provided for the arc > >> and > >> science exposures. For some reason, even though the flat was there, SOS > >> still > >> hadn't processed the arc and without the processed arc the science > >> exposure was > >> not processed either. We aren't sure why SOS was waiting for the flat > >> that > >> was > >> already there before proceeding. > >> > >> sdR-r1-00186106.fit.gz: INFO: Arc exposure, waiting for flat before > >> reducing > >> sdR-b2-00186106.fit.gz: INFO: Arc exposure, waiting for flat before > >> reducing > >> sdR-b1-00186106.fit.gz: INFO: Arc exposure, waiting for flat before > >> reducing > >> sdR-r2-00186106.fit.gz: INFO: Arc exposure, waiting for flat before > >> reducing > >> sdR-r1-00186110.fit.gz: ABORT: Unable to reduce this science exposure > >> (need arc) > >> sdR-r2-00186110.fit.gz: ABORT: Unable to reduce this science exposure > >> (need arc) > >> sdR-b1-00186110.fit.gz: ABORT: Unable to reduce this science exposure > >> (need arc) > >> sdR-b2-00186110.fit.gz: ABORT: Unable to reduce this science exposure > >> (need arc) > >> > >> We manually commanded SOS to reduce the arc using sos_aporeduce as > >> eboss@sdss4-eboss, but it failed with the following error (even though > >> r1 > >> and r2 > >> reduced seemingly OK according to the output on the SOS page): > >> > >> sos_apocommand(arc): > >> /home/sdss4/products/Linux64/idlspec2d/trunk/bin/sos_apocommand: line > >> 69: > >> 7819 > >> Killed idl -e "aporeduce, 'sdR-b1-00186106.fit.gz', > >> indir='/data/spectro/56932', plugfile='plPlugMapM-7157-56932-01.par', > >> outdir='/data/boss/sos/56932', copydir='/data/boss/sos/combined', > >> plugdir='/home/sdss4/products/NULL/speclog/trunk/56932'" > >> sos_apocommand(arc): Failed: failed running idl -e "aporeduce, > >> 'sdR-b1-00186106.fit.gz', indir='/data/spectro/56932', > >> plugfile='plPlugMapM-7157-56932-01.par', outdir='/data/boss/sos/56932', > >> copydir='/data/boss/sos/combined', > >> plugdir='/home/sdss4/products/NULL/speclog/trunk/56932'" > >> sos_apocommand(arc): Goodbye from sos_apocommand! > >> > >> Command failed with rc = 1 > >> > > > > Thanks for reporting this. Do you have the *exact* command you ran? > > "aporeduce" (sos_aporeduce) is an IDL command, but it looks like you ran > > an executable that wrappered that. I was able to run this and it worked: > > > > sos_aporedo -m 56932 -z 186106 -g > > sos_aporedo -m 56932 -z 186108 -g > > sos_aporedo -m 56932 -z 186110 -g > > > > The "Killed" message is suspicious, like it ran out of memory or > > something... > > > > More later. > > > > Stephen > > > > > --089e015382221723b805047797e9 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable = On Thu, Oct 2, 2014 at 2:38 PM, Daniel Oravetz wrote:As eboss@sdss4-eboss, the command= syntax used was 'sos_aporedo -g -z exp#'.Thanks for the clarification.=C2=A0 That is the correct command= (note that -m MJD is needed if you are processing stuff for a prior night,= but it defaults to the current night).My guess i= s that the daemons were simultaneously processing another exposure that had= been taken in the meantime, resulting in an out-of-memory problem killing = off the redo processes.If you encounter this agai= n, please re-retry sos_aporedo when you know that the SOS daemons are not p= rocessing another exposure (e.g. the previous exposure has finished posting= to the sos webpage=C2=A0http://sdss-sos.apo.nmsu.edu/logfile-current.html .Regards,Stephen=C2=A0 > From the SJD 56932 observing log: > > On Thu, Oct 2, 2014 at 4:51 AM, wrote: > >> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 APACHE POINT OBSERVATORY SDSS 2.5M >> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 OBSERVING LOG >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Wednesday October 01, 2014 >> >>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0(SJD 56932) >> >> ----=C2=A0 Observing Team ---- >> >> Karen Kinemuchi >> Dan Oravetz >> >> [...] > > >> SOS no information for cart. 12 exposure >> ---------------------------------------- >> For cart. 12, 7157 we had an arc on disk (exp. # 186106), a flat (= exp. # >> 186108), and the lone science exposure (exp. # 186110), but only h= ad SOS >> output >> for the flat. Below are the associated comments SOS provided for t= he arc >> and >> science exposures. For some reason, even though the flat was there= , SOS >> still >> hadn't processed the arc and without the processed arc the sci= ence >> exposure was >> not processed either. We aren't sure why SOS was waiting for t= he flat >> that >> was >> already there before proceeding. >> >> sdR-r1-00186106.fit.gz: INFO: Arc exposure, waiting for flat befor= e >> reducing >> sdR-b2-00186106.fit.gz: INFO: Arc exposure, waiting for flat befor= e >> reducing >> sdR-b1-00186106.fit.gz: INFO: Arc exposure, waiting for flat befor= e >> reducing >> sdR-r2-00186106.fit.gz: INFO: Arc exposure, waiting for flat befor= e >> reducing >> sdR-r1-00186110.fit.gz: ABORT: Unable to reduce this science expos= ure >> (need arc) >> sdR-r2-00186110.fit.gz: ABORT: Unable to reduce this science expos= ure >> (need arc) >> sdR-b1-00186110.fit.gz: ABORT: Unable to reduce this science expos= ure >> (need arc) >> sdR-b2-00186110.fit.gz: ABORT: Unable to reduce this science expos= ure >> (need arc) >> >> We manually commanded SOS to reduce the arc using sos_aporeduce as= >> eboss@sdss4-eboss, but it failed with the following error (even th= ough >> r1 >> and r2 >> reduced seemingly OK according to the output on the SOS page): >> >> sos_apocommand(arc): >> /home/sdss4/products/Linux64/idlspec2d/trunk/bin/sos_apocommand: l= ine >> 69: >> 7819 >> Killed=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 idl -e "aporeduce, 'sdR-b1-00186106.fit.gz', >> indir=3D'/data/spectro/56932', plugfile=3D'plPlugMapM-= 7157-56932-01.par', >> outdir=3D'/data/boss/sos/56932', copydir=3D'/data/boss= /sos/combined', >> plugdir=3D'/home/sdss4/products/NULL/speclog/trunk/56932'&= quot; >> sos_apocommand(arc): Failed: failed running idl -e "aporeduce= , >> 'sdR-b1-00186106.fit.gz', indir=3D'/data/spectro/56932= ', >> plugfile=3D'plPlugMapM-7157-56932-01.par', outdir=3D'/= data/boss/sos/56932', >> copydir=3D'/data/boss/sos/combined', >> plugdir=3D'/home/sdss4/products/NULL/speclog/trunk/56932'&= quot; >> sos_apocommand(arc): Goodbye from sos_apocommand! >> >> Command failed with rc =3D 1 >> > > Thanks for reporting this.=C2=A0 Do you have the *exact* command you r= an? >=C2=A0 "aporeduce" (sos_aporeduce) is an IDL command, but it = looks like you ran > an executable that wrappered that.=C2=A0 I was able to run this and it= worked: > > sos_aporedo -m 56932 -z 186106 -g > sos_aporedo -m 56932 -z 186108 -g > sos_aporedo -m 56932 -z 186110 -g > > The "Killed" message is suspicious, like it ran out of memor= y or > something... > > More later. > > Stephen > --089e015382221723b805047797e9--