Subject: Re: [SDSS3-infrastructure 7782] Re: [sdss3-25mlog 1280] 2.5m obslog 20130611 (MJD 56455) From: "John K. Parejko" Date: Wed, 12 Jun 2013 11:34:20 -0400 For the record, I just checked and the NFS poker is still running in a = screen session on boss@boss-icc, and is still writing/deleting files in = the spectro data dir.=20 John On 12Jun 2013, at 10:00, Stephen Bailey wrote: > =46rom the obslog last night: >=20 > On Wed, Jun 12, 2013 at 7:37 AM, wrote: >=20 >> APACHE POINT OBSERVATORY SDSS 2.5M >>=20 >> OBSERVING LOG >> Tuesday June 11, 2013 >>=20 >> (MJD 56455) >>=20 >> ... >> =20 >> Notes: >> ------ >>=20 >> The SDSS-DB computer shutdown itself off for the second time in four = days. The >> problem was discovered when we could not do a load-Cart for Cart 12. = We hit the >> power button on SDSS-DB, let it boot-up, and did a 'platedb = reconnect' Only ~5 >> minutes of science time were lost because we'd seen this problem on = Saturday >> during afternoon checkout. >>=20 >> Next, also for Cart 12, SOS stop posting any info for the b2 camera = although a >> redo would cause the info to post. So, we stopped and started SOS. >>=20 >> Later, SOS didn't post any info for the b1 camera although a redo = would cause >> the info to post. This was for Cart 17. So, we stopped and started = SOS, again. >>=20 >> It appears that there were several problems with this last night. In = one case, the IDL license manager was down/unavailable, preventing SOS = from running (or at least from writing output): >>=20 > sos_apocommand(science): % LICENSE MANAGER: License server machine is = down or not responding. > sos_apocommand(science): See the system adminstrator about starting = the license server system, or > sos_apocommand(science): make sure you're referring to the right = host (see LM_LICENSE_FILE). > sos_apocommand(science): Feature: idl > sos_apocommand(science): Hostname: galileo > sos_apocommand(science): License path: = /usr/local/itt/idl71/../license/license.dat:/usr/local/itt/l > sos_apocommand(science): icense/*.lic: > sos_apocommand(science): FLEXnet Licensing error:-96,7 > sos_apocommand(science): For further information, refer to the = FLEXnet Licensing documentation, > sos_apocommand(science): available at "www.acresso.com".. > sos_apocommand(science): License file: = /usr/local/itt/idl71/../license/license.dat:/usr/local/itt/li > sos_apocommand(science): cense/*.lic > sos_apocommand(science): % Entering timed demo mode. Each session is = limited to 7 minutes > sos_apocommand(science): of operation. Printing and file saving are = disabled. > sos_apocommand(science): > sos_apocommand(science): To obtain a temporary license allowing = evaluation of the full > sos_apocommand(science): functionality of this product contact your = sales representative or >=20 > In other cases before and after that, I didn't see the license = manager errors, but files were missing, e.g.: >=20 > sos_apocommand(science): APOREDUCE: sdR-b1-00163250.fit.gz: SPAWN = /usr/bin/convert /data/boss/sos/56455/snplot-56455-6985-00163250.ps = /data/boss/sos/56455/snplot-56455-6985-00163250-b1.jpeg ; \mv = /data/boss/sos/56455/snplot-56455-6985-00163250-b1.jpeg = /data/boss/sos/56455/snplot-56455-6985-00163250.jpeg & > sos_apocommand(science): convert: unable to open image = `/data/boss/sos/56455/snplot-56455-6985-00163250.ps': No such file or = directory. >=20 > The code didn't complain at the time it was trying to write that file. = =46rom the logs it is unclear if the code silently failed to write the = file, or whether it successfully wrote the file but then lost contact to = that directory and couldn't read it back. This problem caused = loadSN2Value to fail several times, e.g. >=20 > sos_apocommand(science): IOError: [Errno 2] No such file or directory: = '/data/boss/sos/56455/sci- > 6985-b1-00163250.fits' > sos_apocommand(science): Failed: failed running loadSN2Value --update = -v /data/boss/sos/56455/sci-6985- > b1-00163250.fits = /home/sdss3/products/NULL/speclog/trunk/56455/plPlugMapM-6985-56449-01.par= >=20 > Given that SOS worked when reprocessed, and we also have current = problems with the archiver talking to the hub, license manager = connection issues, this looks like networking trouble (not sure how that = could explain sdss-db shutting down at random, however). >=20 > Regards, >=20 > Stephen >=20 > _____________________________________________________ > Message 7782 of sdss3-infrastructure - SDSS-III Common APO hardware = and software. > = https://trac.sdss3.org/tracmailman/browser/private/sdss3-infrastructure/al= l/7782.html > sdss3-infrastructure@sdss3.org > https://sdss3.org/mailman/listinfo/sdss3-infrastructure -- ************************* John Parejko john.parejko@yale.edu http://www.astro.yale.edu/~jp727/ 203 432-9759 JWG 465 Department of Physics Yale University New Haven, CT **************************