From: stephanie snedden Subject: Re: [SDSS3-infrastructure 4396] Re: PlateWebApp and 2.5m obslog 20101209 (MJD 55540) Date: Fri, 10 Dec 2010 16:27:21 -0700 --Apple-Mail-7-183829269 Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit My first reply didn't go to infrastructure, so I am repeating it; Well these types of problems never happen when you want them to. Yes, after we restarted the webapp, the plate showed up as "force complete", so it does complete that step. Another diagnostic: Before I restarted the webapp, I checked its status, and the reply was that it was running. The restart did get the webapp working, though. Steph On Dec 10, 2010, at 2:36 PM, Stephen Bailey wrote: > Hi. For the record -- > > Kaike and I just tried to force complete several plates and were > unable to get the webapp to crash/hang. We tried both port 8081 and > 8082. So the problem remains a mystery. > > I have updated the standard port 8081 to use the code which has been > running on 8082 for the past week. Although not perfect, it is more > stable than the previous code (no tracebacks requiring reloads) and > it fixes the occasional missing cart in the display problem. Both > 8081 and 8082 are up and running right now. > > Regards, > > Stephen > > > On Fri, Dec 10, 2010 at 9:13 AM, kaike pan wrote: > Hi Stephen and Adrian, > > If one of you is available by 2:30 PM MST, I would like to try > this today. Can you send me a message when you are available > this afternoon. Thanks, > > -kaike > > On Dec 10, 2010, at 10:02 AM, Stephen Bailey wrote: > >> Hi all. >> >> On Fri, Dec 10, 2010 at 4:50 AM, Viktor M >> wrote: >> >> APACHE POINT OBSERVATORY SDSS 2.5M >> >> OBSERVING LOG >> Thursday December 9, 2010 >> >> (MJD 55540) >> >> ---=== OBSERVING TEAM ===--- >> >> Stephanie Snedden >> Viktor M >> >> >> ---=== OBSERVING PLAN ===--- >> >> Science! >> 9 Dec 2010 55540.0 0.30 01:25 12:27 23:36 10:40 BOSS 11.0 hours >> >> ---=== OBSERVING SUMMARY ===--- >> >> Mostly cloudy night >> >> We observed the next BOSS cartridges: >> >> Cart. 10#4357 >> Cart. 12#4372 >> Cart. 14#4386 DONE >> Cart. 13#4343 DONE >> Cart. 15#4450 DONE >> Cart. 11#3701 DONE >> Cart. 17#3819 DONE >> Cart. 18#3831 >> >> During observations, we tried to force to complete cart 15 plate >> 4450 and >> after that platedb web page was not available. >> Tried to use 8081,8082, and public link >> http://sdss-plates.apo.nmsu.edu/ 8082 and/or 8081 was not responded. >> We restarted the plateWebApp. >> >> I have a doctor's appointment so I won't be available to debug this >> this morning, but I should be back to the lab by early afternoon. >> When the plugging is done we can run some tests to see if we can >> reproduce the force complete hang. >> >> After you restarted the server, were you able to force complete >> plate 4450? Or did you try again and it crashed again? Or did you >> leave it as-is and not try? >> >> Note: debug port 8082 is currently working and I did *not* restart >> the platewebapp. The standard instructions for restarting the >> primary server on 8081 do *not* restart the debug server. So it >> appears that this "crash" may be a "really long hang". >> Operationally that may not make much of a difference (you still >> have to restart the server to get on with things), but it may give >> us a clue about what is going on. e.g. if the write to the DB to >> update the completion status hits a network hiccup, it might have a >> very long timeout. But last time we had this "force complete >> crash", 8082 seemed to be permanently down and I had to restart it. >> >> Regards, >> >> Stephen >> >> [rest of log clipped] >> >> _______________________________________________ >> Message 4393 of sdss3-infrastructure - >> sdss3-infrastructure@sdss3.org >> https://sdss3.org/mailman/listinfo/sdss3-infrastructure > > > _______________________________________________ > Message 4396 of sdss3-infrastructure - > sdss3-infrastructure@sdss3.org > https://sdss3.org/mailman/listinfo/sdss3-infrastructure --Apple-Mail-7-183829269 Content-Type: text/html; charset=US-ASCII Content-Transfer-Encoding: quoted-printable My first reply didn't go to = infrastructure, so I am repeating it;Well these = types of problems never happen when you want them = to. Yes, after we restarted the webapp, = the plate showed up as "force complete", so it does complete that = step.Another diagnostic: Before I restarted = the webapp, I checked its status, and the reply was that it was running. = The restart did get the webapp working, = though.StephOn Dec 10, 2010, = at 2:36 PM, Stephen Bailey wrote:Hi. = For the record --Kaike and I just tried to = force complete several plates and were unable to get the webapp to = crash/hang. We tried both port 8081 and 8082. So the problem = remains a mystery. I have updated the standard = port 8081 to use the code which has been running on 8082 for the past = week. Although not perfect, it is more stable than the previous = code (no tracebacks requiring reloads) and it fixes the occasional = missing cart in the display problem. Both 8081 and 8082 are up and = running right now. = Regards,StephenOn Fri, Dec 10, 2010 at 9:13 AM, kaike = pan = wrote: Hi Stephen and = Adrian, If one of you is = available by 2:30 PM MST, I would like to try this today. Can you send = me a message when you are availablethis afternoon. = Thanks, = -kaikeOn Dec 10, 2010, at 10:02 AM, Stephen Bailey = wrote:Hi all. On Fri, Dec 10, = 2010 at 4:50 AM, Viktor M = wrote: = APACHE = POINT OBSERVATORY SDSS 2.5M = = OBSERVING LOG = Thursday December = 9, 2010 = = (MJD 55540) ---=3D=3D=3D OBSERVING TEAM =3D=3D=3D--- = Stephanie Snedden = Viktor M ---=3D=3D=3D OBSERVING PLAN = =3D=3D=3D--- Science! 9 Dec 2010 = 55540.0 0.30 01:25 12:27 23:36 10:40 BOSS = 11.0 hours ---=3D=3D=3D OBSERVING SUMMARY =3D=3D=3D--- = Mostly cloudy night We observed the next BOSS = cartridges: Cart. 10#4357 Cart. 12#4372 Cart. 14#4386 = DONE Cart. 13#4343 DONE Cart. 15#4450 DONE Cart. 11#3701 = DONE Cart. 17#3819 DONE Cart. 18#3831 During = observations, we tried to force to complete cart 15 plate 4450 and = after that platedb web page was not available. Tried to use = 8081,8082, and public link http://sdss-plates.apo.nmsu.edu/ 8082 and/or 8081 = was not responded. We restarted the = plateWebApp.I have a doctor's = appointment so I won't be available to debug this this morning, but I = should be back to the lab by early afternoon. When the plugging is = done we can run some tests to see if we can reproduce the force complete = hang. After you restarted the server, were you = able to force complete plate 4450? Or did you try again and it = crashed again? Or did you leave it as-is and not = try?Note: debug port 8082 is currently working = and I did *not* restart the platewebapp. The standard instructions = for restarting the primary server on 8081 do *not* restart the debug = server. So it appears that this "crash" may be a "really long = hang". Operationally that may not make much of a difference (you = still have to restart the server to get on with things), but it may give = us a clue about what is going on. e.g. if the write to the DB to = update the completion status hits a network hiccup, it might have a very = long timeout. But last time we had this "force complete crash", = 8082 seemed to be permanently down and I had to restart it. = Regards,Stephen&n= bsp;[rest of log = clipped] _______________________________________________Message 4393 of sdss3-infrastructure - sdss3-infrastructure@sdss3.orghttps://sdss3.org/mailman/listinfo/sdss3-infrastructure = = _______________________________________________Message 4396 of = sdss3-infrastructure -sdss3-infrastructure@sdss3.= orghttps://sdss3.org/mailman/listinfo/sdss3-infrastructure= --Apple-Mail-7-183829269--