java.net.SocketException: Connection reset by peer: socket write error
During a moderate load below error appears in weblogic logs : Connection reset by peer: socket write error
Environment:
They are running Weblogic in a windows 2003 ESX vmware environment.
The configuration starts with a RHEL 5.1 running apache, cluster of 6 servers with the weblogic plugin.
The plugin points to a cisco load balancer which will forward the requests to a weblogic tier servers.
The weblogic tier has a cluster of 4 servers and another 5 managed servers.
ERROR
####<Nov 26, 2011 12:52:31 PM AST> <Error> <HTTP> <TechPaste> <PRE-WL3> <ExecuteThread: '18' for queue: 'weblogic.kernel.Default'> <<WLS Kernel>> <BEA-101017> <http://ServletContext(id=24499159,name=spWeb.war,context-path=) Root cause of ServletException. javax.servlet.jsp.JspException: ServletException in:/games/confirm.jsp Input/output error: java.net.SocketException: Connection reset by peer: socket write error' at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.processException(InsertTag.java:875) at org.apache.struts.taglib.tiles.InsertTag$InsertHandler.doEndTag(InsertTag.java:835) at org.apache.struts.taglib.tiles.InsertTag.doEndTag(InsertTag.java:466) at jsp_servlet._templates.__main_template._jspService(__main_template.java:655) at weblogic.servlet.jsp.JspBase.service(JspBase.java:33) at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348) at weblogic.servlet.internal.RequestDispatcherImpl.forward(RequestDispatcherImpl.java:336) at org.apache.struts.action.RequestProcessor.doForward(RequestProcessor.java:1058) at org.apache.struts.tiles.TilesRequestProcessor.doForward(TilesRequestProcessor.java:269) at org.apache.struts.tiles.TilesRequestProcessor.processTilesDefinition(TilesRequestProcessor.java:249) at org.apache.struts.tiles.TilesRequestProcessor.processForwardConfig(TilesRequestProcessor.java:303) at org.apache.struts.action.RequestProcessor.processActionForward(RequestProcessor.java:401) at org.apache.struts.action.RequestProcessor.process(RequestProcessor.java:279) at car.lce.plays.ui.struts.PsRequestProcessor.process(PsRequestProcessor.java:391) at org.apache.struts.action.ActionServlet.process(ActionServlet.java:1420) at org.apache.struts.action.ActionServlet.doPost(ActionServlet.java:520) at javax.servlet.http.HttpServlet.service(HttpServlet.java:760) at javax.servlet.http.HttpServlet.service(HttpServlet.java:853) at weblogic.servlet.internal.ServletStubImpl$ServletInvocationAction.run(ServletStubImpl.java:1077) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:465) at weblogic.servlet.internal.ServletStubImpl.invokeServlet(ServletStubImpl.java:348) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:7058) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:121) at weblogic.servlet.internal.WebAppServletContext.invokeServlet(WebAppServletContext.java:3902) at weblogic.servlet.internal.ServletRequestImpl.execute(ServletRequestImpl.java:2756) at weblogic.kernel.ExecuteThread.execute(ExecuteThread.java:224) at weblogic.kernel.ExecuteThread.run(ExecuteThread.java:183)
Below are the logs seen in cisco load balancer probes while probing for server
Nov 27 2011 13:40:25 : %ACE-3-251010: Health probe failed for server 10.136.12.243 on port 443, graceful disconnect timeout (no FIN ACK) Nov 27 2011 13:40:55 : %ACE-3-251010: Health probe failed for server 10.136.12.243 on port 443, graceful disconnect timeout (no FIN ACK) Nov 27 2011 13:41:06 : %ACE-3-251010: Health probe failed for server 10.136.21.421 on port 443, graceful disconnect timeout (no FIN ACK) Nov 27 2011 13:41:11 : %ACE-3-251010: Health probe failed for server 10.136.12.243 on port 443, graceful disconnect timeout (no FIN ACK) Nov 27 2011 13:41:11 : %ACE-4-442002: Health probe TCP-443 detected RS-EWAPPS_PRE-WL3 (interface vlan619) in serverfarm SF_WLSAPPS-443 changed state to DOWN The server cannot write to the socket (OS level).
There are bridge errors detected by the apache web server. The user gets a strange error
Failure of server APACHE bridge: _____ No backend server available for connection: timed out after 12 seconds or idempotent set to OFF. _____
CAUSE:
As the load increased, the Cisco load balancer was configured to probe the weblogic cluster managed servers at the port.
If the response from the port 443 probe exceeded a value (number of probes and timeout) the load balancer would set the server status as down, thus getting the connection reset by peer error in the weblogic server logs and a bridge error in the apache logs / end users screen.
SOLUTION:
Turning the port probe off or setting the probe values larger would impact the users in a positive way and would not reset the connection at all or not as quickly as the case may be.
In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.