Solving java.net.SocketException: Write failed: Broken pipe Errors | Weblogic

During high load we usually get very low performance usually in navigating and executing any queries from application server to the back end database. In server log files we usually get broken pipe errors like below.

 

java.net.SocketException: Write failed: Broken pipe
        at jrockit.net.SocketNativeIO.writeBytesPinned(Native Method)
        at jrockit.net.SocketNativeIO.socketWrite(SocketNativeIO.java:46)
        at java.net.SocketOutputStream.socketWrite0(SocketOutputStream.java)
        at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
        at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
        at weblogic.servlet.internal.ChunkOutput.writeChunkTransfer(ChunkOutput.java:568)
        at weblogic.servlet.internal.ChunkOutput.writeChunks(ChunkOutput.java:539)
        at weblogic.servlet.internal.ChunkOutput.flush(ChunkOutput.java:427)
        at weblogic.servlet.internal.CharsetChunkOutput.flush(CharsetChunkOutput.java:298)
        at weblogic.servlet.internal.ChunkOutputWrapper.flush(ChunkOutputWrapper.java:188)
        at weblogic.servlet.internal.ServletOutputStreamImpl.flush(ServletOutputStreamImpl.java:121)
        at weblogic.servlet.internal.ServletResponseImpl.flushBuffer(ServletResponseImpl.java:197)
        at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:176)
        at javax.servlet.ServletResponseWrapper.flushBuffer(ServletResponseWrapper.java:176)

 

Or below kind of errors if service is failing:

javax.xml.ws.soap.SOAPFaultException: java.lang.NullPointerException
        at oracle.j2ee.ws.client.jaxws.DispatchImpl.throwJAXWSSoapFaultException(DispatchImpl.java:1028)
        at oracle.j2ee.ws.client.jaxws.DispatchImpl.invoke(DispatchImpl.java:812)
        at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.synchronousInvocationWithRetry(OracleDispatchImpl.java:235)
        at oracle.j2ee.ws.client.jaxws.OracleDispatchImpl.invoke(OracleDispatchImpl.java:106) 
        at oracle.j2ee.ws.client.jaxws.WsClientProxyInvocationHandler.invoke(WsClientProxyInvocationHandler.java:225)
        at $Proxy453.getDeliveryServiceDefinition(Unknown Source)
        at oracle.xdo.service.client.ReportService.getDeliveryServiceDefinition(ReportService.java:822)
        at oracle.apps.fnd.applcp.srs.model.srsService.applicationModule.ScheduleAMImpl.getDestinationServerList(ScheduleAMImpl.java:4279)
        at oracle.apps.fnd.applcp.srs.view.backing.ScheduleRequest.getDestinationServerList(ScheduleRequest.java:2398)
        at oracle.apps.fnd.applcp.srs.view.backing.OutputHandler.getPrinterList(OutputHandler.java:680)

OR

javax.xml.ws.WebServiceException: java.io.InterruptedIOException
        at com.sun.xml.ws.transport.http.client.HttpTransportPipe.process(HttpTransportPipe.java:306)
        at com.sun.xml.ws.transport.http.client.HttpTransportPipe.processRequest(HttpTransportPipe.java:124)
        at com.sun.xml.ws.transport.DeferredTransportPipe.processRequest(DeferredTransportPipe.java:121)
        at com.sun.xml.ws.api.pipe.Fiber.__doRun(Fiber.java:866)
        at com.sun.xml.ws.api.pipe.Fiber._doRun(Fiber.java:815)
        at com.sun.xml.ws.api.pipe.Fiber.doRun(Fiber.java:778)
        at com.sun.xml.ws.api.pipe.Fiber.runSync(Fiber.java:680)

Note: “SOAPFaultException: java.lang.NullPointerException” & “javax.xml.ws.WebServiceException: java.io.InterruptedIOException” can be ignored as in my case we saw this after a huge number of broken pipes exceptions and at the time when the server load and heap free was way too below threshold. Here we are not investigated above 2 errors assuming the server was heavily loaded and not able to process request but in your case you do take it into consideration.

Usually this socket exception and broken pipe comes because of database connection downtime or overload or long pending idle connections which never got released jamming the con. pool. This(SocketException: Write failed: Broken pipe) is caused by writing to a connection when the other end has already closed it.So you have a poorly defined or implemented application protocol.

Broken pipe simply means that the connection has failed. It is reasonable to assume that this is unrecoverable, and to then perform any required cleanup actions (closing connections, etc). If you are using non-blocking mode then the SocketChannel.connect method will return false, and you will need to use the isConnectionPending and finishConnect methods to insure that the connection is complete.

Usually once the server becomes way too slow you will notice IO related errors in all other working servers too. So instead of getting confused first check from where all started. Usually for quick fix one bounce is enough but its always good to get DB analysis with server IO checks for supresing the error.

Even sometimes uploading a bigger sized file can cause broken pipe errors,If server receives message even with in the limit, but while unmarshalling data, application exceeds heap size, then server will drop the connection established with client abruptly with OutOfMemoryException. Since connection got closed then we do get IOException as above.

Note:Broken pipe message comes for anytime stream could not able to write/read data from file(socket). In general, heart beat will be delivered by server/client in stipulated format. If this max number of attempts cross then application assumes that socket is closed by other party and throws this exception.

To solve broken pipe errors while uploading a bigger sized attachment we can follow below steps for weblogic:

In weblogic server console or config xml, we can set maximum message size only to the http protocol or t3 or all the protocols.

Environment > servers > soa_server > protocols > General > change Maximum Message Size as below

<max-message-size>100000000</max-message-size>
<max-t3-message-size>100000000</max-t3-message-size>
<max-http-message-size>100000000</max-http-message-size>

The above code set 100mb for these properties. These configuration helps to intimate the server that maximum message of receiving capacity. There is no configuration required to be configured to send message from server.

What about client side, to receive 100 mb message?

Yes, we have to set weblogic.MaxMessageSize config property in command line.

-Dweblogic.MaxMessageSize=100000000

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.