Failure of server Apache Bridge Error in OHS

While doing load testing on application hosted in weblogic server with OHS as front end we saw after sometime of load test we use to see frequent “Failure of server Apache Bridge” errors from OHS when we checked in the ohs access logs we saw entries like below

127.0.0.1 - - [21/Jul/2014:08:08:16 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:08:48 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:09:20 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:09:52 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:10:24 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:10:56 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:11:28 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:12:00 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:12:32 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:13:04 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:13:36 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:14:08 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:14:40 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:15:12 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:15:44 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:16:16 -0500] "HEAD /index.html HTTP/1.1" 503 -
127.0.0.1 - - [21/Jul/2014:08:16:48 -0500] "HEAD /index.html HTTP/1.1" 503 -

When we checked the same time stamp in ohs server log we saw below errors:

[2014-06-10T06:37:34.0042-06:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [client_id: 127.0.0.1] [host_id: webhost.myhost.com] [host_addr: 171.27.11.162] [tid: 1317312832] [user: root] [ecid: 004yreNZAfAB5EpqwOjc6G0007vR0008rn] [rid: 0] [VirtualHost: main] ap_proxy: trying HEAD /index.html at backend host '172.27.113.171/7006; got exception 'CONNECTION_REFUSED [os error=115, line 1602 of URL.cpp]: apr_socket_connect call failed with error=111, host=172.27.113.171, port=7006 '

[2014-06-10T06:37:36.0164-06:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [client_id: 127.0.0.1] [host_id: webhost.myhost.com] [host_addr: 171.27.11.162] [tid: 1317312832] [user: root] [ecid: 004yreNZAfAB5EpqwOjc6G0007vR0008rn] [rid: 0] [VirtualHost: main] ap_proxy: trying HEAD /index.html at backend host '172.27.113.171/7006; got exception 'CONNECTION_REFUSED [os error=115, line 1602 of URL.cpp]: apr_socket_connect call failed with error=111, host=172.27.113.171, port=7006 '

[2014-06-10T06:37:38.0233-06:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [client_id: 127.0.0.1] [host_id: webhost.myhost.com] [host_addr: 171.27.11.162] [tid: 1317312832] [user: root] [ecid: 004yreNZAfAB5EpqwOjc6G0007vR0008rn] [rid: 0] [VirtualHost: main] ap_proxy: trying HEAD /index.html at backend host '172.27.113.171/7006; got exception 'CONNECTION_REFUSED [os error=115, line 1602 of URL.cpp]: apr_socket_connect call failed with error=111, host=172.27.113.171, port=7006 '

 

And we see below error in weblogic logs:

 

<Jul 21, 2014 7:06:20 AM CDT> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 171.27.11.162:7006, failure count: 3, failing for 3 seconds, java.net.SocketException: Too many open files>
<Jul 21, 2014 7:06:23 AM CDT> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 171.27.11.162:7006, failure count: 4, failing for 6 seconds, java.net.SocketException: Too many open files>
<Jul 21, 2014 7:06:27 AM CDT> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 171.27.11.162:7006, failure count: 5, failing for 10 seconds, java.net.SocketException: Too many open files>
<Jul 21, 2014 7:06:32 AM CDT> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 171.27.11.162:7006, failure count: 6, failing for 15 seconds, java.net.SocketException: Too many open files>
<Jul 21, 2014 7:06:38 AM CDT> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 171.27.11.162:7006, failure count: 7, failing for 21 seconds, java.net.SocketException: Too many open files>
<Jul 21, 2014 7:06:45 AM CDT> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 171.27.11.162:7006, failure count: 8, failing for 28 seconds, java.net.SocketException: Too many open files>
<Jul 21, 2014 7:06:53 AM CDT> <Critical> <Server> <BEA-002616> <Failed to listen on channel "Default" on 171.27.11.162:7006, failure count: 9, failing for 36 seconds, java.net.SocketException: Too many open files>

 

To fix this issue you have to increase the maximum open files size in your limits.conf file to maximum 65536

Follow this post to increase the maximum open files limit.

OR in case you are getting below error for long running post requests in OHS logs:

[2015-06-25T08:02:29.4091-05:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [client_id: 172.27.4.10] [host_id: OHS_HOSTNAME] [host_addr: 172.21.103.172] [tid: 47775600171328] [user: root] [ecid: 0056LSO_5ADE4UpqwOfd6G0002LG000078] [rid: 0] [VirtualHost: main] ap_proxy: trying POST /myapp/Reportsexport at backend host 172.21.103.172/7006; got exception 'Backend Server not responding'; state: reading status line or response headers from WLS (wrote? Y read? N); not failing over because method not idempotent, referer: http://OHS_HOSTNAME/REPORT_URL_MASKED&report_type=1&start_date=06/25/1995 18:03:58&end_date=06/25/2015 18:04:03&business_unit=&cost_center=&company=&region=&country=&state=&city=&county=&legal_entity=

[2015-06-25T08:14:04.0860-05:00] [OHS] [ERROR:32] [OHS-9999] [core.c] [client_id: 172.27.4.10] [host_id: OHS_HOSTNAME] [host_addr: 172.21.103.172] [tid: 47775757519168] [user: root] [ecid: 0056LT21S4mE4UpqwOfd6G0002LG00007n] [rid: 0] [VirtualHost: main] ap_proxy: trying POST /myapp/Reportsexport at backend host 172.21.103.172/7006; got exception 'Backend Server not responding'; state: reading status line or response headers from WLS (wrote? Y read? N); not failing over because method not idempotent, referer: http://OHS_HOSTNAME/REPORT_URL_MASKED&report_type=1&start_date=06/25/1995 18:03:58&end_date=06/25/2015 18:04:03&business_unit=&cost_center=&company=&region=&country=&state=&city=&county=&legal_entity=

 

AND in the front end you see below error:

Failure of server APACHE bridge: 
      No backend server available for connection: timed out after 10 seconds or idempotent set to OFF or method not idempotent.

 

To solve this kind of issue follow below steps:

1. Open the httpd.conf in OHS server at example location /home/oracle/Middleware/Oracle_WT1/instances/instance1/config/OHS/ohs1 folder.

2. Locate the line which contains mod_wl_ohs.conf weblogic plugin config location.

3. CD to the same location and open the mod_wl_ohs.conf file in any text editor.

4. Update below information’s to higher values according to your requirements

Update below values in between

<IfModule weblogic_module>

and

</IfModule>

to some higher values according to your requirement.

WLIOTimeoutSecs 600
Idempotent OFF

Note : Here we have kept it to 600 seconds or 10 minutes, default is 5minutes or 300 secs.

5. Once above changes are completed validate the configuration using below command and restart the OHS services to make the change take effect.

/home/oracle/Middleware/Oracle_WT1/instances/instance1/bin/opmnctl validate

/home/oracle/Middleware/Oracle_WT1/instances/instance1/bin/opmnctl stopall

/home/oracle/Middleware/Oracle_WT1/instances/instance1/bin/opmnctl startall

 

For reference on available weblogic plugin parameters you can refer this Oracle Link

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.