Solving Stuck Thread in Weblogic logs due to LDAP query

by · Published · Updated

Oracle Weblogic Server

If you see Stuck threads like below in the WebLogic server log then follow the steps mentioned to fix this issue.

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: ‘1’ for queue: ‘weblogic.kernel.Default (self-tuning)’ has been busy for “631” seconds working on the request “weblogic.servlet.internal.ServletRequestImpl@40411df3[
POST /webcenter/wcAuthentication/ HTTP/1.1
Connection: keep-alive
Content-Length: 89
Cache-Control: max-age=0
Origin: …
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/535.19 (KHTML, like Gecko) Chrome/18.0.76.161 Safari/535.19
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Referer: …
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
Cookie: ORA_WEBCENTERAPP_40b8_beee_296c916a23ed_preferredLang=en-AU; JSESSIONID=TnhCy0Kg3hqbB0LWJ1THyS55kJ!161220584

]”, which is more than the configured time (StuckThreadMaxTime) of “600” seconds. Stack trace:
java.lang.Object.wait(Native Method)
com.sun.jndi.ldap.Connection.readReply(Connection.java:438)
com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:611)
com.sun.jndi.ldap.LdapClient.search(LdapClient.java:534)
com.sun.jndi.ldap.LdapCtx.doSearch(LdapCtx.java:1965)
com.sun.jndi.ldap.LdapCtx.searchAux(LdapCtx.java:1827)
com.sun.jndi.ldap.LdapCtx.c_search(LdapCtx.java:1752)
com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(ComponentDirContext.java:368)
com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(PartialCompositeDirContext.java:338)
com.sun.jndi.ldap.LdapReferralContext.search(LdapReferralContext.java:639)
com.sun.jndi.ldap.LdapSearchEnumeration.getReferredResults(LdapSearchEnumeration.java:169)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreReferrals(LdapNamingEnumeration.java:345)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:208)
com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:171)
oracle.security.idm.providers.stdldap.util.DirectSearchResponse.hasNext(DirectSearchResponse.java:192)
oracle.security.idm.providers.stdldap.LDSearchResponse.hasNext(LDSearchResponse.java:78)
oracle.adf.share.security.providers.jps.JpsIdentityManagementProvider.getIdmUserList(JpsIdentityManagementProvider.java:490)
oracle.adf.share.security.providers.jps.JpsIdentityManagementProvider.getUserProfileList(JpsIdentityManagementProvider.java:370)
oracle.adf.share.security.identitymanagement.UserManager.getUserProfileList(UserManager.java:253)
oracle.adf.share.security.identitymanagement.UserProfile.initialize(UserProfile.java:92)
oracle.adf.share.security.identitymanagement.UserProfile.<init>(UserProfile.java:82)
oracle.adf.share.security.providers.jps.JpsSecurityContext.getUserProfile(JpsSecurityContext.java:109)
oracle.portlet.client.adapter.adf.ADFPortletContainerExternalContext.getUserProfile(ADFPortletContainerExternalContext.java:275)
oracle.portlet.client.adapter.adf.ADFPortletContainerExternalContext.<init>(ADFPortletContainerExternalContext.java:235)
oracle.adfinternal.view.faces.portlet.FacesPortletContainerExternalContext.<init>(FacesPortletContainerExternalContext.java:42)
sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
java.lang.reflect.Constructor.newInstance(Constructor.java:513)
oracle.portlet.client.adapter.adf.ADFAdapter$Context.createExternalContext(ADFAdapter.java:1056)
oracle.portlet.client.adapter.adf.ADFAdapter$Context.createPortletContainer(ADFAdapter.java:1079)
oracle.portlet.client.adapter.adf.ADFAdapter$Context.<init>(ADFAdapter.java:876)
oracle.portlet.client.adapter.adf.ADFAdapter.createContext(ADFAdapter.java:674)
oracle.portlet.client.adapter.adf.ADFAdapter.getContext(ADFAdapter.java:544)
oracle.portlet.client.adapter.adf.ADFAdapter.init(ADFAdapter.java:347)
oracle.portlet.client.adapter.adf.ADFPortletSessionListener.getPortletContainer(ADFPortletSessionListener.java:16)
oracle.portlet.client.adapter.base.BasePortletContainerSessionListener.sessionDestroyed(BasePortletContainerSessionListener.java:34)
weblogic.servlet.internal.EventsManager.notifySessionLifetimeEvent(EventsManager.java:276)
weblogic.servlet.internal.session.SessionData.remove(SessionData.java:971)
weblogic.servlet.internal.session.MemorySessionContext.invalidateSession(MemorySessionContext.java:69)
weblogic.servlet.internal.session.SessionData.invalidate(SessionData.java:864)
weblogic.servlet.internal.ServletRequestImpl$SessionHelper.updateSessionId(ServletRequestImpl.java:3009)
weblogic.servlet.security.ServletAuthentication.generateNewSessionID(ServletAuthentication.java:626)
oracle.webcenter.webcenterapp.internal.view.webapp.WebCenterSecurityCheckServlet.doPost(WebCenterSecurityCheckServlet.java:119)
javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.adf.share.http.ServletADFFilter.doFilter(ServletADFFilter.java:62)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.wcps.client.PersonalizationFilter.doFilter(PersonalizationFilter.java:75)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.webcenter.content.integration.servlets.ContentServletFilter.doFilter(ContentServletFilter.java:168)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.adf.library.webapp.LibraryFilter.doFilter(LibraryFilter.java:175)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.security.jps.ee.http.JpsAbsFilter$1.run(JpsAbsFilter.java:111)
java.security.AccessController.doPrivileged(Native Method)
oracle.security.jps.util.JpsSubject.doAsPrivileged(JpsSubject.java:313)
oracle.security.jps.ee.util.JpsPlatformUtil.runJaasMode(JpsPlatformUtil.java:413)
oracle.security.jps.ee.http.JpsAbsFilter.runJaasMode(JpsAbsFilter.java:94)
oracle.security.jps.ee.http.JpsAbsFilter.doFilter(JpsAbsFilter.java:161)
oracle.security.jps.ee.http.JpsFilter.doFilter(JpsFilter.java:71)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
oracle.dms.servlet.DMSServletFilter.doFilter(DMSServletFilter.java:136)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.RequestEventsFilter.doFilter(RequestEventsFilter.java:27)
weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>

<Warning> <oracle.adf.share.security> <BEA-000000> <WARN_NO_USERS_PATTERN
oracle.security.idm.OperationFailureException: javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: dns.tp.fc.local:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ‘dc=obc,dc=local’]; remaining name ”]
at oracle.security.idm.providers.stdldap.util.LDAPRealm.throwException(LDAPRealm.java:773)
at oracle.security.idm.providers.stdldap.util.DirectSearchResponse.hasNext(DirectSearchResponse.java:196)
at oracle.security.idm.providers.stdldap.LDSearchResponse.hasNext(LDSearchResponse.java:78)
at oracle.adf.share.security.providers.jps.JpsIdentityManagementProvider.getIdmUserList(JpsIdentityManagementProvider.java:490)
.
.
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Caused By: javax.naming.PartialResultException [Root exception is javax.naming.ServiceUnavailableException: dns.tp.fc.local:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ‘dc=dkn,dc=local’]; remaining name ”]
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:224)
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreReferrals(LdapNamingEnumeration.java:362)
.
.
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Caused By: javax.naming.ServiceUnavailableException: dns.tp.fc.local:389; socket closed [Root exception is com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ‘dc=dkn,dc=local’]; remaining name ”
at com.sun.jndi.ldap.Connection.readReply(Connection.java:423)
at com.sun.jndi.ldap.LdapClient.getSearchReply(LdapClient.java:611)
.
.
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
Caused By: com.sun.jndi.ldap.LdapReferralException: Continuation Reference; remaining name ‘dc=dkn,dc=local’
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2860)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2820)
at com.sun.jndi.ldap.LdapNamingEnumeration.getNextBatch(LdapNamingEnumeration.java:129)
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMoreImpl(LdapNamingEnumeration.java:198)
at com.sun.jndi.ldap.LdapNamingEnumeration.hasMore(LdapNamingEnumeration.java:171)
at oracle.security.idm.providers.stdldap.util.DirectSearchResponse.hasNext(DirectSearchResponse.java:192)
at oracle.security.idm.providers.stdldap.LDSearchResponse.hasNext(LDSearchResponse.java:78)
.
.
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:178)
>
So It is taking a long time to process due to LDAP querying the referral tree.

Disable the Follow Referrals from weblogic console, Environment > Security Realms > myRealm > Providers > ADProviderName > Configuration > Provider Specific : Follow Referral

Follow Referrals specifies that a search for a user or group within the LDAP Authentication provider will follow referrals to other LDAP servers or branches within the LDAP directory. By default, this attribute is enabled.

Weblogic Follow Referrals link

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.

Tags:

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.