weblogic.store.PersistentStoreFatalException

We faced a very weird scenario where all our servers went down/crashed due to space issue and when we added free space and tried to start the Adminserver and managed servers we faced weblogic.store.PersistentStoreFatalException errors like below and servers went shutdown itself.

After checking for few things we finally found a quick solution to fix weblogic.store.PersistentStoreFatalException errors. So here we are listing out multiple types of weblogic.store.PersistentStoreFatalException seen and steps to solve the same.

Error Type 1:

<Jul 25, 2012 10:54:45 PM PDT> <Error> <Store> <BEA-280061> <The persistent store "AGJMSFileStore_auto_1" could not be deployed: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "AGJMSFileStore_auto_1" cannot open file AGJMSFILESTORE_AUTO_1000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "AGJMSFileStore_auto_1" cannot open file AGJMSFILESTORE_AUTO_1000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
Truncated. see log file for complete stacktrace
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168)
Truncated. see log file for complete stacktrace

If you see above error in logfile and the server goes down after this error then do following.

– cd to <MW_HOME>/domains/<DOMAIN_NAME>/AGJMSFileStore_auto_1/
– Delete AGJMSFILESTORE_AUTO_1000000.DAT file or make a backup of the same file.
– Clear(use rm -rf) tmp and cache folder at

<MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/tmp

and

<MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/cache

– Try starting the server again.

Error Type 2:

<Jul 25, 2012 10:37:33 PM PDT> <Emergency> <Store> <BEA-280060> <The persistent store "WLS_DIAGNOSTICS" encountered a fatal error, and it must be shut down: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
Truncated. see log file for complete stacktrace
Caused By: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
Truncated. see log file for complete stacktrace

>
No matching store mbean found for =WLS_DIAGNOSTICS, must be a default store.
<Jul 25, 2012 10:37:33 PM PDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "WLS_DIAGNOSTICS" cannot open file WLS_DIAGNOSTICS000000.DAT.
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:128)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.xa.PersistentStoreManagerXA.createFileStore(PersistentStoreManagerXA.java:117)
at weblogic.diagnostics.archive.DiagnosticStoreRepository.getStore(DiagnosticStoreRepository.java:91)
at weblogic.diagnostics.lifecycle.DiagnosticSystemService.start(DiagnosticSystemService.java:143)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:138)
at weblogic.store.io.file.StoreFile.open(StoreFile.java:161)
at weblogic.store.io.file.Heap.openStoreFile(Heap.java:401)
at weblogic.store.io.file.Heap.open(Heap.java:325)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
... 8 more

<Jul 25, 2012 10:37:36 PM PDT> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of critical service 'PersistentStore.WLS_DIAGNOSTICS' failed>
<Jul 25, 2012 10:37:36 PM PDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down>
<Jul 25, 2012 10:37:37 PM> <FINEST> <NodeManager> <Waiting for the process to die: 7155>
<Jul 25, 2012 10:37:37 PM> <INFO> <NodeManager> <Server failed during startup so will not be restarted>
<Jul 25, 2012 10:37:37 PM> <FINEST> <NodeManager> <runMonitor returned, setting finished=true and notifying waiters>

If you see above error in logfile and the server goes down after this error then do following.

– cd to

 <MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/data/store/default 

and

<MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/data/store/diagnostics

– Delete/backup(move) all the files in those two folder, let the server regerate all the corrupted files once again.
– Clear(use rm -rf) tmp and cache folder at

<MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/tmp

and

<MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/cache

– Try starting the server again.

Error Type 3:

####<Jul 25, 2012 10:31:25 PM PDT> <Error> <EmbeddedLDAP> <atgserver765> <FINANCE1> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1343280685674> <BEA-171519> <Could not obtain an exclusive lock to the embedded LDAP data files directory: /slot/ems6372/appmgr/APPTOP/instance/domains/atgserver765/ProjectsDomain/servers/FINANCE1/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.>
####<Jul 25, 2012 10:31:25 PM PDT> <Critical> <WebLogicServer> <atgserver765> <FINANCE1> <Main Thread> <<WLS Kernel>> <> <> <1343280685679> <BEA-000362> <Server failed. Reason: Could not obtain an exclusive lock to the embedded LDAP data files directory: /slot/ems6372/appmgr/APPTOP/instance/domains/atgserver765/ProjectsDomain/servers/FINANCE1/data/ldap/ldapfiles because another WebLogic Server is already using this directory. Ensure that the first WebLogic Server is completely shutdown and restart the server.>
####<Jul 25, 2012 10:31:25 PM PDT> <Error> <WebLogicServer> <atgserver765> <FINANCE1> <Main Thread> <<WLS Kernel>> <> <> <1343280685845> <BEA-000383> <A critical service failed. The server will shut itself down>

If you see above error in logfile and the server goes down after this error then do following.

– cd to <MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/data/ldap/ldapfiles
– Remove the EmbeddedLDAP.lok lock file(Example: WLS_DIAGNOSTICS000000.DAT)
– Clear(use rm -rf) tmp and cache folder at

<MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/tmp

and

<MW_HOME>/domains/<DOMAIN_NAME>/servers/<SERVERNAME>/cache

– Try starting the server again.

Error Type 4:

weblogic.store.PersistentStoreFatalException: [Store:280019]There was an error while writing to a file store file
at weblogic.store.io.file.FileStoreIO.flush(FileStoreIO.java:478)
at weblogic.store.internal.PersistentStoreImpl.run(PersistentStoreImpl.java:649)
at java.lang.Thread.run(Thread.java:595)
java.io.IOException: Error writing to file, Disc quota exceeded, errno=49
at weblogic.store.io.file.direct.DirectIONative.write(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.write(DirectFileChannel.java:172)

If you are seeing above error then follow the steps mentioned in weblogic.store.PersistentStoreFatalException Error thread.

Error Type 5:

<Sep 30, 2013 12:33:13 AM PDT> <Critical> <Health> <BEA-310006> <Critical Subsystem PersistentStore._WLS_AdminServer has failed. Setting server state to FAILED.
Reason: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.>
<Sep 30, 2013 12:33:13 AM PDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason:

There are 1 nested errors:

weblogic.management.DeploymentException:
at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:129)
at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191)
at weblogic.store.admin.DefaultStoreService.start(DefaultStoreService.java:60)
at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64)
at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256)
at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)
Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11
at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method)
at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54)
at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179)
at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112)
at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168)
at weblogic.store.io.file.Heap.openStoreFile(Heap.java:398)
at weblogic.store.io.file.Heap.open(Heap.java:325)
at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117)
at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435)
at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423)
at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126)
... 5 more

If you are seeing similar to above error in logs then follow the steps mentioned in Error from fcntl() for file locking, Resource temporarily unavailable thread.

In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.