Error from fcntl() for file locking, Resource temporarily unavailable
While troubleshooting a issue in weblogic server we found that there were huge number of .DAT files which were corrupted and lots of *.lok files were present. Due to this we were not able to start the Admin or managed servers. To start all the servers we had to rename the files to something else so that server will recreate it freshly.
The logs we use to see while starting any server after crash
<Sep 30, 2013 12:33:13 AM PDT> <Critical> <Health> <BEA-310006> <Critical Subsystem PersistentStore._WLS_AdminServer has failed. Setting server state to FAILED. Reason: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.> <Sep 30, 2013 12:33:13 AM PDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: There are 1 nested errors: weblogic.management.DeploymentException: at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:129) at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191) at weblogic.store.admin.DefaultStoreService.start(DefaultStoreService.java:60) at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11 at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method) at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54) at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179) at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112) at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168) at weblogic.store.io.file.Heap.openStoreFile(Heap.java:398) at weblogic.store.io.file.Heap.open(Heap.java:325) at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117) at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435) at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423) at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126) ... 5 more <Sep 30, 2013 12:33:13 AM PDT> <Critical> <Health> <BEA-310006> <Critical Subsystem PersistentStore._WLS_AdminServer has failed. Setting server state to FAILED. Reason: weblogic.store.PersistentStoreFatalException: [Store:280105]The persistent file store "_WLS_AdminServer" cannot open file _WLS_ADMINSERVER000000.DAT.> <Sep 30, 2013 12:33:13 AM PDT> <Critical> <WebLogicServer> <BEA-000362> <Server failed. Reason: There are 1 nested errors: weblogic.management.DeploymentException: at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:129) at weblogic.store.admin.FileAdminHandler.activate(FileAdminHandler.java:191) at weblogic.store.admin.DefaultStoreService.start(DefaultStoreService.java:60) at weblogic.t3.srvr.SubsystemRequest.run(SubsystemRequest.java:64) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221) Caused by: java.io.IOException: Error from fcntl() for file locking, Resource temporarily unavailable, errno=11 at weblogic.store.io.file.direct.DirectIONative.openConsiderLock(Native Method) at weblogic.store.io.file.direct.DirectFileChannel.<init>(DirectFileChannel.java:54) at weblogic.store.io.file.direct.DirectIOManager.open(DirectIOManager.java:179) at weblogic.store.io.file.StoreFile.openInternal(StoreFile.java:112) at weblogic.store.io.file.StoreFile.openDirect(StoreFile.java:168) at weblogic.store.io.file.Heap.openStoreFile(Heap.java:398) at weblogic.store.io.file.Heap.open(Heap.java:325) at weblogic.store.io.file.FileStoreIO.open(FileStoreIO.java:117) at weblogic.store.internal.PersistentStoreImpl.recoverStoreConnections(PersistentStoreImpl.java:435) at weblogic.store.internal.PersistentStoreImpl.open(PersistentStoreImpl.java:423) at weblogic.store.admin.AdminHandler.activate(AdminHandler.java:126) ... 5 more > <Sep 30, 2013 12:33:13 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED> <Sep 30, 2013 12:33:13 AM PDT> <Error> <WebLogicServer> <BEA-000383> <A critical service failed. The server will shut itself down> <Sep 30, 2013 12:33:13 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FORCE_SHUTTING_DOWN> <Sep 30, 2013 12:33:13 AM PDT> <Critical> <WebLogicServer> <BEA-000385> <Server health failed. Reason: health of critical service 'PersistentStore._WLS_AdminServer' failed> <Sep 30, 2013 12:33:13 AM PDT> <Notice> <WebLogicServer> <BEA-000365> <Server state changed to FAILED>
So here is the quick commands to remove the lok files and to find and move files
remove lok files:
find . -name "*.lok" -exec rm -f {} \;
To find, move and rename file:
find . -name "*.DAT" -exec rename '.DAT' '.DAT_OLD' {} \;
In case of any ©Copyright or missing credits issue please check CopyRights page for faster resolutions.