MongoDB Replica Node Stuck in RECOVERING State (Linux)

Original Question or Issue:

This morning we got this error on filecloud: Database Error: Database is not Initialized, please check if database is running.

Environment:

Steps to Reproduce:

Error or Log Message:

Defect or Enhancement Number:

Cause:

DB server 1 is still in RECOVERING state for almost a month

Resolution or Workaround:

Checked the replica set status using rs.status(
DB Nodes 2 and 3 are in a healthy state, allowing the site to remain accessible
Created a full server snapshot of DB Node 1
Stopped the MongoDB service on DB Node 1: service mongod stop
Renamed the existing data directory from /var/lib/mongodb to /var/lib/mongodb_24062025 on DB Node 1
Created a new folder: /var/lib/mongodb on DB Node 1
Set the correct permissions for the MongoDB user on the new data directory:
chown -R mongodb:mongodb /var/lib/mongodb
Started the MongoDB service on DB Node 1: service mongod start
Checked the replication status; Node 1 is currently in the STARTUP2 state, which is expected as it is undergoing initial sync (copying data from another replica set member)

Notes: