Skip to content
English
Manage and Create Tickets
  • There are no suggestions because the search field is empty.

MongdoDB failed on 2/3 VMs in High Availability

Original Question or Issue:

FileCloud Server returns the error "Database Error: Database is not Initialized, please check if database is running" in a High Availability (HA) environment when one MongoDB replica set node is stuck in a RECOVERING state and is unable to rejoin the replica set.

Environment:

  • Product - FileCloud Server
  • Version - Any
  • Platform - Any

Steps to Reproduce:

 

Error or Log Message:

       {
      _id: 0,
      name: 'IP:27017',
      health: 1,
      state: 3,
      stateStr: 'RECOVERING',
      uptime: 23792,
      optime: { ts: Timestamp({ t: 1748470343, i: 201 }), t: Long('28') },
      optimeDurable: { ts: Timestamp({ t: 1748470343, i: 201 }), t: Long('28') },
      optimeDate: ISODate('2025-05-28T22:12:23.000Z'),
      optimeDurableDate: ISODate('2025-05-28T22:12:23.000Z'),
      lastAppliedWallTime: ISODate('2025-05-28T22:12:23.652Z'),
      lastDurableWallTime: ISODate('2025-05-28T22:12:23.652Z'),
      lastHeartbeat: ISODate('2025-06-18T23:33:49.084Z'),
      lastHeartbeatRecv: ISODate('2025-06-18T23:33:49.046Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 5,
      configTerm: 32
    },

Defect or Enhancement Number:

 

Cause:

A MongoDB replica set member (DB Node 1) has been stuck in the RECOVERING state for an extended period and cannot catch up with the primary node's oplog. Because the oplog gap is too large for automatic recovery, the node must be reseeded via an initial sync.

Resolution or Workaround:

  • Checked the replica set status using rs.status()

  • DB Node 1 is in RECOVERING state and has not caught up with the replica set primary. Note the date in the optimeDate field from the rs.status() output — if it is weeks or more in the past, automatic recovery is not possible and a reseed is required May 28, 2025

  • DB Nodes 2 and 3 are in a healthy state, allowing the site to remain accessible

  • Created a full server snapshot of DB Node 1

  • Stopped the MongoDB service on DB Node 1: service mongod stop

  • Renamed the existing data directory from /var/lib/mongodb to /var/lib/mongodb_24062025 on DB Node 1

  • Created a new folder: /var/lib/mongodb on DB Node 1

  • Set the correct permissions for the MongoDB user on the new data directory:
    chown -R mongodb:mongodb /var/lib/mongodb

  • Started the MongoDB service on DB Node 1: service mongod start

  • Checked the replication status; Node 1 should now appear in the STARTUP2 state, which is expected as it is undergoing initial sync (copying data from another replica set member)

Notes:- The service mongod commands shown (service mongod stop / service mongod start) apply to Linux systems using SysVinit or a compatibility layer. On systems using native systemd, use systemctl stop mongod and systemctl start mongod instead. - After Node 1 completes initial sync, it will transition from STARTUP2 to SECONDARY state. Verify this using rs.status() before considering the recovery complete. - The renamed backup directory (/var/lib/mongodb_24062025) can be safely deleted once Node 1 has fully rejoined the replica set as SECONDARY and replication lag is confirmed to be zero. - If Node 1 does not transition out of STARTUP2 within a reasonable time (depending on database size), check the MongoDB logs at /var/log/mongodb/mongod.log for errors.