Skip to content
English
My Tickets
  • There are no suggestions because the search field is empty.

MongdoDB failed on 2/3 VMs in High Availability

Original Question or Issue:

This morning we got this error on filecloud: Database Error: Database is not Initialized, please check if database is running (HA Environment)

Environment:

  • Product - FileCloud Server
  • Version - Any
  • Platform - Any

Steps to Reproduce:

 

Error or Log Message:

       {
      _id: 0,
      name: 'IP:27017',
      health: 1,
      state: 3,
      stateStr: 'RECOVERING',
      uptime: 23792,
      optime: { ts: Timestamp({ t: 1748470343, i: 201 }), t: Long('28') },
      optimeDurable: { ts: Timestamp({ t: 1748470343, i: 201 }), t: Long('28') },
      optimeDate: ISODate('2025-05-28T22:12:23.000Z'),
      optimeDurableDate: ISODate('2025-05-28T22:12:23.000Z'),
      lastAppliedWallTime: ISODate('2025-05-28T22:12:23.652Z'),
      lastDurableWallTime: ISODate('2025-05-28T22:12:23.652Z'),
      lastHeartbeat: ISODate('2025-06-18T23:33:49.084Z'),
      lastHeartbeatRecv: ISODate('2025-06-18T23:33:49.046Z'),
      pingMs: Long('0'),
      lastHeartbeatMessage: '',
      syncSourceHost: '',
      syncSourceId: -1,
      infoMessage: '',
      configVersion: 5,
      configTerm: 32
    },

Defect or Enhancement Number:

 

Cause:

DB server 1 is still in RECOVERING state for almost a month

Resolution or Workaround:

  • Checked the replica set status using rs.status()

  • DB Node 1 is in RECOVERING state and has not caught up, with the last sync dated May 28, 2025

  • DB Nodes 2 and 3 are in a healthy state, allowing the site to remain accessible

  • Created a full server snapshot of DB Node 1

  • Stopped the MongoDB service on DB Node 1: service mongod stop

  • Renamed the existing data directory from /var/lib/mongodb to /var/lib/mongodb_24062025 on DB Node 1

  • Created a new folder: /var/lib/mongodb on DB Node 1

  • Set the correct permissions for the MongoDB user on the new data directory:
    chown -R mongodb:mongodb /var/lib/mongodb

  • Started the MongoDB service on DB Node 1: service mongod start

  • Checked the replication status; Node 1 is currently in the STARTUP2 state, which is expected as it is undergoing initial sync (copying data from another replica set member)

Notes: