Deploying Metadata Servers

Each CephFS file system requires at least one MDS. The cluster operator will generally use their automated deployment tool to launch required MDS servers as needed. Rook and ansible (via the ceph-ansible playbooks) are recommended tools for doing this. For clarity, we also show the systemd commands here which may be run by the deployment technology if executed on bare-metal.

See MDS Config Reference for details on configuring metadata servers.

Provisioning Hardware for an MDS

The present version of the MDS is single-threaded and CPU-bound for most activities, including responding to client requests. Even so, an MDS under the most aggressive client loads still uses about 2 to 3 CPU cores. This is due to the other miscellaneous upkeep threads working in tandem.

Even so, it is recommended that an MDS server be well provisioned with an advanced CPU with sufficient cores. Development is on-going to make better use of available CPU cores in the MDS; it is expected in future versions of Ceph that the MDS server will improve performance by taking advantage of more cores.

The other dimension to MDS performance is the available RAM for caching. The MDS necessarily manages a distributed and cooperative metadata cache among all clients and other active MDSs. Therefore it is essential to provide the MDS with sufficient RAM to enable faster metadata access and mutation.

Generally, an MDS serving a large cluster of clients (1000 or more) will use at least 64GB of cache (see also Understanding MDS Cache Size Limits). An MDS with a larger cache is not well explored in the largest known community clusters; there may be diminishing returns where management of such a large cache negatively impacts performance in surprising ways. It would be best to do analysis with expected workloads to determine if provisioning more RAM is worthwhile.

In a bare-metal cluster, the best practice is to over-provision hardware for the MDS server. Even if a single MDS daemon is unable to fully utilize the hardware, it may be desirable later on to start more active MDS daemons on the same node to fully utilize the available cores and memory. Additionally, it may become clear with workloads on the cluster that performance improves with multiple active MDS on the same node rather than over-provisioning a single MDS.

Finally, be aware that CephFS is a highly-available file system by supporting standby MDS (see also Terminology) for rapid failover. To get a real benefit from deploying standbys, it is usually necessary to distribute MDS daemons across at least two nodes in the cluster. Otherwise, a hardware failure on a single node may result in the file system becoming unavailable.

Co-locating the MDS with other Ceph daemons (hyperconverged) is an effective and recommended way to accomplish this so long as all daemons are configured to use available hardware within certain limits. For the MDS, this generally means limiting its cache size.

Adding an MDS

  1. Create an mds data point /var/lib/ceph/mds/ceph-${id}. The daemon only uses this directory to store its keyring.

  2. Edit ceph.conf and add MDS section.

    [mds.${id}]
host = {hostname}

  3. Create the authentication key, if you use CephX.

    $ sudo ceph auth get-or-create mds.${id} mon 'profile mds' mgr 'profile mds' mds 'allow *' osd 'allow *' > /var/lib/ceph/mds/ceph-${id}/keyring

  4. Start the service.

    $ sudo systemctl start mds.${id}

  5. The status of the cluster should show:

    mds: ${id}:1 {0=${id}=up:active} 2 up:standby

Removing an MDS

If you have a metadata server in your cluster that you’d like to remove, you may use the following method.

  1. (Optionally:) Create a new replacement Metadata Server. If there are no replacement MDS to take over once the MDS is removed, the file system will become unavailable to clients. If that is not desirable, consider adding a metadata server before tearing down the metadata server you would like to take offline.

  2. Stop the MDS to be removed.

    $ sudo systemctl stop mds.${id}

    The MDS will automatically notify the Ceph monitors that it is going down. This enables the monitors to perform instantaneous failover to an available standby, if one exists. It is unnecessary to use administrative commands to effect this failover, e.g. through the use of ceph mds fail mds.${id}.

  3. Remove the /var/lib/ceph/mds/ceph-${id} directory on the MDS.

    $ sudo rm -rf /var/lib/ceph/mds/ceph-${id}