Mantle

Warning

Mantle is for research and development of metadata balancer algorithms, not for use on production CephFS clusters.

Multiple, active MDSs can migrate directories to balance metadata load. The policies for when, where, and how much to migrate are hard-coded into the metadata balancing module. Mantle is a programmable metadata balancer built into the MDS. The idea is to protect the mechanisms for balancing load (migration, replication, fragmentation) but stub out the balancing policies using Lua. Mantle is based on [1] but the current implementation does NOT have the following features from that paper:

  1. Balancing API: in the paper, the user fills in when, where, how much, and load calculation policies; currently, Mantle only requires that Lua policies return a table of target loads (e.g., how much load to send to each MDS)

  2. “How much” hook: in the paper, there was a hook that let the user control the fragment selector policy; currently, Mantle does not have this hook

  3. Instantaneous CPU utilization as a metric

[1] Supercomputing ‘15 Paper: http://sc15.supercomputing.org/schedule/event_detail-evid=pap168.html

Quickstart with vstart

Warning

Developing balancers with vstart is difficult because running all daemons and clients on one node can overload the system. Let it run for a while, even though you will likely see a bunch of lost heartbeat and laggy MDS warnings. Most of the time this guide will work but sometimes all MDSs lock up and you cannot actually see them spill. It is much better to run this on a cluster.

As a prerequisite, we assume you have installed mdtest or pulled the Docker image. We use mdtest because we need to generate enough load to get over the MIN_OFFLOAD threshold that is arbitrarily set in the balancer. For example, this does not create enough metadata load:

while true; do
  touch "/cephfs/blah-`date`"
done

Mantle with vstart.sh

  1. Start Ceph and tune the logging so we can see migrations happen:

cd build
../src/vstart.sh -n -l
for i in a b c; do
  bin/ceph --admin-daemon out/mds.$i.asok config set debug_ms 0
  bin/ceph --admin-daemon out/mds.$i.asok config set debug_mds 2
  bin/ceph --admin-daemon out/mds.$i.asok config set mds_beacon_grace 1500
done

  1. Put the balancer into RADOS:

bin/rados put --pool=cephfs_metadata_a greedyspill.lua ../src/mds/balancers/greedyspill.lua

  1. Activate Mantle:

bin/ceph fs set cephfs max_mds 5
bin/ceph fs set cephfs_a balancer greedyspill.lua

  1. Mount CephFS in another window:

  bin/ceph-fuse /cephfs -o allow_other &
  tail -f out/mds.a.log


Note that if you look at the last MDS (which could be a, b, or c -- it's
random), you will see an an attempt to index a nil value. This is because the
last MDS tries to check the load of its neighbor, which does not exist.

  1. Run a simple benchmark. In our case, we use the Docker mdtest image to create load:

for i in 0 1 2; do
  docker run -d \
    --name=client$i \
    -v /cephfs:/cephfs \
    michaelsevilla/mdtest \
    -F -C -n 100000 -d "/cephfs/client-test$i"
done

  1. When you are done, you can kill all the clients with:

for i in 0 1 2 3; do docker rm -f client$i; done

Output

Looking at the log for the first MDS (could be a, b, or c), we see that everyone has no load:

2016-08-21 06:44:01.763930 7fd03aaf7700  0 lua.balancer MDS0: < auth.meta_load=0.0 all.meta_load=0.0 req_rate=1.0 queue_len=0.0 cpu_load_avg=1.35 > load=0.0
2016-08-21 06:44:01.763966 7fd03aaf7700  0 lua.balancer MDS1: < auth.meta_load=0.0 all.meta_load=0.0 req_rate=0.0 queue_len=0.0 cpu_load_avg=1.35 > load=0.0
2016-08-21 06:44:01.763982 7fd03aaf7700  0 lua.balancer MDS2: < auth.meta_load=0.0 all.meta_load=0.0 req_rate=0.0 queue_len=0.0 cpu_load_avg=1.35 > load=0.0
2016-08-21 06:44:01.764010 7fd03aaf7700  2 lua.balancer when: not migrating! my_load=0.0 hisload=0.0
2016-08-21 06:44:01.764033 7fd03aaf7700  2 mds.0.bal  mantle decided that new targets={}

After the jobs starts, MDS0 gets about 1953 units of load. The greedy spill balancer dictates that half the load goes to your neighbor MDS, so we see that Mantle tries to send 1953 load units to MDS1.

2016-08-21 06:45:21.869994 7fd03aaf7700  0 lua.balancer MDS0: < auth.meta_load=5834.188908912 all.meta_load=1953.3492228857 req_rate=12591.0 queue_len=1075.0 cpu_load_avg=3.05 > load=1953.3492228857
2016-08-21 06:45:21.870017 7fd03aaf7700  0 lua.balancer MDS1: < auth.meta_load=0.0 all.meta_load=0.0 req_rate=0.0 queue_len=0.0 cpu_load_avg=3.05 > load=0.0
2016-08-21 06:45:21.870027 7fd03aaf7700  0 lua.balancer MDS2: < auth.meta_load=0.0 all.meta_load=0.0 req_rate=0.0 queue_len=0.0 cpu_load_avg=3.05 > load=0.0
2016-08-21 06:45:21.870034 7fd03aaf7700  2 lua.balancer when: migrating! my_load=1953.3492228857 hisload=0.0
2016-08-21 06:45:21.870050 7fd03aaf7700  2 mds.0.bal  mantle decided that new targets={0=0,1=976.675,2=0}
2016-08-21 06:45:21.870094 7fd03aaf7700  0 mds.0.bal    - exporting [0,0.52287 1.04574] 1030.88 to mds.1 [dir 100000006ab /client-test2/ [2,head] auth pv=33 v=32 cv=32/0 ap=2+3+4 state=1610612802|complete f(v0 m2016-08-21 06:44:20.366935 1=0+1) n(v2 rc2016-08-21 06:44:30.946816 3790=3788+2) hs=1+0,ss=0+0 dirty=1 | child=1 dirty=1 authpin=1 0x55d2762fd690]
2016-08-21 06:45:21.870151 7fd03aaf7700  0 mds.0.migrator nicely exporting to mds.1 [dir 100000006ab /client-test2/ [2,head] auth pv=33 v=32 cv=32/0 ap=2+3+4 state=1610612802|complete f(v0 m2016-08-21 06:44:20.366935 1=0+1) n(v2 rc2016-08-21 06:44:30.946816 3790=3788+2) hs=1+0,ss=0+0 dirty=1 | child=1 dirty=1 authpin=1 0x55d2762fd690]

Eventually load moves around:

2016-08-21 06:47:10.210253 7fd03aaf7700  0 lua.balancer MDS0: < auth.meta_load=415.77414300449 all.meta_load=415.79000078186 req_rate=82813.0 queue_len=0.0 cpu_load_avg=11.97 > load=415.79000078186
2016-08-21 06:47:10.210277 7fd03aaf7700  0 lua.balancer MDS1: < auth.meta_load=228.72023977691 all.meta_load=186.5606496623 req_rate=28580.0 queue_len=0.0 cpu_load_avg=11.97 > load=186.5606496623
2016-08-21 06:47:10.210290 7fd03aaf7700  0 lua.balancer MDS2: < auth.meta_load=0.0 all.meta_load=0.0 req_rate=1.0 queue_len=0.0 cpu_load_avg=11.97 > load=0.0
2016-08-21 06:47:10.210298 7fd03aaf7700  2 lua.balancer when: not migrating! my_load=415.79000078186 hisload=186.5606496623
2016-08-21 06:47:10.210311 7fd03aaf7700  2 mds.0.bal  mantle decided that new targets={}

Implementation Details

Most of the implementation is in MDBalancer. Metrics are passed to the balancer policies via the Lua stack and a list of loads is returned back to MDBalancer. It sits alongside the current balancer implementation and it’s enabled with a Ceph CLI command (“ceph fs set cephfs balancer mybalancer.lua”). If the Lua policy fails (for whatever reason), we fall back to the original metadata load balancer. The balancer is stored in the RADOS metadata pool and a string in the MDSMap tells the MDSs which balancer to use.

Exposing Metrics to Lua

Metrics are exposed directly to the Lua code as global variables instead of using a well-defined function signature. There is a global “mds” table, where each index is an MDS number (e.g., 0) and each value is a dictionary of metrics and values. The Lua code can grab metrics using something like this:

mds[0]["queue_len"]

This is in contrast to cls-lua in the OSDs, which has well-defined arguments (e.g., input/output bufferlists). Exposing the metrics directly makes it easier to add new metrics without having to change the API on the Lua side; we want the API to grow and shrink as we explore which metrics matter. The downside of this approach is that the person programming Lua balancer policies has to look at the Ceph source code to see which metrics are exposed. We figure that the Mantle developer will be in touch with MDS internals anyways.

The metrics exposed to the Lua policy are the same ones that are already stored in mds_load_t: auth.meta_load(), all.meta_load(), req_rate, queue_length, cpu_load_avg.

Compile/Execute the Balancer

Here we use lua_pcall instead of lua_call because we want to handle errors in the MDBalancer. We do not want the error propagating up the call chain. The cls_lua class wants to handle the error itself because it must fail gracefully. For Mantle, we don’t care if a Lua error crashes our balancer – in that case, we will fall back to the original balancer.

The performance improvement of using lua_call over lua_pcall would not be leveraged here because the balancer is invoked every 10 seconds by default.

Returning Policy Decision to C++

We force the Lua policy engine to return a table of values, corresponding to the amount of load to send to each MDS. These loads are inserted directly into the MDBalancer “my_targets” vector. We do not allow the MDS to return a table of MDSs and metrics because we want the decision to be completely made on the Lua side.

Iterating through tables returned by Lua is done through the stack. In Lua jargon: a dummy value is pushed onto the stack and the next iterator replaces the top of the stack with a (k, v) pair. After reading each value, pop that value but keep the key for the next call to lua_next.

Reading from RADOS

All MDSs will read balancing code from RADOS when the balancer version changes in the MDS Map. The balancer pulls the Lua code from RADOS synchronously. We do this with a timeout: if the asynchronous read does not come back within half the balancing tick interval the operation is cancelled and a Connection Timeout error is returned. By default, the balancing tick interval is 10 seconds, so Mantle will use a 5 second second timeout. This design allows Mantle to immediately return an error if anything RADOS-related goes wrong.

We use this implementation because we do not want to do a blocking OSD read from inside the global MDS lock. Doing so would bring down the MDS cluster if any of the OSDs are not responsive – this is tested in the ceph-qa-suite by setting all OSDs to down/out and making sure the MDS cluster stays active.

One approach would be to asynchronously fire the read when handling the MDS Map and fill in the Lua code in the background. We cannot do this because the MDS does not support daemon-local fallbacks and the balancer assumes that all MDSs come to the same decision at the same time (e.g., importers, exporters, etc.).

Debugging

Logging in a Lua policy will appear in the MDS log. The syntax is the same as the cls logging interface:

BAL_LOG(0, "this is a log message")

It is implemented by passing a function that wraps the dout logging framework (dout_wrapper) to Lua with the lua_register() primitive. The Lua code is actually calling the dout function in C++.

Warning and Info messages are centralized using the clog/Beacon. Successful messages are only sent on version changes by the first MDS to avoid spamming the ceph -w utility. These messages are used for the integration tests.

Testing

Testing is done with the ceph-qa-suite (tasks.cephfs.test_mantle). We do not test invalid balancer logging and loading the actual Lua VM.