Cluster

Logical Volume Manager

(CLVM)

Linux-Kongress 2004

Heinz Mauelshagen
Consulting Development Engineer

Linux-Kongress 2004, Erlangen

Cluster Logical Volume Manager

Top

Linux-Kongress 2004, Erlangen

Requirements

Software Architecture

User Interface

Metadata Locations

Project Status/Summary

URLs

Cluster Logical Volume Manager

Requirements (1)

Linux-Kongress 2004, Erlangen

CLI close to HP-UX LVM because of the ease of use
and quick learn curve

Storage devices (aka Physical Volumes or PVs)
are grouped together in…

Volume Groups (VGs; “logical disks”); total capacity of a VG is
slightly less than sum of all PV capacities because we
need to store metadata on PVs

VG-Capacity allocated dynamically to Logical Volumes
(LVs; “logical Partitions”; /dev/VGName/LVName)

Virtualisation designed as a simple 3 Layer-Model

Cluster Logical Volume Manager

Requirements (2)

Linux-Kongress 2004, Erlangen

Create PVs, VGs and LVs (linear, striped, …)

Grow and shrink VGs and LVs online

Display attributes of PVs, VGs und LVs

Relocate (parts of) LVs online

Symmetrical command model; commands can run
on arbitrary nodes

Support whatever block device Linux supports

Functionality

-> Large flexibility to adjust to varying
storage capacity and quality requirements!

Cluster Logical Volume Manager

Virtualisation: 3 Layer-Model

Linux-Kongress 2004, Erlangen

...

LV1

LV2

PV1

PVn

PV2

PV3

PV4

VG1

Cluster Logical Volume Manager

Mapping: logical -> physical

Linux-Kongress 2004, Erlangen

N != M != n

LV1

LE1

.

.

.

LE2

LE3

LEn

PV1

PE1

.

PE2

.

PEm

n != m != o
LE-Size typically 32MB

PV2

PE1

.

PE2

.

PEo

Cluster Logical Volume Manager

CLVM Software-Architecture (1)

Linux-Kongress 2004, Erlangen

Device-Mapper as the generic mapping runtime platform in
the Linux kernel (since 2.5)

can be used by multiple applications which need
block device mapping (eg, dmraid or EVMS)

manages Mapped Devices (MDs) (create, remove, …)

handles text formatted Mapping-Tables (load, unload, reload)

all table loading actions happen online to reflect changed
layouts (eg, a Mapped-Device size change) while the device
is suspended for data integrity reasons

code to handle mappings factored out into Mapping-Targets,
which handle linear, striped, mirrored, snapshot, multipath,
zero, error and crypt layouts

maps to arbitrary block devices (eg, iSCSI)

Cluster Logical Volume Manager

CLVM Software-Architecture (2)

Linux-Kongress 2004, Erlangen

Device-Mapper as the generic mapping runtime platform in
the Linux kernel

Mapping-Targets are dynamically loadable and register with
the Device-Mapper core

Mapping targets “hide” clustering (eg, Cluster Snapshots)

Mapped Devices can be stacked in order to build complex
mappings (eg, to enable pvmove via a temporary mirror)

no 256 Limit on PVs, VGs or LVs any more
(tested with thousands of LVs)

more than 2 Terabytes per LV in Linux 2.6 with
CONFIG_LBD set

comes with a user space library to be interfaced by
Volume Management applications and a test tool dmsetup

lib creates nodes to access Mapped Devices in /dev/mapper/

Cluster Logical Volume Manager

CLVM Software-Architecture (3)

Linux-Kongress 2004, Erlangen

Examples of Device-Mapper tables which can be activated
using the dmsetup tool

0 1024 linear /dev/hde1 40
1024 2048 striped 2 64 /dev/hde1 1064 /dev/hdf1 0

0 1024 zero
1024 1000 error

0 83886080 mirror core 1 64 2 /dev/sda 0 /dev/sdb 0

Cluster Logical Volume Manager

CLVM Software-Architecture (4)

Linux-Kongress 2004, Erlangen

CLVM Application, which is a superset of LVM2
to manage ondisk metadata of the PVs, VGs und LVs

user configurable with regards to device nodes to use/ignore
(Device Name Filter), logging of runtime information and more
in /etc/lvm/lvm.conf (/etc/lvm/ can be changed in environment
variable LVM_SYSTEM_DIR)

keeps list of used (filtered) block device names in
a cache file /etc/lvm/.cache for performance reasons

transforms between ondisk metadata and internal
unified representation of PVs, VGs and LVs

reads+writes LVM1 and new LVM2/CLVM (redundant metadata
copies, transaction oriented updates) formats

other metadata format handlers can be added

Cluster Logical Volume Manager

CLVM Software-Architecture (5)

Linux-Kongress 2004, Erlangen

CLVM Application to manage ondisk metadata of
the PVs, VGs und LVs

bidirectional migration of LVM1 <-> LVM2/CLVM metadata

stores one metadata backup per VG automatically in
/etc/lvm/backup/

stores multiple metadata archives per VG automatically in
/etc/lvm/archive/ (metadata history of changes)

Cluster Logical Volume Manager

CLVM Software-Architecture (6)

Linux-Kongress 2004, Erlangen

other CLVM Application functional areas

create the Device-Mapper tables for the Mapped Devices

create symlinks from /dev/VGName/LVName to the
Device-Mapper nodes in /dev/mapper/

log information

interactive CLVM shell

CLVM daemon (clvmd), which executes remote table (re)loads

clvmd (indirectly) uses cluster infrastructure for membership,
fencing and locking

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (1)

Linux-Kongress 2004, Erlangen

CLVM relies on the following cluster subsystems

CCS - Cluster Configuration System

RLM - Centralized, replicated Lock Manager
-or-

CMAN - Cluster Manager, has a connection part, which
manages membership and quorum and a service part,
which manages which nodes are
using which services (eg, CLVM)

FENCE - I/O Fencing System

GDLM - Distributed Lock Manager

Subsystems are designed to be stand-alone and
independent of CLVM or other applications (eg, GFS)

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (2)

Linux-Kongress 2004, Erlangen

RLM centralized Lock Manager

Integrates heartbeating, recovery and fencing

One master and multiple slaves

Hot standby of lock state

Quorum among lock servers enforced to
handle net splits

Lock space can be partitioned among multiple
RLM servers for better scalability in large clusters

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (3)

Linux-Kongress 2004, Erlangen

Cluster Configuration System

Provides static configuration information to other cluster
infrastructure components
- depends on no other components

User library and kernel module provide a common interface
to the CCS information

A CCS daemon (ccsd) runs on every node
- ccsd is accessed by the library to read configuration data
from variable back-end source
- Back-end is modular, data can come from shared block
device (usual), network server, local files, …

Back-end retrieves data from multiple files (cluster.ccs,
node.ccs, fence.ccs)

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (4)

Linux-Kongress 2004, Erlangen

Cluster Manager

Has two unique parts:
- Connection Manager
- Service Manager

Both parts completely symmetric and in the kernel

CMAN is standalone, its only dependency is CCS

CMAN is general, with no specific relation to any
application (eg, CLVM)

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (5)

Linux-Kongress 2004, Erlangen

Connection Manager

Tracks lifeness of nodes

List of life nodes defines a general, “global” cluster

Assigns a unique node id to each nod

A node joins the cluster by name and its name
must be in the cluster configuration file in order to
join successfully

Quorum majority voting prevents split-brain clusters

A node must be a cluster member before higher level
clustering systems can be started (eg, lock management)

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (6)

Linux-Kongress 2004, Erlangen

Service Manager

Manages Service-Groups which do have unique names

Types of Service Groups are:
- Fence Domain
- Lock Space

Tracks which nodes are using instances of services, I.e.
“Service Group” membership

Manages changes to Service Groups

When a node fails, which gets detected by the Connection
Manager, the Service Manager stops affected groups and
restarts them one level at a time

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (7)

Linux-Kongress 2004, Erlangen

I/O Fencing System

Symmetric with four parts:
- Fence Domain manager (a Service Group in the kernel)
determines which nodes to I/O fence
- Fence Driver passes fence requests from the kernel
to a user-land daemon
- Fence Daemon accepts fence requests for node, looks
up node-specific fence parameters from CCS to carry
out the operation
- Fence Agents are programs called by the daemon
to fence a node using a particular method

A node joins a global Fence Domain before using CLVM or
other cluster applications (eg, GFS)

Cluster Logical Volume Manager

Cluster Infrastructure-Architecture (8)

Linux-Kongress 2004, Erlangen

DLM distributed Lock Manager

Fully distributed and symmetric

Modular

VMS-like kernel interface

64-bytes binary resource names

Synchronous user-land interface

Hierarchical lock spaces

First node requesting a lock becomes master for that lock

Arbitrary number of lock spaces uniquely identified
by lock space name

Optional request parameters: parent lock, lock range,
lock value block

Deadlock detection by timeout

Cluster Logical Volume Manager

CLVM-Architecture Overview

Linux-Kongress 2004, Erlangen

clvmd

clvmd

tools

lib

dm

libdm

Node1

Noden

libdm

dm

target

target

target

target

target

target

lld

lld

Lm = Lock Manager(RLM/GDLM)

lld = low level driver

LM

LM

Cluster Logical Volume Manager

User Interface

Linux-Kongress 2004, Erlangen

CLI (tools) directly derived from the 3 layer-model (PV, VG, LV)
and the actions to perform (change, create, remove, extend,
reduce, rename, scan, s(how), display, move, split, merge)

A lot of tools are simply named by prefixing
the actions to perform with pv, vg or lv:

pvchange, pvcreate, pvremove, pvdisplay, pvmove, pvscan, pvs

vgchange, vgcreate, vgremove, vgdisplay,
vgextend, vgreduce, vgscan, vgimport, vgexport,
vgsplit, vgmerge, vgs

lvchange, lvcreate, lvremove, lvdisplay,
lvextend, lvreduce, lvscan, lvs

Cluster Logical Volume Manager

User Interface

Linux-Kongress 2004, Erlangen

Other supporting commands are:

fsadm

lvmdiskscan

lvm dumpconfig

lvresize

vgcfgbackup, vgcfgrestore, vgck, vgconvert, vgmknodes

All commands take -v, which can be entered multiple times
to increase the output verbosity

‘commandname --help’, ‘man lvm’ and
‘man commandname’ are your friends :-)

Cluster Logical Volume Manager

Commands (Physical Volume)

Linux-Kongress 2004, Erlangen

pvcreate:
initializes a block device to be used as a PV, which is a
prerequisite to get included in a VG

pvremove:
zeroes the CLVM metadata area on an empty PV

pvdisplay:
display PV properties (eg. size, extents, VG) in a fixed format

pvchange:
change attributes of the PV (eg, allocation)

Cluster Logical Volume Manager

Commands (Physical Volume)

Linux-Kongress 2004, Erlangen

pvmove:
moves data off a PV to (an)other PV(s) online; uses
temporary mirror to be restartable after a system failure

pvscan:
scans all filtered (lvm.conf) devices for PV metadata and lists hits

pvs (new):
format selectable, flexible output of PV properties;
neat for scripting (eg, pvs -o pv_name,pv_size -Opv_free)

Cluster Logical Volume Manager

Commands (Volume Group)

Linux-Kongress 2004, Erlangen

vgcreate:
creates a new VG by name and adds at least one PV to it

vgremove:
removes an empty VG (with no LVs in it)

vgdisplay:
display VG properties (eg. size, extents, # of PVs) in a fixed format

vgchange:
(de)activates (partial) VG(s) and changes various attributes
(eg, maximum # of PVs/LVs allowed, resizability)

vgextend:
grows a VG’s capacity by adding one or more free PV(s)

Cluster Logical Volume Manager

Commands (Volume Group)

Linux-Kongress 2004, Erlangen

vgreduce:
shrinks a VG’s capacity by removing one or more empty PV(s)
(eg, to use those in different VGs or remove them
from the system)

vgscan:
scans all filtered (lvm.conf) devices for LVM metadata and
stores all hits in the cache file, which is used by the other tools

vgimport:
makes a VG accessible on a machine after its PVs got attached

vgexport:
makes an inactive VG inaccesible In order to detach its PVs

Cluster Logical Volume Manager

Commands (Volume Group)

Linux-Kongress 2004, Erlangen

vgsplit:
splits PV(s) of a VG creating a new VG; all LVs allocated to
the PV(s) can’t have extents allocated outside those PVs
and no other LVs may have extents allocated on those
(optionally use pvmove to force this)

vgmerge:
group 2 VGs into one

vgs (new):
format selectable, flexible output of VG properties;
neat for scripting(eg, vgs -o vg_name,vg_uuid -Ovg_size)

Cluster Logical Volume Manager

Commands (Logical Volume)

Linux-Kongress 2004, Erlangen

lvcreate:
creates a new LV with layout (eg, striped) and size

lvremove:
removes an inactive LV

lvdisplay:
display LV properties (eg. size, layout, mapping) in a fixed format

lvchange:
change attributes of an LV (eg, allocation)

lvextend/lvreduce:
grow and shrink an LV; shrinking an LV makes VG capacity free
to get allocated to other LVs in the same VG

Cluster Logical Volume Manager

Commands (Logical Volume)

Linux-Kongress 2004, Erlangen

lvscan:
scan for all LVs in the system and list them

lvs (new):
format selectable, flexible output of LV properties
(eg, lvs -o lv_name,lv_attr -O-lv_name);
did I mention this is neat for scripting yet ? ;-)

Cluster Logical Volume Manager

Commands (Supporting)

Linux-Kongress 2004, Erlangen

fsadm:
resizing of an LV and the filesystem within it in one go

lvmdiskscan:
scan for block devices that may be used as PVs

lvm dumpconfig:
displays the life config on stdout which can be redirected
to a config file (eg, /etc/lvm/lvm.conf)

lvresize:
can be used instead of lvextend/lvreduce

vgcfgbackup:
creates metadata archives which are automatically created
by default on every VG configuration change

Cluster Logical Volume Manager

Commands (Supporting)

Linux-Kongress 2004, Erlangen

vgcfgrestore:
restores metadata of a VG from archive to all PVs.
May need a “pvcreate --uuid …” in case a PV can’t be found

vgck:
checks VG consistency

vgconvert:
converts LVM1 metadata format <-> CLVM
(backward convertability subject to LVM1 restrictions like
maximum # of LVs < 256 in LVM1)

vgmknodes:
creates all devnodes for VGs and LVs
(very useful in case of accidential deletion)

Cluster Logical Volume Manager

Metadata locations

Linux-Kongress 2004, Erlangen

LVM1 format is binary only and one copy is
at the very beginning of every PV

LVM2/CLVM format is an ASCII text format which is at the
beginning, after a disk label, of every PV in 2 copies by default

in large configurations, LVM2 metadata copies can optionally
be avoided on PVs (pvcreate --metadatacopies 0 /dev/..) to have
faster metadata updates on configuration changes

vgconvert is capable to inplace migrate the binary LVM1
format to LVM2 text format and back

Cluster Logical Volume Manager

Status

Linux-Kongress 2004, Erlangen

Device-Mapper in Linux 2.6 Mainline
-> Distributors started to ship it with LVM2 tools
(eg, SuSE 9.1, Red Hat Fedora)

CLVM+LVM2 compatible to LVM1 with enhancements
(eg, atomic metadata updates)

Multipathing with Device-Mapper in Linux 2.6;
Multipath-Tool available

(Ex-)Sistina GFS, CLVM and cluster infrastructure
open-sourced end June 2004

Cluster Logical Volume Manager

Summary

Linux-Kongress 2004, Erlangen

virtualizes mass storage; no restrictions to physical disk
sizes any more

enables online adjustments to varying storage requirements

supports life data migration (ie. migrate data from LVM1/2
DAS to SAN) in order to scale from single host to a storage cluster

is freely available together with the cluster infrastructure

Cluster Linux Logical Volume Manager

-> CLVM scales into large, cheap storage clusters and
reduces operational costs by shrinking the downtime
of Linux systems for storage management

Cluster Logical Volume Manager

URLs

Linux-Kongress 2004, Erlangen

ftp://ftp.kernel.org/pub/linux/kernel (better: a regional mirror)

http://sources.redhat.com/cluster (CLVM+cluster infrastructure)

http://sources.redhat.com/dm (Device-Mapper tool+library)

http://www.redhat.com/mailman/listinfo/linux-lvm
to subscribe for linux-lvm@redhat.com

http://www.redhat.com/mailman/listinfo/dm-devel
to subscribe for dm-devel@redhat.com

http://christophe.varoqui.free.fr/multipath.html

mauelshagen@redhat.com

PowerPoint-Präsentation

2004-09-24T15:38:06