1. Documentation

Complete Example of Dynamic Provisioning Using Dedicated GlusterFS

Overview

This example assumes a functioning OpenShift Origin cluster along with Heketi and GlusterFS. All oc commands are executed on the OpenShift Origin master host.

Container Native Storage (CNS) using GlusterFS and Heketi is a great way to perform dynamic provisioning for shared filesystems in a Kubernetes-based cluster like OpenShift Origin. However, if an existing, dedicated Gluster cluster is available external to the OpenShift Origin cluster, you can also provision storage from it rather than a containerized GlusterFS implementation.

This example:

  • Shows how simple it is to install and configure a Heketi server to work with OpenShift Origin to perform dynamic provisioning.

  • Assumes some familiarity with Kubernetes and the Kubernetes Persistent Storage model.

  • Assumes you have access to an existing, dedicated GlusterFS cluster that has raw devices available for consumption and management by a Heketi server. If you do not have this, you can create a three node cluster using your virtual machine solution of choice. Ensure sure you create a few raw devices and give plenty of space (at least 100GB recommended). See Red Hat Gluster Storage Installation Guide.

Environment and Prerequisites

This example uses the following environment and prerequisites:

  • GlusterFS cluster running Red Hat Gluster Storage (RHGS) 3.1. Three nodes, each with at least two 100GB RAW devices:

    • gluster23.rhs (192.168.1.200)

    • gluster24.rhs (192.168.1.201)

    • gluster25.rhs (192.168.1.202)

  • Heketi service/client node running Red Hat Enterprise Linux (RHEL) 7.x or RHGS 3.1. Heketi can be installed on one of the Gluster nodes:

    • glusterclient2.rhs (192.168.1.203)

  • OpenShift Origin node. This example uses an all-in-one OpenShift Origin cluster (master and node on a single host), though it can work using a standard, multi-node cluster as well.

    • k8dev2.rhs (192.168.1.208)

Installing and Configuring Heketi

Heketi is used to manage the Gluster cluster storage (adding volumes, removing volumes, etc.). As stated, this can be RHEL or RHGS, and can be installed on one of the existing Gluster storage nodes. This example uses a stand-alone RHGS 3.1 node running Heketi.

The Red Hat Gluster Storage Administration Guide can be used a reference during this process.

  1. Install Heketi and the Heketi client. From the host designated to run Heketi and the Heketi client, run:

    # yum install heketi heketi-client -y

    The Heketi server can be any of the existing hosts, though typically this will be the OpenShift Origin master host. This example, however, uses a separate host not part of the GlusterFS or OpenShift Origin cluster.

  2. Create and install Heketi private keys on each GlusterFS cluster node. From the host that is running Heketi:

    # ssh-keygen -f /etc/heketi/heketi_key -t rsa -N ''
# ssh-copy-id -i /etc/heketi/heketi_key.pub root@gluster23.rhs
# ssh-copy-id -i /etc/heketi/heketi_key.pub root@gluster24.rhs
# ssh-copy-id -i /etc/heketi/heketi_key.pub root@gluster25.rhs
# chown heketi:heketi /etc/heketi/heketi_key*

  3. Edit the /usr/share/heketi/heketi.json file to setup the SSH executor. Below is an excerpt from the /usr/share/heketi/heketi.json file; the parts to configure are the executor and SSH sections:

            "executor": "ssh", (1)

        "_sshexec_comment": "SSH username and private key file information",
        "sshexec": {
            "keyfile": "/etc/heketi/heketi_key", (2)
            "user": "root", (3)
            "port": "22", (4)
            "fstab": "/etc/fstab" (5)
        },
    1 Change executor from mock to ssh.
    2 Add in the public key directory specified in previous step.
    3 Update user to a user that has sudo or root access.
    4 Set port to 22 and remove all other text.
    5 Set fstab to the default, /etc/fstab and remove all other text.

  4. Restart and enable service:

    # systemctl restart heketi
# systemctl enable heketi

  5. Test the connection to Heketi:

    # curl http://glusterclient2.rhs:8080/hello
Hello from Heketi

  6. Set an environment variable for the Heketi server:

    # export HEKETI_CLI_SERVER=http://glusterclient2.rhs:8080

Loading Topology

Topology is used to tell Heketi about the environment and what nodes and devices it will manage.

Heketi is currently limited to managing raw devices only. If a device is already a Gluster volume, it will be skipped and ignored.

  1. Create and load the topology file. There is a sample file located in /usr/share/heketi/toplogy-sample.json by default, or /etc/heketi depending on how it was installed.

    {
  "clusters": [
    {
      "nodes": [
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster23.rhs"
              ],
              "storage": [
                "192.168.1.200"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster24.rhs"
              ],
              "storage": [
                "192.168.1.201"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
        {
          "node": {
            "hostnames": {
              "manage": [
                "gluster25.rhs"
              ],
              "storage": [
                "192.168.1.202"
              ]
            },
            "zone": 1
          },
          "devices": [
            "/dev/sde",
            "/dev/sdf"
          ]
        },
      ]
    }
  ]
}

  2. Using heketi-cli, run the following command to load the topology of your environment.

    # heketi-cli topology load --json=topology.json

    	Found node gluster23.rhs on cluster bdf9d8ca3fa269ff89854faf58f34b9a
   		Adding device /dev/sde ... OK
   	 	Adding device /dev/sdf ... OK
    	Creating node gluster24.rhs ... ID: 8e677d8bebe13a3f6846e78a67f07f30
   	 	Adding device /dev/sde ... OK
   	 	Adding device /dev/sdf ... OK
...
...

  3. Create a Gluster volume to verify Heketi:

    # heketi-cli volume create --size=50

  4. View the volume information from one of the the Gluster nodes:

    # gluster volume info

	Volume Name: vol_335d247ac57ecdf40ac616514cc6257f (1)
	Type: Distributed-Replicate
	Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
	Status: Started
...
...
    1 Volume created by heketi-cli.

Dynamically Provision a Volume

  1. Create a StorageClass object definition. The definition below is based on the minimum requirements needed for this example to work with OpenShift Origin. See Dynamic Provisioning and Creating Storage Classes for additional parameters and specification definitions.

    kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  name: gluster-dyn
provisioner: kubernetes.io/glusterfs
parameters:
  resturl: "http://glusterclient2.rhs:8080" (1)
  restauthenabled: "false" (2)
    1 The Heketi server from the HEKETI_CLI_SERVER environment variable.
    2 Since authentication is not turned on in this example, set to false.

  2. From the OpenShift Origin master host, create the storage class:

    # oc create -f glusterfs-storageclass1.yaml
storageclass "gluster-dyn" created

  3. Create a persistent volume claim (PVC), requesting the newly-created storage class. For example:

    apiVersion: v1
kind: PersistentVolumeClaim
metadata:
 name: gluster-dyn-pvc
 annotations:
   volume.beta.kubernetes.io/storage-class: gluster-dyn
spec:
 accessModes:
  - ReadWriteMany
 resources:
   requests:
        storage: 30Gi

  4. From the OpenShift Origin master host, create the PVC:

    # oc create -f glusterfs-pvc-storageclass.yaml
persistentvolumeclaim "gluster-dyn-pvc" created

  5. View the PVC to see that the volume was dynamically created and bound to the PVC:

    # oc get pvc
NAME          	STATUS	VOLUME                                 		CAPACITY   	ACCESSMODES   	STORAGECLASS   	AGE
gluster-dyn-pvc Bound	pvc-78852230-d8e2-11e6-a3fa-0800279cf26f   	30Gi   		RWX       	gluster-dyn	42s

  6. Verify and view the new volume on one of the Gluster nodes:

    # gluster volume info

	Volume Name: vol_335d247ac57ecdf40ac616514cc6257f (1)
	Type: Distributed-Replicate
	Volume ID: 75be7940-9b09-4e7f-bfb0-a7eb24b411e3
	Status: Started
        ...
	Volume Name: vol_f1404b619e6be6ef673e2b29d58633be (2)
	Type: Distributed-Replicate
	Volume ID: 7dc234d0-462f-4c6c-add3-fb9bc7e8da5e
	Status: Started
	Number of Bricks: 2 x 2 = 4
	...
    1 Volume created by heketi-cli.
    2 New dynamically created volume triggered by Kubernetes and the storage class.

Creating a NGINX Pod That Uses the PVC

At this point, you have a dynamically created GlusterFS volume bound to a PVC. You can now now utilize this PVC in a pod. In this example, create a simple NGINX pod.

  1. Create the pod object definition:

    apiVersion: v1
kind: Pod
metadata:
  name: gluster-pod1
  labels:
    name: gluster-pod1
spec:
  containers:
  - name: gluster-pod1
    image: gcr.io/google_containers/nginx-slim:0.8
    ports:
    - name: web
      containerPort: 80
    securityContext:
      privileged: true
    volumeMounts:
    - name: gluster-vol1
      mountPath: /usr/share/nginx/html
  volumes:
  - name: gluster-vol1
    persistentVolumeClaim:
      claimName: gluster-dyn-pvc  (1)
    1 The name of the PVC created in the previous step.

  2. From the OpenShift Origin master host, create the pod:

    # oc create -f nginx-pod.yaml
pod "gluster-pod1" created

  3. View the pod. Give it a few minutes, as it might need to download the image if it does not already exist:

    # oc get pods -o wide
NAME                               READY     STATUS    RESTARTS   AGE       IP               NODE
gluster-pod1                       1/1       Running   0          9m        10.38.0.0        node1

  4. Now remote into the container with oc exec and create an index.html file:

    # oc exec -ti gluster-pod1 /bin/sh
$ cd /usr/share/nginx/html
$ echo 'Hello World from GlusterFS!!!' > index.html
$ ls
index.html
$ exit

  5. Now curl the URL of the pod:

    # curl http://10.38.0.0
Hello World from GlusterFS!!!