Ceph RBD With LXC Containers

laurentbarbe

A simple way to secure your data with containers is to use a distributed storage such as Ceph for LXC root storage.

For exemple :

1

# lxc-create -n mycontainer -t debian -B rbd --pool rbd --rbd mycontainer --fstype ext4 --fssize 500

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42

mke2fs 1.42.5 (29-Jul-2012)
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=4096 blocks, Stripe width=4096 blocks
128016 inodes, 512000 blocks
25600 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67633152
63 block groups
8192 blocks per group, 8192 fragments per group
2032 inodes per group
Superblock backups stored on blocks:
  8193, 24577, 40961, 57345, 73729, 204801, 221185, 401409

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done 

Note: Usually the template option is called with a configuration
file option too, mostly to configure the network.
For more information look at lxc.conf (5)

debootstrap is /usr/sbin/debootstrap
Checking cache download in /var/cache/lxc/debian/rootfs-wheezy-amd64 ...
Copying rootfs to /var/lib/lxc/mycontainer/rootfs...Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing
update-rc.d: using dependency based boot sequencing

Current default time zone: 'America/New_York'
Local time is now:      Tue Nov 18 09:34:16 EST 2014.
Universal Time is now:  Tue Nov 18 14:34:16 UTC 2014.

Root password is 'root', please change !
'debian' template installed
'mycontainer' created

1
2

# mount | grep mycontainer
/dev/rbd1 on /var/lib/lxc/mycontainer/rootfs type ext4 (rw,relatime,stripe=4096,data=ordered)

Diff file for lxc-create :

1

# diff -u /usr/bin/lxc-create.orig /usr/bin/lxc-create

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73

--- /usr/bin/lxc-create.orig    2014-11-17 04:16:41.181942000 -0500
+++ /usr/bin/lxc-create  2014-11-17 04:35:27.225942000 -0500
@@ -24,6 +24,7 @@
     echo "usage: lxc-create -n <name> [-f configuration] [-t template] [-h] [fsopts] -- [template_options]"
     echo "   fsopts: -B none"
     echo "   fsopts: -B lvm [--lvname lvname] [--vgname vgname] [--fstype fstype] [--fssize fssize]"
+    echo "   fsopts: -B rbd [--pool poolname] [--rbd rbd] [--fstype fstype] [--fssize fssize]"
     echo "   fsopts: -B btrfs"
     echo "           flag is not necessary, if possible btrfs support will be used"
 #    echo "   fsopts: -B union [--uniontype overlayfs]"
@@ -64,7 +65,7 @@
 }

 shortoptions='hn:f:t:B:'
-longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,fssize:'
+longoptions='help,name:,config:,template:,backingstore:,fstype:,lvname:,vgname:,pool:,rbd:,fssize:'
 localstatedir=/var
 lxc_path=${localstatedir}/lib/lxc
 bindir=/usr/bin
@@ -119,6 +120,16 @@
      vgname=$1
      shift
      ;;
+        --pool)
+        shift
+        pool=$1
+        shift
+        ;;
+        --rbd)
+        shift
+        rbd=$1
+        shift
+        ;;
      --fstype)
      shift
      fstype=$1
@@ -161,7 +172,7 @@
 fi

 case "$backingstore" in
- lvm|none|btrfs|_unset) :;;
+    lvm|rbd|none|btrfs|_unset) :;;
     *) echo "'$backingstore' is not known ('none', 'lvm', 'btrfs')"
         usage
         exit 1
@@ -216,6 +227,13 @@
         echo "please delete it (using \"lvremove $rootdev\") and try again"
         exit 1
     fi
+elif [ "$backingstore" = "rbd" ]; then
+    which rbd > /dev/null
+    if [ $? -ne 0 ]; then
+        echo "rbd command not found. Please install ceph-common package"
+        exit 1
+    fi
+    rootdev=/dev/rbd/$pool/$rbd
 elif [ "$backingstore" = "btrfs" ]; then
     mkdir "$lxc_path/$lxc_name"
     if ! out=$(btrfs subvolume create "$rootfs" 2>&1); then
@@ -257,6 +275,14 @@
     mkfs -t $fstype $rootdev || exit 1
     mount -t $fstype $rootdev $rootfs
 fi
+
+if [ $backingstore = "rbd" ]; then
+    [ -d "$rootfs" ] || mkdir $rootfs
+    rbd create $pool/$rbd --size=$fssize || exit 1
+    rbd map $pool/$rbd || exit 1
+    mkfs -t $fstype $rootdev || exit 1
+    mount -t $fstype $rootdev $rootfs
+fi

 if [ ! -z $lxc_template ]; then

If you want to make persistent after reboot, you must add rbd in /etc/ceph/rbdmap and add line in fstab.