Collocating Ceph volumes and instances in a multi-datacenter setup

Nov 12, 2013 loic

OpenStack Havana is installed on machines rented from OVH and Hetzner. An aggregate is created for machines hosted at OVH and another for machines hosted at Hetzner. A Ceph cluster is created with a pool using disks from OVH and another pool using disks from Hetzner. A cinder backend is created for each Ceph pool. From the dashboard, an instance can be created in the OVH availability zone using a Ceph volume provided by the matching OVH pool.

Creating availability zones ¶

The availability zones are created as a side effect of creating an aggregate.

# nova aggregate-create ovh ovh +----+------+-------------------+-------+--------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+------+-------------------+-------+--------------------------------+ | 2 | ovh | ovh | [] | {u'availability_zone': u'ovh'} | +----+------+-------------------+-------+--------------------------------+

nova aggregate-create hetzner hetzner

+----+---------+-------------------+-------+------------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+---------+-------------------+-------+------------------------------------+ | 3 | hetzner | hetzner | [] | {u'availability_zone': u'hetzner'} | +----+---------+-------------------+-------+------------------------------------+

The hosts are assigned to their availability zone:

# nova aggregate-add-host ovh bm0015.the.re Aggregate 2 has been successfully updated. +----+------+-------------------+--------------------+--------------------------------+ | Id | Name | Availability Zone | Hosts | Metadata | +----+------+-------------------+--------------------+--------------------------------+ | 2 | ovh | ovh | [u'bm0015.the.re'] | {u'availability_zone': u'ovh'} | +----+------+-------------------+--------------------+--------------------------------+

The result can be checked with

Creating the Ceph pools ¶

The crush map is extracted with

ceph osd getcrushmap -o crush.bin crushtool -d crush.bin -o crush.txt

It is edited to add

datacenter ovh { id -5 alg straw hash 0 item bm0014 weight 1.820 item bm0015 weight 1.820 }

rule ovh { ruleset 3 type replicated min_size 1 max_size 10 step take ovh step chooseleaf firstn 0 type host step emit }

and sent back to the Ceph monitors with

crushtool -c crush.txt -o crush.bin ceph osd setcrushmap crush.bin

An ovh pool is created and set to use the ovh ruleset:

ceph osd pool create ovh 128 ceph osd pool set ovh crush_ruleset 3

The crush.txt file also contains the ruleset for the hetzner pool.

Creating cinder backends ¶

In the /etc/cinder/cinder.conf file of the host running cinder-volume, one cinder backend is defined for each Ceph pool:

enabled_backends=rbd-hetzner,rbd-ovh [rbd-hetzner] volume_driver=cinder.volume.driver.RBDDriver rbd_pool=hetzner volume_backend_name=RBD_HETZNER [rbd-ovh] volume_driver=cinder.volume.driver.RBDDriver rbd_pool=ovh volume_backend_name=RBD_OVH

In order to enable the –volume-type ovh option of cinder create, the corresponding type keys must be created:

# cinder type-create ovh +--------------------------------------+------+ | ID | Name | +--------------------------------------+------+ | 48645332-4835-4a9b-9078-cd735f47dae5 | ovh | +--------------------------------------+------+

cinder type-key ovh set volume_backend_name=RBD_OVH

cinder extra-specs-list

+--------------------------------------+---------+------------------------------------------+ | ID | Name | extra_specs | +--------------------------------------+---------+------------------------------------------+ | 48645332-4835-4a9b-9078-cd735f47dae5 | ovh | {u'volume_backend_name': u'RBD_OVH'} | +--------------------------------------+---------+------------------------------------------+

Check that the cinder scheduler is set as follows in /etc/cinder/cinder.conf

scheduler_driver=cinder.scheduler.filter_scheduler.FilterScheduler

Assembling instance and volume ¶

After creating a volume using the OVH cinder backend:

cinder create --volume-type ovh --display-name test 1

An instance is created in the OVH availability zone:

nova boot --availability-zone ovh \ --image 'cirros image' \ --key-name key_loic \ --nic net-id=e1d72366-1f25-42c1-a953-a944c9f932e3 \ --flavor m1.tiny --poll try

The volume is attached to the instance

nova volume-attach try 045d1cae-cd9b-4d64-b0b8-544f5b6d0c5a /dev/vdb