Ceph : Reduce OSD Scrub Priority

Let’s assume ! on a nice sunny day , you receives complaints that your Ceph storage cluster is not performing as it was performing yesterday. After checking cluster status you found that placement groups scrubbing is going on and depending on your scenario , you would like to decrease its priority. Here is how you can do it.

Note : OSD disk thread I/O priority can only be changed if the disk scheduler is cfq.

• Check disk scheduler, if its not cfq you can change it to cfq dynamically.

1
2
3

$ sudo cat /sys/block/sda/queue/scheduler
noop [deadline] cfq
$ sudo echo cfq > /sys/block/sda/queue/scheduler

• Next check for the current values of OSD disk thread io priority , the default values should be as shown below.

1
2

$ sudo ceph daemon osd.0 config get osd_disk_thread_ioprio_class
{ "osd_disk_thread_ioprio_class": ""}

1
2

$ sudo ceph daemon osd.0 config get osd_disk_thread_ioprio_priority
{ "osd_disk_thread_ioprio_priority": "-1"}

• Reduce the osd_disk_thread_ioprio by executing

1
2

$ sudo ceph tell osd.* injectargs '--osd_disk_thread_ioprio_priority 7'
$ sudo ceph tell osd.* injectargs '--osd_disk_thread_ioprio_class idle'

• Finally recheck osd_disk_thread_ioprio

1
2

$ sudo ceph daemon osd.0 config get osd_disk_thread_ioprio_class
{ "osd_disk_thread_ioprio_class": "idle"}

1
2

$ sudo ceph daemon osd.0 config get osd_disk_thread_ioprio_priority
{ "osd_disk_thread_ioprio_priority": "7"}

This should reduce OSD scrubbing priority and is useful to slow down scrubbing on an OSD that is busy handling client operations. Once the coast is clear , its a good idea to revert back the changes.