Slicing the Pie: CRUSH Failure Domains and Your Capacity

Anthony D'Atri

In the United States, the fourth Thursday of November is Thanksgiving, and for many of us the best part of the table is the pie. Our nautilus has dressed up for the occasion, and this post is about slicing pies.

Every placement group (PG) in a Ceph pool is a pie. A replicated pool cuts each one into as many identical slices as the pool's size. An erasure-coded pool cuts it into k data slices and m coding slices. CRUSH then serves the slices to the failure domains named in the pool's CRUSH rule, at most one slice to each, so that losing a host, a rack, or a room costs each PG at most one slice.

That one-slice rule keeps your data safe. It also decides how much of your raw capacity a pool can use, and the answer is sometimes surprising. Below are a few examples, and a calculator right in this page to try your own. At the end is the rewritten PG Calc, for deciding how many pies to bake.

Failure domains

A failure domain is the CRUSH bucket type that a pool's CRUSH rule spreads each PG across: usually host, sometimes rack, row, room, or datacenter. The default replicated rule and the default erasure-code profile both use host. A failure domain's CRUSH weight is the sum of the weights of the OSDs under it, and an OSD's weight is by default its capacity in TiB. ceph osd df tree shows them all.

Call the number of slices in each PG n: the size of a replicated pool, or k+m for an erasure-coded pool. Each PG needs n failure domains and puts one slice in each. So:

  • A failure domain can never hold more than 1/n of what the pool stores, however big it is.
  • With exactly n failure domains, every PG puts a slice in every one of them, so each holds the same amount, and the smallest one sets the pace.
  • Ceph stops writes when any OSD reaches the full ratio, 0.95 by default, and backfill stops at the backfillfull ratio, 0.90 by default.

Stranded capacity

Take a replicated pool of size 3 across three hosts with CRUSH weights of 100, 100, and 1138. You might hope to store about a third of the 1338 raw. But every PG puts one replica on each host, so each host holds the same amount, and the two small ones fill first. The pool stores 95 at the full ratio, and 1043 of the big host's capacity, 78% of the raw, is stranded: nothing CRUSH places can use it.

A fourth host helps less than you might think. With hosts of CRUSH weights 100, 100, 100, and 1701, a PG chooses three hosts of the four, but the big one can still hold at most one replica of each PG, which is a third of what's stored. The pool stores 142.5, and again 78% of the 2001 raw is stranded. With the other three at 100, the fourth host could be as big as 150 with nothing stranded.

The rule of thumb: no failure domain can usefully hold more than 1/n of the pool's raw capacity. Put another way, the biggest failure domain can be at most the sum of the others divided by n-1. Grow failure domains together, not one at a time.

CRUSH alone and the balancer

The figures above are for a perfectly balanced placement. With uneven failure domains, CRUSH alone, meaning CRUSH placement without the balancer's upmap remapping, doesn't quite get there. Its straw2 algorithm draws each failure domain in proportion to its weight, but when it draws one that the PG already uses, it draws again. So a heavy failure domain is picked less often than its weight says, and the light ones fill early. With hosts of CRUSH weights 100, 100, 100, and 150, a balanced placement stores 142.5 with nothing stranded, but CRUSH alone stops at about 132, when the first small host fills.

The balancer Manager module closes that gap by remapping individual PGs. It is on by default, in upmap mode. Leave it on.

When something fails

Failure domains matter most when something breaks. Say an OSD of weight 10 fails in one of three hosts of 100, in a replicated pool of size 3. Every PG already has a replica on each of the other two hosts, so the failed OSD's data can be re-protected only on the surviving OSDs of its own host. That host is now 90, and since every PG has a replica on it, it sets the pace. The pool's ceiling drops from 95 to 85.5: the 9.5 that the failed OSD could hold, plus 9.5 on each of the other two hosts that can no longer be used. Worse, backfill stops at the backfillfull ratio, so if the pool holds more than 81, 85% of its ceiling, when the OSD fails, backfill stops with PGs still degraded.

Add a fourth host of 100, and the same failure lowers the ceiling only by what the failed OSD could hold, from 126.7 to 123.5. Full re-protection needs the pool under 117, 92% of its ceiling. Lose a whole host out of three, and there's nowhere to re-protect at all: the PGs stay degraded until the host returns. Out of four, the PGs re-protect on the other three, as long as the pool holds less than 90, 71% of its ceiling.

Erasure coding follows the same arithmetic, with k+m in place of the size. A 4+2 pool across six hosts of 100 stores 380. When an OSD of weight 10 fails, the ceiling drops to 342, and re-protection needs the pool under 324. Across seven hosts, the pool stores 443.3, the same failure drops the ceiling to 437, and re-protection needs the pool under 414.

So plan on at least one more failure domain than the pool's size or k+m, and more if you can. The more failure domains there are, the less any one failure moves, and the less headroom you need below the backfillfull ratio.

While we're at it: if you choose m=1, expect data unavailability during maintenance and data loss when component failures overlap. The docs explain why, and the calculator below warns about it.

Try it

The calculator takes:

  • Data protection: replicated or erasure coded, the size or k and m, and the full and backfillfull ratios. Leave Shards per failure domain at 1 unless you really know what you're doing: values above 1 model CRUSH rules that put several shards of a PG in one failure domain.
  • Failure domains: the CRUSH weight of each, from the WEIGHT column of ceph osd df tree. Add or remove as many as you like.
  • What fails: nothing, an OSD of a given weight in a failure domain you choose, or a whole failure domain.

The examples load the cases above. The result is marked green when the arrangement is OK, yellow when there's no spare failure domain, orange when raw capacity is stranded, and red when there aren't enough failure domains to place each PG at all. It shows what a balanced placement can store, what CRUSH alone stores before the first failure domain fills, why, and what changes when something fails. The same calculator is in the Ceph docs as FD Calc.

Data protection

* Leave this at 1 unless you really know what you’re doing. Values above 1 model CRUSH rules that put several shards of a PG in one failure domain: MSR rules, or an EC profile’s crush-osds-per-failure-domain.

Failure domains (CRUSH weight of each)
What fails
Examples

How many pies: PG Calc

Failure domains decide where each PG's slices go. How many PGs a pool should have is a separate question. The PG autoscaler answers it well for many clusters running recent releases, once mon_target_pg_per_osd is at least 200, which is its default beginning with the Umbrella release. To internalize how the numbers interact, or where the autoscaler is infeasible or undesirable, use PG Calc in the Ceph docs.

PG Calc has been rewritten. It no longer loads jQuery or the old website's theme from the Internet Archive, and its presets match the pools that RGW and CephFS create today. Each pool can be replicated or erasure coded with its own k and m, and Generate commands produces the CRUSH rules, erasure-code profiles, pools, and application tags, ready to review and paste. It marks a pool red when there aren't enough OSDs to place its PGs, and amber when it has fewer than three replicas or m=1.

Leftovers

  • Have at least one more failure domain than the pool's size or k+m.
  • Keep failure domains of about the same weight. None can usefully hold more than 1/n of the raw capacity.
  • Add capacity across failure domains, not to one.
  • Keep the balancer on.
  • Leave room below the backfillfull ratio for recovery.

Happy Thanksgiving to all who celebrate, and thanks to everyone who makes Ceph what it is. Enjoy the pie.