12.2.7 Luminous released

TheAnalyst

This is the seventh bugfix release of Luminous v12.2.x long term stable release series. This release contains several fixes for regressions in the v12.2.6 and v12.2.5 releases. We recommend that all users upgrade to v12.2.7.

note:The v12.2.6 release has serious known regressions. While 12.2.6 wasn't formally announced, if you've still installed this release, please see the upgrade procedure below.
note:The v12.2.5 release has a potential data corruption issue with erasure coded pools. If you ran v12.2.5 with erasure coding, please see below.

Upgrading from v12.2.6

v12.2.6 included an incomplete backport of an optimization for BlueStore OSDs that avoids maintaining both the per-object checksum and the internal BlueStore checksum. Due to the accidental omission of a critical follow-on patch, v12.2.6 corrupts (fails to update) the stored per-object checksum value for some objects. This can result in an EIO error when trying to read those objects.

  1. If your cluster uses FileStore only, no special action is required. This problem only affects clusters with BlueStore.

  2. If your cluster has only BlueStore OSDs (no FileStore), then you should enable the following OSD option:

    osd skip data digest = true

    This will avoid setting and start ignoring the full-object digests whenever the primary for a PG is BlueStore.

  3. If you have a mix of BlueStore and FileStore OSDs, then you should enable the following OSD option:

    osd distrust data digest = true

    This will avoid setting and start ignoring the full-object digests in all cases. This weakens the data integrity checks for FileStore (although those checks were always only opportunistic).

If your cluster includes BlueStore OSDs and was affected, deep scrubs will generate errors about mismatched CRCs for affected objects. Currently the repair operation does not know how to correct them (since all replicas do not match the expected checksum it does not know how to proceed). These warnings are harmless in the sense that IO is not affected and the replicas are all still in sync. The number of affected objects is likely to drop (possibly to zero) on their own over time as those objects are modified. We expect to include a scrub improvement in v12.2.8 to clean up any remaining objects.

Additionally, see the notes below, which apply to both v12.2.5 and v12.2.6.

Upgrading from v12.2.5 or v12.2.6

If you used v12.2.5 or v12.2.6 in combination with erasure coded pools, there is a small risk of corruption under certain workloads. Specifically, when:

  • An erasure coded pool is in use
  • The pool is busy with successful writes
  • The pool is also busy with updates that result in an error result to the librados user. RGW garbage collection is the most common example of this (it sends delete operations on objects that don’t always exist.)
  • Some OSDs are reasonably busy. One known example of such load is FileStore splitting, although in principle any load on the cluster could also trigger the behavior.
  • One or more OSDs restarts.

This combination can trigger an OSD crash and possibly leave PGs in a state where they fail to peer.

Notably, upgrading a cluster involves OSD restarts and as such may increase the risk of encountering this bug. For this reason, for clusters with erasure coded pools, we recommend the following upgrade procedure to minimize risk:

  1. Install the v12.2.7 packages.

  2. Temporarily quiesce IO to cluster:

    ceph osd pause

  3. Restart all OSDs and wait for all PGs to become active.

  4. Resume IO:

    ceph osd unpause

This will cause an availability outage for the duration of the OSD restarts. If this in unacceptable, an more risky alternative is to disable RGW garbage collection (the primary known cause of these rados operations) for the duration of the upgrade:

1. Set ``rgw_enable_gc_threads = false`` in ceph.conf

2. Restart all radosgw daemons

3. Upgrade and restart all OSDs

4. Remove ``rgw_enable_gc_threads = false`` from ceph.conf

5. Restart all radosgw daemons

Upgrading from other versions

If your cluster did not run v12.2.5 or v12.2.6 then none of the above issues apply to you and you should upgrade normally.

v12.2.7 Changelog

Notable Changes in v12.2.6

note:This is a broken release with serious known regressions. Do not install it. The release notes below are to track the fixes that were a part of 12.2.6 (and hence 12.2.7)

  • Auth:

    • In 12.2.4 and earlier releases, keyring caps were not checked for validity, so the caps string could be anything. As of 12.2.6, caps strings are validated and providing a keyring with an invalid caps string to, e.g., “ceph auth add” will result in an error.
    • CVE 2018-1128: auth: cephx authorizer subject to replay attack (issue#24836, Sage Weil)
    • CVE 2018-1129: auth: cephx signature check is weak (issue#24837, Sage Weil)
    • CVE 2018-10861: mon: auth checks not correct for pool ops (issue#24838, Jason Dillaman)

  • The config-key interface can store arbitrary binary blobs but JSON can only express printable strings. If binary blobs are present, the ‘ceph config-key dump’ command will show them as something like <<< binary blob of length N >>>.

v12.2.6 Changelog