While playing with your Ceph cluster , you might have seen HEALTH_WARN cluster status.
Cluster warning can occur due to several reasons of component malfunctioning such as MON,OSD,PG & MDS.
In my case i saw warning due to Ceph monitors, which was like :
1
| health HEALTH_WARN 1 mons down, quorum 0,1 ceph-mon1,ceph-mon2
|
At first i tried restarting MON service , but no luck.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
| [root@ceph-mon3 ~]# service ceph status mon
=== mon.ceph-mon3 ===
mon.ceph-mon3: not running.
[root@ceph-mon3 ~]# service ceph start mon
=== mon.ceph-mon3 ===
Starting Ceph mon.ceph-mon3 on ceph-mon3...
Invalid argument: /var/lib/ceph/mon/ceph-ceph-mon3/store.db: does not exist (create_if_missing is false)
IO error: /var/lib/ceph/mon/ceph-ceph-mon3/store.db/000001.dbtmp: Input/output error
2015-05-22 11:44:38.065906 7fad6c6967a0 -1 failed to create new leveldb store
failed: 'ulimit -n 131072; /usr/bin/ceph-mon -i ceph-mon3 --pid-file /var/run/ceph/mon.ceph-mon3.pid -c /etc/ceph/ceph.conf --cluster ceph '
Starting ceph-create-keys on ceph-mon3...
[root@ceph-mon3 ~]#
[root@ceph-mon3 ~]# service ceph status mon
=== mon.ceph-mon3 ===
mon.ceph-mon3: not running.
[root@ceph-mon3 ~]#
|
The error message that i received was not something normal, so i started playing rough with my cluster , by moving monitor store.db files. !!! Be Cautious
1
| mv /var/lib/ceph/mon/ceph-ceph-mon3/store.db /var/lib/ceph/mon/ceph-ceph-mon3/store.db.orig
|
And this broke MON really badly, so now i know another way that causes a new error YAY :-)
1
2
3
4
5
6
7
| [root@ceph-mon3 ceph-ceph-mon3]# service ceph start mon
=== mon.ceph-mon3 ===
Starting Ceph mon.ceph-mon3 on ceph-mon3...
2015-05-22 11:59:45.385826 7faa43dfb7a0 -1 unable to read magic from mon data.. did you run mkcephfs?
failed: 'ulimit -n 131072; /usr/bin/ceph-mon -i ceph-mon3 --pid-file /var/run/ceph/mon.ceph-mon3.pid -c /etc/ceph/ceph.conf --cluster ceph '
Starting ceph-create-keys on ceph-mon3...
[root@ceph-mon3 ceph-ceph-mon3]#
|
Show Time begins :-)
Then i started doing real work by reading monitor logs and what i found was monitor IP address were incorrect , they need to have a different address range.
To fix this first we need to change monitor IP address to the correct rage.
Changing Ceph Monitor IP Address ¶
- Get monitor maps , you could see the current IP range is 80.50.X.X , we need to change this to the correct range.
1
2
3
4
5
6
7
8
9
10
11
12
13
| [root@ceph-mon1 ~]# ceph mon getmap -o /tmp/monmap
got monmap epoch 3
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]# monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 3
fsid 98d89661-f616-49eb-9ccf-84d720e179c0
last_changed 2015-05-18 14:42:01.287460
created 2015-05-18 14:41:00.514879
0: 80.50.50.35:6789/0 mon.ceph-mon1
1: 80.50.50.37:6789/0 mon.ceph-mon2
2: 80.50.50.39:6789/0 mon.ceph-mon3
[root@ceph-mon1 ~]#
|
- Remove monitor nodes from monitor map
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
| [root@ceph-mon1 ~]# monmaptool --rm ceph-mon1 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: removing ceph-mon1
monmaptool: writing epoch 3 to /tmp/monmap (2 monitors)
[root@ceph-mon1 ~]# monmaptool --rm ceph-mon2 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: removing ceph-mon2
monmaptool: writing epoch 3 to /tmp/monmap (1 monitors)
[root@ceph-mon1 ~]# monmaptool --rm ceph-mon3 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: removing ceph-mon3
monmaptool: writing epoch 3 to /tmp/monmap (0 monitors)
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]# monmaptool --print /tmp/monmap
monmaptool: monmap file /tmp/monmap
epoch 3
fsid 98d89661-f616-49eb-9ccf-84d720e179c0
last_changed 2015-05-18 14:42:01.287460
created 2015-05-18 14:41:00.514879
[root@ceph-mon1 ~]#
|
- Add the correct hostname and IP address for monitor nodes
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| [root@ceph-mon1 ~]# monmaptool --add ceph-mon1-ib 10.100.1.101:6789 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: writing epoch 3 to /tmp/monmap (1 monitors)
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]# monmaptool --add ceph-mon2-ib 10.100.1.102:6789 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: writing epoch 3 to /tmp/monmap (2 monitors)
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]# monmaptool --add ceph-mon3-ib 10.100.1.103:6789 /tmp/monmap
monmaptool: monmap file /tmp/monmap
monmaptool: writing epoch 3 to /tmp/monmap (3 monitors)
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]#
[root@ceph-mon1 ~]# monmaptool --print /tmp/monmap monmaptool: monmap file /tmp/monmap
epoch 3
fsid 98d89661-f616-49eb-9ccf-84d720e179c0
last_changed 2015-05-18 14:42:01.287460
created 2015-05-18 14:41:00.514879
0: 10.100.1.101:6789/0 mon.ceph-mon1-ib
1: 10.100.1.102:6789/0 mon.ceph-mon2-ib
2: 10.100.1.103:6789/0 mon.ceph-mon3-ib
[root@ceph-mon1 ~]#
|
- Before injecting the new monmap , bring down monitor services and then inject the monmap.
1
2
3
4
5
6
| [root@ceph-mon1 ~]# service ceph stop mon
=== mon.ceph-mon1 ===
Stopping Ceph mon.ceph-mon1 on ceph-mon1...kill 441540...done
[root@ceph-mon1 ~]#
[root@ceph-mon1 ceph]# ceph-mon -i ceph-mon1 --inject-monmap /tmp/monmap
[root@ceph-mon1 ceph]#
|
- Repeat these steps for the other monitors in your cluster , to save some time you can copy the new monmap file from first monitor node (ceph-mon1) to other monitor nodes and simply inject this new monmap into their ceph monitor instance.
- Finally bring up the monitor services on all the monitor nodes.
1
2
3
4
5
6
7
8
9
| [root@ceph-mon1 ceph]# service ceph start mon
=== mon.ceph-mon1 ===
Starting Ceph mon.ceph-mon1 on ceph-mon1...
Starting ceph-create-keys on ceph-mon1...
[root@ceph-mon1 ceph]#
[root@ceph-mon1 ceph]# service ceph status mon
=== mon.ceph-mon1 ===
mon.ceph-mon1: running {"version":"0.80.9"}
[root@ceph-mon1 ceph]#
|
If you still see monitor problems, you can redeploy the monitor node
1
| [root@ceph-mon1 ceph]# ceph-deploy --overwrite-conf mon create ceph-mon3
|
- Finally your cluster should attain Health_OK status
1
2
3
4
5
6
7
8
9
| [root@ceph-mon3 ceph]# ceph -s
cluster 98d89661-f616-49eb-9ccf-84d720e179c0
health HEALTH_OK
monmap e4: 3 mons at {ceph-mon1=10.100.1.101:6789/0,ceph-mon2=10.100.1.102:6789/0,ceph-mon3=10.100.1.103:6789/0}, election epoch 18, quorum 0,1,2 ceph-mon1,ceph-mon2,ceph-mon3
osdmap e244: 55 osds: 54 up, 54 in
pgmap v693: 192 pgs, 3 pools, 0 bytes data, 0 objects
5327 MB used, 146 TB / 146 TB avail
192 active+clean
[root@ceph-mon3 ceph]#
|
This might give you some idea of Ceph monitor troubleshooting. You could also follow more detailed steps mentioned in Ceph documentation.