Post

Ceph Orchestration Pitfalls

Posted
Ceph Octopus release mascot artwork
Ceph Octopus release mascot artwork
By Lee, Chien-Pang
18 min read

Cloud native tools usually give me excellent user experience that they feel like a plug and play, with cool out-of-the-box capabilities like auto deployments, scaling, healing, rolling-updates, etc. This ease of use convenience, however, is usually under the big assumption that it’s internet-reachable. This has always been my encounter with orchestration tools and infra. They are not air-gap friendly because you never know where the programs’/binaries’ default registries, mirrors or download values are hard-coded and if your overwrite settings are comprehensive enough that at some point the hidden misconfigurations would act up and block your way in a silent and hard-to-detect manner. My journey from Ceph native (legacy) to cephadm is no exception.

There are three main ways people manage Ceph, namely cephadm, Rook and legacy-native. Over the years I have been operating Ceph via the less favored approach, legacy-native - which is optimal for performance (without orchestration layer) and flexible in deployments as well as the granularities like how we want a bluestore/rocksdb to be partitioned or how to organize a disk into one or multiple osd, etc. Debugging a broken service is to me also straightforward but this is true since I am familiar with such bare-metal setups for long enough. An extra word on how old-fashioned I have been, I don’t even manage Ceph packages with Ansible, nor ceph-deploy. Everything is handled flat, rpm, systemd, monmap, keyring, mon, mgr, mds, rgw to partitioning disk and bringing up bluestore-backed osds, not to mention quorum establishment and version upgrades. All was well until loose ends appeared. I started to see outdated maintenance and operation scripts here and there that needed to be updated. The majority would eventually link to the final boss of all pains, “upgrade”, which has always been the biggest maintenance issue, keeping all things up to date IN TIME. Doing all the logistics is possible; nonetheless doing them fast is unlikely in a stretched release cycle. I cannot help but consider the orchestration path of managing Ceph storage.

Rook is immediately out of the choice list because I don’t have k8s infra for Rook to sit on top. Now cephadm is all that’s left; it is also the one that Ceph leans to. By the Octopus (v15.2.0) release back in 2020 cephadm was decided to be the future orchestrator. The biggest motivation was to fix a broken deployment ecosystem and fragmented tools such as ceph-deploy, ceph-ansible, ceph-volume, DeepSea, etc. That’s enough background. As I made up my mind and dug into the route to cephadm, things in general all moved on at a good pace until I hit an issue that baffled me.

Cutting the crap short, cephadm works like a charm when it’s Internet-reachable. This is generally the case for Cloud native technologies. But my ambition is to let it work in both Internet-reachable and air-gapped environments. cephadm works by running on top of containers (either Docker or Podman) which naturally depends on a Ceph image (quay.io/ceph/ceph:v20.2.0 as of this writing). Instead of hosting a registry for podman later to pull from, I go with skopeo

1

skopeo copy docker://$(CEPH_IMAGE) docker-archive:$(CEPH_IMAGE_PKG):$(CEPH_IMAGE)

and use podman to load it into cache.

1
2
3
4
5
6

[h1 ~]# podman load -i /frbe/usr/share/ceph/ceph_ceph_v20.2.0.tar
[h1 ~]# podman images
[h1 ~]# podman images
REPOSITORY         TAG         IMAGE ID      CREATED        SIZE
quay.io/ceph/ceph  v20.2.0     524f3da27646  10 months ago  1.56 GB
[h1 ~]#

Then we smuggle cephadm into host filesystem where we can invoke cephadm with correct version matching ceph image.

1
2
3
4
5
6
7
8
9

[h1 ~]# podman run --rm --entrypoint cat 'quay.io/ceph/ceph:v20.2.0' /usr/sbin/cephadm > /usr/local/sbin/cephadm
[h1 ~]# chmod 755 /usr/local/sbin/cephadm
[h1 ~]# which cephadm
/usr/local/sbin/cephadm
[h1 ~]# file /usr/local/sbin/cephadm
/usr/local/sbin/cephadm: a /usr/bin/python3 -s script executable (Zip archive)
[h1 ~]# cephadm version
cephadm version 20.2.0 (69f84cc2651aa259a15bc192ddaabd3baba07489) tentacle (stable)
[h1 ~]#

Bootstrap

1
2
3

cephadm --image 'quay.io/ceph/ceph:v20.2.0' bootstrap --mon-ip '192.168.1.161' --fsid '...' \
    --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring \
    --skip-dashboard --skip-monitoring-stack --skip-firewalld --skip-pull --allow-overwrite

Cluster-wide config pin + mon orchestration. Since everything starts with a single node, fixing replica size to 1 can avoid undersize complains.

1
2
3
4
5

[h1 ~]# ceph config set global container_image 'quay.io/ceph/ceph:v20.2.0'
[h1 ~]# ceph config set global osd_pool_default_size 1
[h1 ~]# ceph config set global osd_pool_default_min_size 1
[h1 ~]# ceph orch host label add 'h1' mon
[h1 ~]# ceph orch apply mon --placement="label:mon"

mgr/mds/rgw orchestration (this is where the air-gapped deployment silently stalls)

1
2
3
4
5
6

[h1 ~]# ceph orch host label add 'h1' mgr
[h1 ~]# ceph orch apply mgr --placement="label:mgr"
[h1 ~]# ceph orch host label add 'h1' mds
[h1 ~]# ceph orch apply mds cephfs --placement="label:mds"
[h1 ~]# ceph orch host label add 'h1' rgw
[h1 ~]# ceph orch apply rgw default --placement="label:rgw"

This is where I was blocked, for days honestly. The big difference between a working cluster vs failing one is the access to Internet. With this, my instinct is to look into status and logs to seek out exactly what’s gating the deployment.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

[h1 ~]# ceph orch host ls
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
HOST  ADDR           LABELS          STATUS
h1    192.168.1.161  _admin,mon,mgr  1 hosts in cluster
[h1 ~]# ceph orch apply mon --placement="h1"
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
Scheduled mon update...
[h1 ~]# ceph orch host ls
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
HOST  ADDR           LABELS          STATUS
h1    192.168.1.161  _admin,mon,mgr  1 hosts in cluster
[h1 ~]# ceph orch apply mgr --placement="h1"
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
Scheduled mgr update...
[h1 ~]# ceph cephadm get-pub-key >> /root/.ssh/authorized_keys
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
[h1 ~]# chmod 600 /root/.ssh/authorized_keys
[h1 ~]# ceph orch ls
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
NAME   PORTS  RUNNING  REFRESHED  AGE  PLACEMENT
crash             0/0  -          9m   *
mgr               0/0  -          30s  h1
mon               0/0  -          42s  h1
[h1 ~]# ceph orch host rescan h1
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
Ok
[h1 ~]# ceph orch ls --refresh
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
NAME   PORTS  RUNNING  REFRESHED  AGE  PLACEMENT
crash             0/0  -          9m   *
mgr               0/0  -          50s  h1
mon               0/0  -          62s  h1
[h1 ~]# ceph orch ps
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
No daemons reported
[h1 ~]#

From the status I could tell that cephadm doesn’t see any daemon. How about from a logs perspective?

1
2
3
4

[h1 ~]# ceph config set mgr mgr/cephadm/log_to_cluster true
[h1 ~]# ceph config set mgr mgr/cephadm/log_to_cluster_log true
[h1 ~]# ceph orch host rescan h1
[h1 ~]# ceph log last 100 cephadm

There were lots of log details (and noise) I had to screen out, leading to potential culprits that I have checked against.

  • Hostname changes?

Label h1 for all required roles (mon, mgr, mds, rgw) and bind the service specifications directly to host h1.

1
2
3
4
5
6
7
8
9
10

ceph orch host label add h1 mon
ceph orch host label add h1 mgr
ceph orch host label add h1 mds
ceph orch host label add h1 rgw
ceph orch apply mon --placement="h1"
ceph orch apply mgr --placement="h1"
ceph orch apply mds.cephfs --placement="h1"
ceph orch host rescan h1
ceph orch ls --refresh
ceph orch ps

The above didn’t work. Looking into logs:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137

[h1 ~]# ceph log last 50 cephadm
Inferring fsid c6e64c49-09cf-463b-9d1c-b6645b4b3b85
Inferring config /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.h1/config
2026-09-17T22:43:40.457215+0000 mgr.h1.svgpaj (mgr.14120) 33 : cephadm [ERR] Cannot place <MONSpec for service_name=mon>: No matching hosts for label mon
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3797, in apply
    raise e
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3791, in apply
    res = self._apply(spec)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3487, in _apply
    return self._apply_service_spec(cast(ServiceSpec, spec))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3756, in _apply_service_spec
    HostAssignment(
  File "/usr/share/ceph/mgr/cephadm/schedule.py", line 242, in validate
    raise OrchestratorValidationError(
orchestrator._interface.OrchestratorValidationError: Cannot place <MONSpec for service_name=mon>: No matching hosts for label mon
2026-09-17T22:44:04.544369+0000 mgr.h1.svgpaj (mgr.14120) 51 : cephadm [ERR] host unconfigured does not exist
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 2064, in add_host_label
    self.inventory.add_label(host, label)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 187, in add_label
    host = self._get_stored_name(host)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 133, in _get_stored_name
    self.assert_host(host)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 161, in assert_host
    raise OrchestratorError('host %s does not exist' % host)
orchestrator._interface.OrchestratorError: host unconfigured does not exist
2026-09-17T22:44:04.857756+0000 mgr.h1.svgpaj (mgr.14120) 54 : cephadm [ERR] Cannot place <ServiceSpec for service_name=mgr>: No matching hosts for label mgr
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3797, in apply
    raise e
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3791, in apply
    res = self._apply(spec)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3487, in _apply
    return self._apply_service_spec(cast(ServiceSpec, spec))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3756, in _apply_service_spec
    HostAssignment(
  File "/usr/share/ceph/mgr/cephadm/schedule.py", line 242, in validate
    raise OrchestratorValidationError(
orchestrator._interface.OrchestratorValidationError: Cannot place <ServiceSpec for service_name=mgr>: No matching hosts for label mgr
2026-09-17T22:44:05.147581+0000 mgr.h1.svgpaj (mgr.14120) 56 : cephadm [ERR] host unconfigured does not exist
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 2064, in add_host_label
    self.inventory.add_label(host, label)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 187, in add_label
    host = self._get_stored_name(host)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 133, in _get_stored_name
    self.assert_host(host)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 161, in assert_host
    raise OrchestratorError('host %s does not exist' % host)
orchestrator._interface.OrchestratorError: host unconfigured does not exist
2026-09-17T22:44:05.453924+0000 mgr.h1.svgpaj (mgr.14120) 58 : cephadm [ERR] Cannot place <MDSSpec for service_name=mds.cephfs>: No matching hosts for label mds
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3797, in apply
    raise e
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3791, in apply
    res = self._apply(spec)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3487, in _apply
    return self._apply_service_spec(cast(ServiceSpec, spec))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3756, in _apply_service_spec
    HostAssignment(
  File "/usr/share/ceph/mgr/cephadm/schedule.py", line 242, in validate
    raise OrchestratorValidationError(
orchestrator._interface.OrchestratorValidationError: Cannot place <MDSSpec for service_name=mds.cephfs>: No matching hosts for label mds
2026-09-17T22:44:15.279821+0000 mgr.h1.svgpaj (mgr.14275) 3 : cephadm [ERR] host unconfigured does not exist
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 2064, in add_host_label
    self.inventory.add_label(host, label)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 187, in add_label
    host = self._get_stored_name(host)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 133, in _get_stored_name
    self.assert_host(host)
  File "/usr/share/ceph/mgr/cephadm/inventory.py", line 161, in assert_host
    raise OrchestratorError('host %s does not exist' % host)
orchestrator._interface.OrchestratorError: host unconfigured does not exist
2026-09-17T22:44:15.597347+0000 mgr.h1.svgpaj (mgr.14275) 4 : cephadm [INF] [17/Sep/2026:22:44:15] ENGINE Bus STARTING
2026-09-17T22:44:15.616241+0000 mgr.h1.svgpaj (mgr.14275) 6 : cephadm [ERR] Cannot place <RGWSpec for service_name=rgw.default>: No matching hosts for label rgw
Traceback (most recent call last):
  File "/usr/share/ceph/mgr/orchestrator/_interface.py", line 140, in wrapper
    return OrchResult(f(*args, **kwargs))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3797, in apply
    raise e
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3791, in apply
    res = self._apply(spec)
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3487, in _apply
    return self._apply_service_spec(cast(ServiceSpec, spec))
  File "/usr/share/ceph/mgr/cephadm/module.py", line 3756, in _apply_service_spec
    HostAssignment(
  File "/usr/share/ceph/mgr/cephadm/schedule.py", line 242, in validate
    raise OrchestratorValidationError(
orchestrator._interface.OrchestratorValidationError: Cannot place <RGWSpec for service_name=rgw.default>: No matching hosts for label rgw
2026-09-17T22:44:15.717374+0000 mgr.h1.svgpaj (mgr.14275) 7 : cephadm [INF] [17/Sep/2026:22:44:15] ENGINE Serving on https://192.168.1.161:7150
2026-09-17T22:44:15.717964+0000 mgr.h1.svgpaj (mgr.14275) 8 : cephadm [INF] [17/Sep/2026:22:44:15] ENGINE Client ('192.168.1.161', 54022) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)')
2026-09-17T22:44:15.818840+0000 mgr.h1.svgpaj (mgr.14275) 9 : cephadm [INF] [17/Sep/2026:22:44:15] ENGINE Serving on http://192.168.1.161:8765
2026-09-17T22:44:15.819049+0000 mgr.h1.svgpaj (mgr.14275) 10 : cephadm [INF] [17/Sep/2026:22:44:15] ENGINE Bus STARTED
2026-09-17T22:44:38.698506+0000 mgr.h1.svgpaj (mgr.14388) 3 : cephadm [INF] [17/Sep/2026:22:44:38] ENGINE Bus STARTING
2026-09-17T22:44:38.822076+0000 mgr.h1.svgpaj (mgr.14388) 4 : cephadm [INF] [17/Sep/2026:22:44:38] ENGINE Serving on https://192.168.1.161:7150
2026-09-17T22:44:38.824843+0000 mgr.h1.svgpaj (mgr.14388) 5 : cephadm [INF] [17/Sep/2026:22:44:38] ENGINE Client ('192.168.1.161', 55008) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)')
2026-09-17T22:44:38.925673+0000 mgr.h1.svgpaj (mgr.14388) 6 : cephadm [INF] [17/Sep/2026:22:44:38] ENGINE Serving on http://192.168.1.161:8765
2026-09-17T22:44:38.926104+0000 mgr.h1.svgpaj (mgr.14388) 7 : cephadm [INF] [17/Sep/2026:22:44:38] ENGINE Bus STARTED
2026-09-17T22:47:02.542598+0000 mgr.h1.svgpaj (mgr.14388) 117 : cephadm [INF] Set ssh ssh_config
2026-09-17T22:47:21.332533+0000 mgr.h1.svgpaj (mgr.14525) 3 : cephadm [INF] [17/Sep/2026:22:47:21] ENGINE Bus STARTING
2026-09-17T22:47:21.449102+0000 mgr.h1.svgpaj (mgr.14525) 4 : cephadm [INF] [17/Sep/2026:22:47:21] ENGINE Serving on https://192.168.1.161:7150
2026-09-17T22:47:21.450975+0000 mgr.h1.svgpaj (mgr.14525) 5 : cephadm [INF] [17/Sep/2026:22:47:21] ENGINE Client ('192.168.1.161', 42898) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)')
2026-09-17T22:47:21.551798+0000 mgr.h1.svgpaj (mgr.14525) 6 : cephadm [INF] [17/Sep/2026:22:47:21] ENGINE Serving on http://192.168.1.161:8765
2026-09-17T22:47:21.552001+0000 mgr.h1.svgpaj (mgr.14525) 7 : cephadm [INF] [17/Sep/2026:22:47:21] ENGINE Bus STARTED
2026-09-17T22:47:41.490013+0000 mgr.h1.svgpaj (mgr.14525) 17 : cephadm [INF] disk rescan request sent to host "h1"
2026-09-17T22:47:44.007116+0000 mgr.h1.svgpaj (mgr.14525) 21 : cephadm [INF] disk rescan: Ok. 7 adapters detected: 7 rescanned, 0 skipped, 0 failed (1.87s)
2026-09-17T22:51:03.986563+0000 mgr.h1.svgpaj (mgr.14525) 187 : cephadm [INF] Added label mon to host h1
2026-09-17T22:51:24.234134+0000 mgr.h1.svgpaj (mgr.14525) 206 : cephadm [INF] Added label mgr to host h1
2026-09-17T22:52:03.129108+0000 mgr.h1.svgpaj (mgr.14525) 240 : cephadm [INF] Saving service mon spec with placement h1
2026-09-17T22:52:14.715137+0000 mgr.h1.svgpaj (mgr.14525) 253 : cephadm [INF] Saving service mgr spec with placement h1
2026-09-17T22:52:56.340400+0000 mgr.h1.svgpaj (mgr.14525) 291 : cephadm [INF] disk rescan request sent to host "h1"
2026-09-17T22:52:58.863722+0000 mgr.h1.svgpaj (mgr.14525) 294 : cephadm [INF] disk rescan: Ok. 7 adapters detected: 7 rescanned, 0 skipped, 0 failed (1.88s)
2026-09-17T22:54:07.847086+0000 mgr.h1.svgpaj (mgr.14525) 353 : cephadm [INF] Cleared cluster SSH key
2026-09-17T22:54:15.714123+0000 mgr.h1.svgpaj (mgr.14525) 361 : cephadm [INF] Generating ssh key...
2026-09-17T22:55:33.008197+0000 mgr.h1.svgpaj (mgr.14889) 3 : cephadm [INF] [17/Sep/2026:22:55:33] ENGINE Bus STARTING
2026-09-17T22:55:33.110096+0000 mgr.h1.svgpaj (mgr.14889) 4 : cephadm [INF] [17/Sep/2026:22:55:33] ENGINE Serving on http://192.168.1.161:8765
2026-09-17T22:55:33.228225+0000 mgr.h1.svgpaj (mgr.14889) 5 : cephadm [INF] [17/Sep/2026:22:55:33] ENGINE Serving on https://192.168.1.161:7150
2026-09-17T22:55:33.228357+0000 mgr.h1.svgpaj (mgr.14889) 6 : cephadm [INF] [17/Sep/2026:22:55:33] ENGINE Bus STARTED
2026-09-17T22:55:33.228874+0000 mgr.h1.svgpaj (mgr.14889) 7 : cephadm [INF] [17/Sep/2026:22:55:33] ENGINE Client ('192.168.1.161', 55848) lost — peer dropped the TLS connection suddenly, during handshake: (6, 'TLS/SSL connection has been closed (EOF) (_ssl.c:1147)')
2026-09-17T22:55:33.745807+0000 mgr.h1.svgpaj (mgr.14889) 9 : cephadm [INF] disk rescan request sent to host "h1"
2026-09-17T22:55:36.281869+0000 mgr.h1.svgpaj (mgr.14889) 11 : cephadm [INF] disk rescan: Ok. 7 adapters detected: 7 rescanned, 0 skipped, 0 failed (1.88s)
2026-09-17T22:59:49.656961+0000 mgr.h1.svgpaj (mgr.14889) 214 : cephadm [ERR] host unconfigured does not exist
Traceback (most recent call last):

Does this log reflect new config changes?

To cut the story shorter, I managed to rule out hostname changes, including verifying if DNS resolution was playing the villain, and landed on another theory that there were some authentication errors based on logs. So I also went through the route of regenerating ssh keys, populating them so all mon, mgr, mds, rgw and osd can share and use them and verified SSH executions, key lookups and host scanning (all coming back: Ok 7 adapters detected: 7 rescanned).

  • DNS resolution?

As this is an air-gapped env., the original podman images of cephadm was imported from a static image. Certainly I have set the image to “ceph config set global container_image quay.io/ceph/ceph:v20.2.0”. Through the course of debugging, I then saw

1
2
3
4
5

podman exec -ti 66b0dd44ef08 ceph-volume lvm create --data /dev/sdb
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph-authtool --gen-print-key
Running command: /usr/bin/ceph --cluster ceph --name client.bootstrap-osd --keyring /var/lib/ceph/bootstrap-osd/ceph.keyring -i - osd new d8f824d2-353b-45c6-879c-4f872e7650f0
 stderr: 2026-09-18T08:25:16.237+0000 725b03bff640 -1 auth: unable to find a keyring on /var/lib/ceph/bootstrap-osd/ceph.keyring

Now the hint is about keyring not available to ceph.

  • SSH & auth keys?

I’ve tried to create a new ceph.keyring, making it visible in the container and attempted to manually create osd, etc.

1
2
3

ceph auth get-or-create client.bootstrap-osd mon 'allow profile bootstrap-osd' -o /var/lib/ceph/bootstrap-osd/ceph.keyring
podman exec -ti 66b0dd44ef08 ls -la /var/lib/ceph/bootstrap-osd/ceph.keyring
podman exec -ti 66b0dd44ef08 ceph-volume lvm create --data /dev/sdb

At this point I was of the belief that missing keyring is the root cause of all these troubles as cephadm and its components could not communicate with each other and host and were unable to see all resources/devices. I tried to write the keyring directly into the MON container’s mounted data volume path.

1
2

mkdir -p /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.unconfigured/data/bootstrap-osd
ceph auth get-or-create client.bootstrap-osd mon 'allow profile bootstrap-osd' -o /var/lib/ceph/c6e64c49-09cf-463b-9d1c-b6645b4b3b85/mon.unconfigured/data/bootstrap-osd/ceph.keyring

Such trials and errors went on. Well, no. Nothing I did actually helped. I knew I was lost.

With AI tools’ help, I had also gone through these validations:

  • CherryPy shared-bus-restart theory?

By comparing closely the logs of working & failing Ceph cluster with orchestrator deployments, we saw the mgr’s cephadm module was seen restarting 3 times in a minute right around ceph_mgr_module_enable calls, theorizing dashboard/restful/prometheus sharing one HTTP engine disrupted the async loop. Disproven — reproduced identically on a fresh bootstrap with the dashboard never enabled and prometheus/telemetry explicitly disabled.

  • Direct syscall-level evidence (no py-spy/gdb/strace available offline)?

With agents doing monitoring work, they read /proc//syscall and /proc//task/*/syscall directly — 30 samples over 9s, full per-thread scan of ~175 mgr threads, everything parked on the same futex. Ruled out "actively blocked mid-DNS-lookup" and confirmed a stable deadlock, not a slow-but-progressing call.

  • Known upstream Ceph bug (found via web search)?

Still clueless, I couldn’t help but wondering if it’s actually a bug of this Ceph release. A bit of searching came back with some known issues of mgr/prometheus’s serve() doing a synchronous mon_command() with no timeout, capable of wedging the whole mgr (shared GIL). I didn’t enable prometheus anyway so there’s nothing concrete.

  • Root cause, found by source tracing (should’ve done this earlier :/). Extracted cephadm’s own zipapp (python3 -m zipapp) and grepped its mgr module source for the reconcile engine:
    • serve.py’s convert_tags_to_repo_digest() iterates every configured container_image (mon/mgr/mds plain tag quay.io/ceph/ceph:v20.2.0, not a digest) and calls _get_container_image_info() to resolve it — which needs to actually contact quay.io.
    • registry.py’s get_token() does requests.get(realm, params=params) with zero timeout — Python’s requests [does not enforce a default timeout] so, in a genuinely air-gapped network, this can hang forever, not just for a bounded DNS-retry window.
    • Manually testing cephadm inspect-image/cephadm –image … inspect-image (the fallback path) [worked correctly] — ruling out that specific path, but not the periodic re-resolution inside the normal reconcile loop.

Decisive confirmation: captured two full journalctl logs from the same deployment flow — once with internet up, once after cutting it — and diffed them. Both were byte-for-byte identical through the 5th cephadm-module reload’s ENGINE Bus STARTED line. Immediately after that: with internet, real daemon deploys began within seconds; without it, the cephadm-channel log simply stopped forever. This pinned the stall to exactly the transition where the orchestrator first needs to resolve an image reference — matching the convert_tags_to_repo_digest() finding.

I finally got a narrow enough clue that directly linked to our biggest suspect, Internet access. The image and digest. With further fiddling and poking I magically got the right configs I actually need

1
2

[h1 ~]# ceph config set global container_image <local-podman-image-id>
[h1 ~]# ceph config set mgr mgr/cephadm/use_repo_digest false

These 2 settings are the final piece of the puzzle

ceph orch ls, ceph -s, ceph osd tree

During the third or maybe fourth day when I was stuck by this issue, I had seriously considered if I should operate Ceph in dual modes, online with cephadm and offline with legacy but what if an online deployment decides to later go offline? I went back to Ceph’s documents which were positive about this direction of cephadm. It simply could not be true that such a bug exists without being screamed out in the public. Anyway, it’s gotta be my configs somewhere I ought to pursue. Now I have the honor (clear path) to explore more on how ceph orch works with auto scaling, upgrades and rolling-updates. The first noticeable gain, was ceph orch apply osd –all-available-devices which did all the heavy lifting work and brought the osds right up as shown in the screenshot above. This is a big advancement to me as opposed to legacy-native mode when I had to write all tools and functions to automate the whole disk preparation, partitioning, formatting, daemon services, key synching, /var/lib/ceph/osd arrangements, mounting, etc.

Tech
Storage
This post is licensed under CC BY 4.0 by the author.

Trending Tags

CI/CD Security minds Storage