cert requests not handled when the original leader vault is not available
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
vault-charm |
Fix Released
|
High
|
Cory Johns |
Bug Description
After leader unit of vault is unavailable or removed for whatever reason,
adding unit to kubernetes-master will be stuck in "Waiting for master components to start" status.
The steps to reproduce are as follows.
1. Deploy CDK with Vault HA
I used this bundle.
2. Remove or take the leader unit of vault down
$ juju run -a vault is-leader
- Stdout: |
False
UnitId: vault/0
- Stdout: |
True
UnitId: vault/1
- Stdout: |
False
UnitId: vault/2
$ juju remove-unit --force vault/1
3. Add unit for kubernetes-master
$ juju add-unit kubernetes-master
After a while, the added kubernetes-master unit will be stuck in "Waiting for master components to start" status.
Since the "tls_client.
I believe update_certs() function[0] somehow fails to retrieve the certificates from Vault when the original leader unit is not there.
$ juju run -a kubernetes-master -- "charms.reactive -p get_flags | grep tls_client.
- Stdout: |2
'tls_
UnitId: kubernetes-master/0
- ReturnCode: 1
Stdout: ""
UnitId: kubernetes-master/1
$ juju run -a kubernetes-master -- sudo ls -al /root/cdk
- Stdout: |
total 52
drwxrwx--- 4 root root 4096 Jul 12 09:22 .
drwx------ 7 root root 4096 Jul 12 09:29 ..
drwxr-xr-x 2 root root 4096 Jul 12 09:23 audit
-rw-r--r-- 1 root root 61 Jul 12 09:05 basic_auth.csv
-r--r----- 1 root root 1245 Jul 12 09:22 ca.crt
-rw-r--r-- 1 root root 1406 Jul 12 09:22 client.crt
-rw-r--r-- 1 root root 1678 Jul 12 09:22 client.key
drwxr-xr-x 2 root root 4096 Jul 12 09:22 etcd
-rw-r--r-- 1 root root 385 Jul 12 09:10 known_tokens.csv
-rw------- 1 root root 2014 Jul 12 09:33 kubeproxyconfig
-rw-r--r-- 1 root root 1670 Jul 12 09:22 server.crt
-rw-r--r-- 1 root root 1674 Jul 12 09:22 server.key
-rw------- 1 root root 1675 Jul 12 09:05 serviceaccount.key
UnitId: kubernetes-master/0
- Stdout: |
total 20
drwxr-xr-x 2 root root 4096 Jul 12 09:40 .
drwx------ 6 root root 4096 Jul 12 09:46 ..
-rw-r--r-- 1 root root 60 Jul 12 10:11 basic_auth.csv
-rw-r--r-- 1 root root 385 Jul 12 10:11 known_tokens.csv
-rw-r--r-- 1 root root 1675 Jul 12 10:11 serviceaccount.key
UnitId: kubernetes-master/1
[0] https:/
Changed in charm-kubernetes-master: | |
importance: | Undecided → Critical |
Changed in vault-charm: | |
assignee: | nobody → Cory Johns (johnsca) |
status: | New → In Progress |
Changed in vault-charm: | |
status: | In Progress → Fix Committed |
milestone: | none → 19.07 |
importance: | Undecided → High |
status: | Fix Committed → Fix Released |
Here is the bundle that I have used.