Juju Charms Collection
ceph-osd package

osd_upgrade_key relation data from ceph is actually a list of commands (can't decode key)

Bug #1665149 reported by Ryan Beisner on 2017-02-15

This bug affects 3 people

	Status	Importance	Assigned to	Milestone
Autopilot Log Analyser	Fix Committed	High	Chad Smith
Ceph Monitor Charm	Fix Released	Critical	Chris Holcombe	Ceph Monitor Charm 17.02
Ceph OSD Charm	Fix Released	Critical	Chris Holcombe	Ceph OSD Charm 17.02
Landscape Server	Fix Released	High	Unassigned	Landscape Server 17.02
OPNFV	Fix Released	Critical	Unassigned
OpenStack Ceph Charm (Retired)	Fix Released	Critical	Chris Holcombe	OpenStack Ceph Charm (Retired) 17.02
charms.ceph	Fix Released	Critical	Chris Holcombe
ceph (Juju Charms Collection)	Invalid	Critical	Chris Holcombe
ceph-mon (Juju Charms Collection)	Invalid	Critical	Chris Holcombe
ceph-osd (Juju Charms Collection)	Invalid	Critical	Chris Holcombe

Bug Description

A list of commands is passed to ceph-authtool as a key, and it naturally "can't decode key"

2017-02-15 19:27:18 INFO mon-relation-changed can't decode key '"allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del""'

ceph-osd/0 error idle 1.25.9.1 0 talon.maas hook failed: "mon-relation-changed" for ceph:osd

2017-02-15 19:27:18 INFO mon-relation-changed Traceback (most recent call last):
2017-02-15 19:27:18 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/mon-relation-changed", line 520, in <module>
2017-02-15 19:27:18 INFO mon-relation-changed hooks.execute(sys.argv)
2017-02-15 19:27:18 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/charmhelpers/core/hookenv.py", line 731, in execute
2017-02-15 19:27:18 INFO mon-relation-changed self._hooks[hook_name]()
2017-02-15 19:27:18 INFO mon-relation-changed File "/var/lib/juju/agents/unit-ceph-osd-0/charm/hooks/mon-relation-changed", line 446, in mon_relation
2017-02-15 19:27:18 INFO mon-relation-changed ceph.import_osd_upgrade_key(upgrade_key)
2017-02-15 19:27:18 INFO mon-relation-changed File "lib/ceph/__init__.py", line 862, in import_osd_upgrade_key
2017-02-15 19:27:18 INFO mon-relation-changed subprocess.check_call(cmd)
2017-02-15 19:27:18 INFO mon-relation-changed File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
2017-02-15 19:27:18 INFO mon-relation-changed raise CalledProcessError(retcode, cmd)
2017-02-15 19:27:18 INFO mon-relation-changed subprocess.CalledProcessError: Command '['sudo', '-u', 'ceph', 'ceph-authtool', '/var/lib/ceph/osd/ceph.client.osd-upgrade.keyring', '--create-keyring', '--name=client.osd-upgrade', '--add-key="allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del""']' returned non-zero exit status 1
2017-02-15 19:27:18 ERROR juju.worker.uniter.operation runhook.go:107 hook "mon-relation-changed" failed: exit status 1

Tags:

Ryan Beisner (1chb1n) on 2017-02-15

no longer affects:

ceph (Ubuntu)

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-15:

juju-stat-tabular-collect.txt Edit (9.3 KiB, text/plain)

summary:

- osd_upgrade_key relation data from received from ceph is actually a list
- of commands (can't decode key)
+ osd_upgrade_key relation data from ceph is actually a list of commands
+ (can't decode key)

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-15:

juju-stat-yaml-collect.txt Edit (27.6 KiB, text/plain)

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-15:

ceph-0-var-log (3).tar.bz2 Edit (154.9 KiB, application/x-tar)

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-15:

ceph-osd-0-var-log (10).tar.bz2 Edit (468.5 KiB, application/x-tar)

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-15:

This was observed on Xenial-Mitaka but appears to be impacting all mojo spec metal tests.

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-15:

Bundle which produces this:

http://bazaar.launchpad.net/~ost-maintainers/openstack-mojo-specs/mojo-openstack-specs/view/head:/helper/bundles/baremetal7-next.yaml

Revision history for this message

Chris Holcombe (xfactor973) wrote on 2017-02-16:

I might be crazy but it appears this line has never worked: https://github.com/openstack/charms.ceph/blob/master/ceph/__init__.py#L909. I'm wondering if we called the ceph auth get-or-create like the get_named_keyring file if that would work.

Narinder Gupta (narindergupta) on 2017-02-16

Changed in opnfv:
importance:	Undecided → Critical

Ryan Beisner (1chb1n) on 2017-02-16

Changed in charms.ceph:
assignee:	nobody → Chris Holcombe (xfactor973)
Changed in ceph (Juju Charms Collection):
assignee:	nobody → Chris Holcombe (xfactor973)
Changed in ceph-mon (Juju Charms Collection):
assignee:	nobody → Chris Holcombe (xfactor973)
Changed in ceph-osd (Juju Charms Collection):
assignee:	nobody → Chris Holcombe (xfactor973)
Changed in ceph (Juju Charms Collection):
importance:	Undecided → Critical
Changed in ceph-osd (Juju Charms Collection):
importance:	Undecided → Critical
Changed in ceph (Juju Charms Collection):
status:	New → Confirmed
Changed in ceph-osd (Juju Charms Collection):
status:	New → Confirmed

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-16:

https://github.com/openstack/charm-ceph-osd

Last known good combo was Feb 14 on these revs:
cs:~openstack-charmers-next/ceph-osd-264 (4ae0bd8)
cs:~openstack-charmers-next/ceph-271 (3dfeff7)

This issue started Feb 15 on these revs:
cs:~openstack-charmers-next/ceph-274 (459ccae)
cs:~openstack-charmers-next/ceph-osd-265 (9af7679)

Revision history for this message

Ryan Beisner (1chb1n) wrote on 2017-02-16:

#10

https://github.com/openstack/charm-ceph

Revision history for this message

Chris MacNaughton (chris.macnaughton) wrote on 2017-02-16:

#11

Quite strange, it looks like ceph-0 successfully ran the command to create that key:

2017-02-15 19:19:57 DEBUG juju-log osd:50: Calling subprocess.check_output: ['sudo', '-u', 'ceph', 'ceph', '--name', 'mon.', '--keyring', '/var/lib/ceph/mon/ceph-thump/keyring', 'auth', 'get-or-create', 'client.osd-upgrade', 'mon', 'allow command "config-key"; allow command "osd tree"; allow command "config-key list"; allow command "config-key put"; allow command "config-key get"; allow command "config-key exists"; allow command "osd out"; allow command "osd in"; allow command "osd rm"; allow command "auth del"']

This was called within the get_named_key function. I have changed that function recently but the logs indicate that it progressed through that function correctly, given that there was no existing key for it.

I'm still digging into why the command, rather than the key, was sent over the relation

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-16: Fix proposed to charms.ceph (master)

#12

Fix proposed to branch: master
Review: https://review.openstack.org/435066

Changed in charms.ceph:
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-16: Fix merged to charms.ceph (master)

#13

Reviewed: https://review.openstack.org/435066
Committed: https://git.openstack.org/cgit/openstack/charms.ceph/commit/?id=e422e65fc98b60417573ce2a6f10e447a9f7d54f
Submitter: Jenkins
Branch: master

commit e422e65fc98b60417573ce2a6f10e447a9f7d54f
Author: Chris Holcombe <email address hidden>
Date: Thu Feb 16 11:00:50 2017 -0800

Fix parse_key

If the returned cephx key from ceph contains [caps] the
parse_key function returns the wrong output.

Partial-Bug: 1665149
Change-Id: Icca4c174fa15b9637732a919ab5149f34260ae0f

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-16: Fix proposed to charm-ceph (master)

#14

Fix proposed to branch: master
Review: https://review.openstack.org/435084

Changed in ceph (Juju Charms Collection):
status:	Confirmed → In Progress
Changed in ceph-mon (Juju Charms Collection):
status:	New → In Progress

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-16: Fix proposed to charm-ceph-mon (master)

#15

Fix proposed to branch: master
Review: https://review.openstack.org/435085

James Page (james-page) on 2017-02-17

Changed in charms.ceph:
status:	In Progress → Fix Released
Changed in ceph-osd (Juju Charms Collection):
status:	Confirmed → Triaged

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-17: Fix merged to charm-ceph-mon (master)

#16

Reviewed: https://review.openstack.org/435085
Committed: https://git.openstack.org/cgit/openstack/charm-ceph-mon/commit/?id=1d0406f0c4b9352b0df2a5b3c2abbc65f86cff4a
Submitter: Jenkins
Branch: master

commit 1d0406f0c4b9352b0df2a5b3c2abbc65f86cff4a
Author: Chris Holcombe <email address hidden>
Date: Thu Feb 16 11:47:01 2017 -0800

Fix parse_key

parse_key returns incorrect data when the cephx key
contains [caps].

Change-Id: I61aa5fd888e19f778151239436fb7654d7cc48b5
Closes-Bug: 1665149

Changed in ceph-mon (Juju Charms Collection):
status:	In Progress → Fix Committed
Changed in ceph (Juju Charms Collection):
status:	In Progress → Fix Committed

Revision history for this message

OpenStack Infra (hudson-openstack) wrote on 2017-02-17: Fix merged to charm-ceph (master)

#17

Reviewed: https://review.openstack.org/435084
Committed: https://git.openstack.org/cgit/openstack/charm-ceph/commit/?id=ea87ae4feb2dd673c979b8610848efcd6dd33cfa
Submitter: Jenkins
Branch: master

commit ea87ae4feb2dd673c979b8610848efcd6dd33cfa
Author: Chris Holcombe <email address hidden>
Date: Thu Feb 16 11:49:09 2017 -0800

Fix parse_key

parse_key returns incorrect data when the cephx key
contains [caps].

Closes-Bug: 1665149
Change-Id: I0b5a9d3db94a43f60e785c3c2cb73a7ed84e208b

Chad Smith (chad.smith) on 2017-02-17

Changed in landscape:
milestone:	none → 17.02
importance:	Undecided → High
status:	New → Confirmed

Chris MacNaughton (chris.macnaughton) on 2017-02-17

Changed in ceph-osd (Juju Charms Collection):
status:	Triaged → Fix Committed

Chris MacNaughton (chris.macnaughton) on 2017-02-17

Changed in ceph-mon (Juju Charms Collection):
importance:	Undecided → Critical
Changed in charms.ceph:
importance:	Undecided → Critical

Chad Smith (chad.smith) on 2017-02-17

Changed in autopilot-log-analyser:
assignee:	nobody → Chad Smith (chad.smith)
importance:	Undecided → High
status:	New → In Progress

🤖 Landscape Builder (landscape-builder) on 2017-02-17

Changed in autopilot-log-analyser:
status:	In Progress → Fix Committed

James Page (james-page) on 2017-02-23

Changed in charm-ceph:
assignee:	nobody → Chris Holcombe (xfactor973)
importance:	Undecided → Critical
status:	New → Fix Committed
Changed in ceph (Juju Charms Collection):
status:	Fix Committed → Invalid
Changed in charm-ceph-mon:
assignee:	nobody → Chris Holcombe (xfactor973)
importance:	Undecided → Critical
status:	New → Fix Committed
Changed in ceph-mon (Juju Charms Collection):
status:	Fix Committed → Invalid
Changed in charm-ceph-osd:
assignee:	nobody → Chris Holcombe (xfactor973)
importance:	Undecided → Critical
status:	New → Fix Committed
Changed in ceph-osd (Juju Charms Collection):
status:	Fix Committed → Invalid

James Page (james-page) on 2017-02-23

Changed in charm-ceph:
milestone:	none → 17.02
Changed in charm-ceph-mon:
milestone:	none → 17.02
Changed in charm-ceph-osd:
milestone:	none → 17.02

James Page (james-page) on 2017-02-23

Changed in charm-ceph:
status:	Fix Committed → Fix Released
Changed in charm-ceph-mon:
status:	Fix Committed → Fix Released
Changed in charm-ceph-osd:
status:	Fix Committed → Fix Released

Chad Smith (chad.smith) on 2017-03-16

Changed in landscape:
status:	Confirmed → Fix Committed

Andreas Hasenack (ahasenack) on 2017-04-06

Changed in landscape:
status:	Fix Committed → Fix Released

Narinder Gupta (narindergupta) on 2017-09-07

Changed in opnfv:
status:	New → Fix Released

Report a bug

This report contains Public information

Everyone can see this information.

You are

Subscribing...

Edit bug mail

Other bug subscribers

Bug attachments

Add attachment

Remote bug watches

Bug watches keep track of this bug in other bug trackers.

Juju Charms Collectionceph-osd package

osd_upgrade_key relation data from ceph is actually a list of commands (can't decode key)

Bug Description

Other bug subscribers

Bug attachments

Remote bug watches

Juju Charms Collection
ceph-osd package