radosgw crash on armhf architecture

Bug #1049582 reported by James Page
8
This bug affects 1 person
Affects Status Importance Assigned to Milestone
ceph (Ubuntu)
Fix Released
High
James Page

Bug Description

radosgw is non-functional on the armhf architecture; ceph starts up just fine.

--- begin dump of recent events ---
   -83> 2012-09-12 11:04:53.728795 b5eef3b0 5 asok(0xb8a96858) register_command perfcounters_dump hook 0xb8a977f8
   -82> 2012-09-12 11:04:53.728886 b5eef3b0 5 asok(0xb8a96858) register_command 1 hook 0xb8a977f8
   -81> 2012-09-12 11:04:53.728917 b5eef3b0 5 asok(0xb8a96858) register_command perf dump hook 0xb8a977f8
   -80> 2012-09-12 11:04:53.728978 b5eef3b0 5 asok(0xb8a96858) register_command perfcounters_schema hook 0xb8a977f8
   -79> 2012-09-12 11:04:53.728978 b5eef3b0 5 asok(0xb8a96858) register_command 2 hook 0xb8a977f8
   -78> 2012-09-12 11:04:53.729008 b5eef3b0 5 asok(0xb8a96858) register_command perf schema hook 0xb8a977f8
   -77> 2012-09-12 11:04:53.729008 b5eef3b0 5 asok(0xb8a96858) register_command config show hook 0xb8a977f8
   -76> 2012-09-12 11:04:53.729039 b5eef3b0 5 asok(0xb8a96858) register_command config set hook 0xb8a977f8
   -75> 2012-09-12 11:04:53.729069 b5eef3b0 5 asok(0xb8a96858) register_command log flush hook 0xb8a977f8
   -74> 2012-09-12 11:04:53.729069 b5eef3b0 5 asok(0xb8a96858) register_command log dump hook 0xb8a977f8
   -73> 2012-09-12 11:04:53.729100 b5eef3b0 5 asok(0xb8a96858) register_command log reopen hook 0xb8a977f8
   -72> 2012-09-12 11:04:53.765700 b5eef3b0 1 librados: starting msgr at :/0
   -71> 2012-09-12 11:04:53.765883 b5eef3b0 1 librados: starting objecter
   -70> 2012-09-12 11:04:53.766036 b5eef3b0 1 -- :/0 messenger.start
   -69> 2012-09-12 11:04:53.766158 b5eef3b0 1 librados: setting wanted keys
   -68> 2012-09-12 11:04:53.766188 b5eef3b0 1 librados: calling monclient init
   -67> 2012-09-12 11:04:53.766249 b5eef3b0 2 auth: CephX auth is not supported.
   -66> 2012-09-12 11:04:53.766738 b5eef3b0 1 -- :/1030304 --> 192.168.1.54:6789/0 -- auth(proto 0 40 bytes epoch 0) v1 -- ?+0 0xb8ac36d8 con 0xb8ac3470
   -65> 2012-09-12 11:04:53.767898 b1e94410 1 -- 192.168.1.54:0/1030304 learned my addr 192.168.1.54:0/1030304
   -64> 2012-09-12 11:04:53.769699 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 191 (0 -> 191)
   -63> 2012-09-12 11:04:53.770065 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 24 (191 -> 215)
   -62> 2012-09-12 11:04:53.770065 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 1 ==== mon_map v1 ==== 191+0+0 (1904510341 0 0) 0xb19005f8 con 0xb8ac3470
   -61> 2012-09-12 11:04:53.770218 b3e94410 1 monclient(hunting): found mon.a
   -60> 2012-09-12 11:04:53.770248 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 191 (0xb667696c -> 24)
   -59> 2012-09-12 11:04:53.770279 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 2 ==== auth_reply(proto 1 0 Success) v1 ==== 24+0+0 (3158048645 0 0) 0xb19008a8 con 0xb8ac3470
   -58> 2012-09-12 11:04:53.770401 b3e94410 1 -- 192.168.1.54:0/1030304 --> 192.168.1.54:6789/0 -- mon_subscribe({monmap=0+}) v2 -- ?+0 0xb8ac3968 con 0xb8ac3470
   -57> 2012-09-12 11:04:53.770492 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 24 (0xb667696c -> 0)
   -56> 2012-09-12 11:04:53.770523 b5eef3b0 5 monclient: authenticate success, global_id 4208
   -55> 2012-09-12 11:04:53.770737 b5eef3b0 5 asok(0xb8a96858) register_command objecter_requests hook 0xb8aa7f98
   -54> 2012-09-12 11:04:53.770798 b5eef3b0 1 -- 192.168.1.54:0/1030304 --> 192.168.1.54:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0xb8ac40a8 con 0xb8ac3470
   -53> 2012-09-12 11:04:53.770859 b5eef3b0 1 -- 192.168.1.54:0/1030304 --> 192.168.1.54:6789/0 -- mon_subscribe({monmap=2+,osdmap=0}) v2 -- ?+0 0xb8ac4338 con 0xb8ac3470
   -52> 2012-09-12 11:04:53.770889 b5eef3b0 1 librados: waiting for osdmap
   -51> 2012-09-12 11:04:53.771378 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 191 (0 -> 191)
   -50> 2012-09-12 11:04:53.771500 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 20 (191 -> 211)
   -49> 2012-09-12 11:04:53.771591 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 3 ==== mon_map v1 ==== 191+0+0 (1904510341 0 0) 0xb19008a8 con 0xb8ac3470
   -48> 2012-09-12 11:04:53.771652 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 191 (0xb667696c -> 20)
   -47> 2012-09-12 11:04:53.771683 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 4 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1612381846 0 0) 0xb1900aa8 con 0xb8ac3470
   -46> 2012-09-12 11:04:53.771713 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 20 (0xb667696c -> 0)
   -45> 2012-09-12 11:04:53.772538 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 2249 (0 -> 2249)
   -44> 2012-09-12 11:04:53.772721 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 20 (2249 -> 2269)
   -43> 2012-09-12 11:04:53.772721 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 5 ==== osd_map(16..16 src has 1..16) v3 ==== 2249+0+0 (1610641032 0 0) 0xb19005b0 con 0xb8ac3470
   -42> 2012-09-12 11:04:53.773179 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 2249 (0xb667696c -> 20)
   -41> 2012-09-12 11:04:53.773209 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 6 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1612381846 0 0) 0xb19013a8 con 0xb8ac3470
   -40> 2012-09-12 11:04:53.773240 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 20 (0xb667696c -> 0)
   -39> 2012-09-12 11:04:53.773392 b5eef3b0 1 librados: init done
   -38> 2012-09-12 11:04:53.773514 b5eef3b0 5 throttle(objecter_bytes 0xb8ac2430) get_or_fail 0 success (0 -> 0)
   -37> 2012-09-12 11:04:53.773514 b5eef3b0 5 throttle(objecter_ops 0xb8ac2480) get_or_fail 1 success (0 -> 1)
   -36> 2012-09-12 11:04:53.773728 b5eef3b0 1 -- 192.168.1.54:0/1030304 --> 192.168.1.54:6805/29517 -- osd_op(client.4208.0:1 notify [create 0~0] 4.1740577a) v4 -- ?+0 0xb8ac4f30 con 0xb8ac4da0
   -35> 2012-09-12 11:04:53.774094 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 2249 (0 -> 2249)
   -34> 2012-09-12 11:04:53.774735 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 7 ==== osd_map(16..16 src has 1..16) v3 ==== 2249+0+0 (1610641032 0 0) 0xb19005b0 con 0xb8ac3470
   -33> 2012-09-12 11:04:53.774827 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 2249 (0xb667696c -> 0)
   -32> 2012-09-12 11:04:53.775132 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 20 (0 -> 20)
   -31> 2012-09-12 11:04:53.775254 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 8 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1612381846 0 0) 0xb1900aa8 con 0xb8ac3470
   -30> 2012-09-12 11:04:53.775315 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 20 (0xb667696c -> 0)
   -29> 2012-09-12 11:04:53.777666 b07ff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 105 (0 -> 105)
   -28> 2012-09-12 11:04:53.777910 b3e94410 1 -- 192.168.1.54:0/1030304 <== osd.1 192.168.1.54:6805/29517 1 ==== osd_op_reply(1 notify [create 0~0] ondisk = 0) v4 ==== 105+0+0 (4157653173 0 0) 0xb1a02c20 con 0xb8ac4da0
   -27> 2012-09-12 11:04:53.778032 b3e94410 5 throttle(objecter_bytes 0xb8ac2430) put 0 (0xb667696c -> 0)
   -26> 2012-09-12 11:04:53.778063 b3e94410 5 throttle(objecter_ops 0xb8ac2480) put 1 (0xb667696c -> 0)
   -25> 2012-09-12 11:04:53.778093 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 105 (0xb667696c -> 0)
   -24> 2012-09-12 11:04:53.778185 b5eef3b0 5 throttle(objecter_bytes 0xb8ac2430) get_or_fail 0 success (0 -> 0)
   -23> 2012-09-12 11:04:53.778215 b5eef3b0 5 throttle(objecter_ops 0xb8ac2480) get_or_fail 1 success (0 -> 1)
   -22> 2012-09-12 11:04:53.778246 b5eef3b0 1 -- 192.168.1.54:0/1030304 --> 192.168.1.54:6805/29517 -- osd_op(client.4208.0:2 notify [watch 1~0] 4.1740577a) v4 -- ?+0 0xb8ac5510 con 0xb8ac4da0
   -21> 2012-09-12 11:04:53.838198 b07ff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 105 (0 -> 105)
   -20> 2012-09-12 11:04:53.838472 b3e94410 1 -- 192.168.1.54:0/1030304 <== osd.1 192.168.1.54:6805/29517 2 ==== osd_op_reply(2 notify [watch 1~0] = 0) v4 ==== 105+0+0 (223526802 0 0) 0xb1a031b0 con 0xb8ac4da0
   -19> 2012-09-12 11:04:53.838564 b3e94410 5 throttle(objecter_bytes 0xb8ac2430) put 0 (0xb667696c -> 0)
   -18> 2012-09-12 11:04:53.838564 b3e94410 5 throttle(objecter_ops 0xb8ac2480) put 1 (0xb667696c -> 0)
   -17> 2012-09-12 11:04:53.838625 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 105 (0xb667696c -> 0)
   -16> 2012-09-12 11:04:53.847783 b5eef3b0 10 allocated request req=0xb8ace720
   -15> 2012-09-12 11:04:53.847844 b5eef3b0 5 throttle(rgw_ops 0xbe839cc0) get 1 (0 -> 1)
   -14> 2012-09-12 11:04:58.771044 b18ff410 1 -- 192.168.1.54:0/1030304 --> 192.168.1.54:6789/0 -- mon_subscribe({monmap=2+,osdmap=17}) v2 -- ?+0 0xb8ace908 con 0xb8ac3470
   -13> 2012-09-12 11:04:58.771349 b18ff410 1 -- 192.168.1.54:0/1030304 --> 192.168.1.54:6805/29517 -- ping v1 -- ?+0 0xb8aceb78 con 0xb8ac4da0
   -12> 2012-09-12 11:04:58.774310 b1bff410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) get 20 (0 -> 20)
   -11> 2012-09-12 11:04:58.774677 b3e94410 1 -- 192.168.1.54:0/1030304 <== mon.0 192.168.1.54:6789/0 9 ==== mon_subscribe_ack(300s) v1 ==== 20+0+0 (1612381846 0 0) 0xb1904918 con 0xb8ac3470
   -10> 2012-09-12 11:04:58.774829 b3e94410 5 throttle(msgr_dispatch_throttler-radosclient 0xb8ac1838) put 20 (0xb667696c -> 0)
    -9> 2012-09-12 11:04:59.837480 b5eef3b0 20 enqueued request req=0xb8ace720
    -8> 2012-09-12 11:04:59.837572 b5eef3b0 20 RGWWQ:
    -7> 2012-09-12 11:04:59.837602 b5eef3b0 20 req: 0xb8ace720
    -6> 2012-09-12 11:04:59.837633 b5eef3b0 10 allocated request req=0xb8ad3448
    -5> 2012-09-12 11:04:59.837663 b5eef3b0 5 throttle(rgw_ops 0xbe839cc0) get 1 (1 -> 2)
    -4> 2012-09-12 11:04:59.838060 b06ff410 20 dequeued request req=0xb8ace720
    -3> 2012-09-12 11:04:59.838091 b06ff410 20 RGWWQ: empty
    -2> 2012-09-12 11:04:59.838701 b06ff410 2 req 1:0.000488::::initializing
    -1> 2012-09-12 11:04:59.838213 b06ff410 1 ====== starting new request req=0xb8ace720 =====
     0> 2012-09-12 11:04:59.843768 b06ff410 -1 *** Caught signal (Aborted) **
 in thread b06ff410

 ceph version (commit:)
 1: (()+0x4c0a6) [0xb6ef50a6]
 2: (__default_sa_restorer_v2()+0) [0xb64879d0]
 3: (()+0x171e6) [0xb64791e6]
 4: (gsignal()+0x29) [0xb6486db2]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- end dump of recent events ---

ProblemType: Bug
DistroRelease: Ubuntu 12.10
Package: radosgw 0.48.1-0ubuntu1
ProcVersionSignature: Ubuntu 3.5.0-209.15-omap4 3.5.2
Uname: Linux 3.5.0-209-omap4 armv7l
ApportVersion: 2.5.1-0ubuntu7
Architecture: armhf
Date: Wed Sep 12 11:07:32 2012
SourcePackage: ceph
UpgradeStatus: Upgraded to quantal on 2012-06-06 (97 days ago)

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Installing debug symbols to see if I can get a better trace.

Revision history for this message
James Page (james-page) wrote :
Revision history for this message
James Page (james-page) wrote :

Full backtrace captured and attached to bug report.

Dave Walker (davewalker)
Changed in ceph (Ubuntu):
importance: Undecided → High
Revision history for this message
Yehuda Sadeh (yehudasa) wrote :

I can't really reproduced, don't have a similar setup ready. However, analyzing the backtrace and the code, the following may fix the issue:
https://github.com/ceph/ceph/commit/2fddfb0a95288d39cee2e82376297ce43296743b

It has already been pushed to the ceph stable branch, though it's not in any tagged version yet.

Revision history for this message
James Page (james-page) wrote :

Thanks Yehuda

I'll patch this in a give it a try.

Changed in ceph (Ubuntu):
assignee: nobody → James Page (james-page)
status: New → In Progress
Revision history for this message
James Page (james-page) wrote :

Fix worked for me, I'll upload a new version of ceph ASAP.

Changed in ceph (Ubuntu):
status: In Progress → Fix Committed
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package ceph - 0.48.2-0ubuntu2

---------------
ceph (0.48.2-0ubuntu2) quantal; urgency=low

  * Fix crash in radosgw on armhf architecture (LP: #1049582):
    - debian/patches/extra-byte-for-null.patch: Cherry picked patch
      from upstream VCS. Thanks to Yehuda Sadeh for pointing to this
      fix.
  * d/ceph.lintian-overrides: Added libcls_rgw.so.1.0.0 to binaries
    which are not stripped.
 -- James Page <email address hidden> Wed, 26 Sep 2012 13:28:15 +0100

Changed in ceph (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.