Freebsd 10.0 can't start cluster server [terminate called after throwing an instance of 'gu::NotFound'] 5.5.34

Bug #1281956 reported by Byrnedo
12
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Galera
Fix Released
Undecided
Unassigned
Percona XtraDB Cluster moved to https://jira.percona.com/projects/PXC
Incomplete
Undecided
Unassigned

Bug Description

Using gcc/++48 built 5.5.34 latest, then built the galera source given in the same downloads page.

Built percona using the BUILD/compile-pentium64-wsrep and again just with cmake:
cmake . -DCMAKE_INSTALL_PREFIX=/usr/local/percona -DENABLE_ASSEMBLER=1 -DWITH_EXTRA_CHARSETS=complex -DENABLE_THREAD_SAFE_CLIENT=1 -DWITH_BIG_TABLES=1 -DWITH_READLINE=1 -DWITH_WSREP=1 -DENABLE_LOCAL_INFILE=1

I'm filing this with galera too on Rghavendra's suggestion.

Set provider to the galera so.
Set wsrep_cluster_address=gcomm:/ as it's the first node. ( it runs ok if I remove the 'gcomm:/' part, but then fails again once I try and add ip of another percona node later)
Started mysql server.
Get the error

##
terminate called after throwing an instance of 'gu::NotFound'
08:05:51 UTC - mysqld got signal 6 ;
##

uname -a
10.0-RELEASE FreeBSD 10.0-RELEASE #0 r260789:

Full .err log:
-------------------------------------------------------------------------------------------------------------------------------------
140219 09:05:51 mysqld_safe Starting mysqld daemon with databases from /var/db/mysql
140219 09:05:51 mysqld_safe Skipping wsrep-recover for empty datadir: /var/db/mysql
140219 09:05:51 mysqld_safe Assigning 00000000-0000-0000-0000-000000000000:-1 to wsrep_start_position
140219 9:05:51 [Note] WSREP: wsrep_start_position var submitted: '00000000-0000-0000-0000-000000000000:-1'
140219 9:05:51 [Note] WSREP: Setting wsrep_ready to 0
140219 9:05:51 [Note] WSREP: Read nil XID from storage engines, skipping position init
140219 9:05:51 [Note] WSREP: wsrep_load(): loading provider library '/usr/local/src/galera/percona-xtradb-cluster-galera/libgalera_smm.so'
140219 9:05:51 [Note] WSREP: wsrep_load(): Galera 2.8(rXXXX) by Codership Oy <email address hidden> loaded successfully.
140219 9:05:51 [Warning] WSREP: Could not open saved state file for reading: /var/db/mysql//grastate.dat
140219 9:05:51 [Note] WSREP: Found saved state: 00000000-0000-0000-0000-000000000000:-1
140219 9:05:51 [Note] WSREP: Preallocating 134219048/134219048 bytes in '/var/db/mysql//galera.cache'...
140219 9:05:51 [Note] WSREP: Passing config to GCS: base_host = <server ip>; base_port = 4567; cert.log_conflicts = no; gcache.dir = /var/db/mysql/; gcache.keep_pages_size = 0; gcache.mem_size = 0; gcache.name = /var/db/mysql//galera.cache; gcache.page_size = 128M; gcache.size = 128M; gcs.fc_debug = 0; gcs.fc_factor = 1; gcs.fc_limit = 16; gcs.fc_master_slave = NO; gcs.max_packet_size = 64500; gcs.max_throttle = 0.25; gcs.recv_q_hard_limit = 9223372036854775807; gcs.recv_q_soft_limit = 0.25; gcs.sync_donor = NO; replicator.causal_read_timeout = PT30S; replicator.commit_order = 3
140219 9:05:51 [Note] WSREP: Assign initial position for certification: -1, protocol version: -1
140219 9:05:51 [Note] WSREP: wsrep_sst_grab()
140219 9:05:51 [Note] WSREP: Start replication
140219 9:05:51 [Note] WSREP: Setting initial position to 00000000-0000-0000-0000-000000000000:-1
140219 9:05:51 [Note] WSREP: protonet asio version 0
140219 9:05:51 [Note] WSREP: backend: asio
terminate called after throwing an instance of 'gu::NotFound'
08:05:51 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.
Please help us make Percona XtraDB Cluster better by reporting any
bugs at https://bugs.launchpad.net/percona-xtradb-cluster

key_buffer_size=0
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
connection_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 334800 K bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
-------------------------------------------------------------------------------------------------------------------------------------

My my.cnf:
-------------------------------------------------------------------------------------------------------------------------------------
[mysqld]
wsrep_debug=1

datadir=/var/db/mysql
user=mysql

# Path to Galera library
wsrep_provider=/usr/local/src/galera/percona-xtradb-cluster-galera/libgalera_smm.so

# Empty gcomm address is being used when cluster is getting bootstrapped, the first server that is
wsrep_cluster_address=gcomm:/

# In order for Galera to work correctly binlog format should be ROW
binlog_format=ROW

# MyISAM storage engine has only experimental support
default_storage_engine=InnoDB

# This is a recommended tuning variable for performance
innodb_locks_unsafe_for_binlog=1

# This changes how InnoDB autoincrement locks are managed and is a requirement for Galera
innodb_autoinc_lock_mode=2

# Node #1 address
wsrep_node_address=<server ip>

# SST method
wsrep_sst_method=xtrabackup

# Cluster name
wsrep_cluster_name=xxxx

# Authentication for SST method
wsrep_sst_auth="xxxx:xxxx"

-------------------------------------------------------------------------------------------------------------------------------------

Tried the recommended patched source for galera:
http://t.co/1BnhyUtQhP
From this : https://bugs.launchpad.net/galera/+bug/1261996

No difference

Byrnedo (byrnedo)
description: updated
Revision history for this message
Teemu Ollakka (teemu-ollakka) wrote :

Config line

wsrep_cluster_address=gcomm:/

should be

wsrep_cluster_address=gcomm://

Revision history for this message
Teemu Ollakka (teemu-ollakka) wrote :

However, even if wsrep_cluster_address is not valid, process should not crash like that. Could you post also stack trace from error log, starting after line

"Hope that's ok; if not, decrease some variables in the equation."

Revision history for this message
Byrnedo (byrnedo) wrote : Re: [Bug 1281956] Re: Freebsd 10.0 can't start cluster server [terminate called after throwing an instance of 'gu::NotFound'] 5.5.34

Sorry, that was a typo, I changed it just to be sure and it still
crashes at startup.

On 2014-02-19 15:20, Teemu Ollakka wrote:
> Config line
>
> wsrep_cluster_address=gcomm:/
>
> should be
>
> wsrep_cluster_address=gcomm://
>

Revision history for this message
Byrnedo (byrnedo) wrote :

There was no stack trace in the log itself, is there an option i need to
pass to mysqld_safe to enable that?
I did build a debug binary and do a backtrace and got:

gdb) bt
#0 0x0000000801f7426a in ?? ()
#1 0x0000000801182fdb in ?? ()
#2 0x0000000802806400 in ?? ()
#3 0x00007fffffffb640 in ?? ()
#4 0x00007fffffffba20 in ?? ()
#5 0x0000000802806400 in ?? ()
#6 0x00007fffffffb6b0 in ?? ()
#7 0x0000000000000006 in ?? ()
#8 0x00007fffffffb1d0 in ?? ()
#9 0x00000000008cfac4 in my_strntol_mb2_or_mb4 (cs=0x802806400, nptr=<value optimized out>, l=<value optimized out>, base=-18880, endptr=0x246,
    err=0x800f5a0bd) at /usr/local/src/percona/Percona-XtraDB-Cluster-5.5.34/strings/ctype-ucs2.c:151

On 2014-02-19 15:27, Teemu Ollakka wrote:
> However, even if wsrep_cluster_address is not valid, process should not
> crash like that. Could you post also stack trace from error log,
> starting after line
>
> "Hope that's ok; if not, decrease some variables in the equation."
>

Revision history for this message
Teemu Ollakka (teemu-ollakka) wrote :

That backtrace does not make sense at all. Sounds like problem in build process, needs to be reproduced to troubleshoot further.

Revision history for this message
Raghavendra D Prabhu (raghavendra-prabhu) wrote :

@Byrnedo

a) How did you build galera?
b) Also, are both mysqld binary and libgalera_smm.so unstripped?

Revision history for this message
Byrnedo (byrnedo) wrote :

Hi Raghavendra,

I followed the steps on the galera site for compiling from source. I
built the deps and set CXX flags etc like they suggest.
No, only the mysqld binary was, maybe I can get a minute tomorrow to
build galera as debug.

//Donal

On 19/02/14 23:19, Raghavendra D Prabhu wrote:
> @Byrnedo
>
> a) How did you build galera?
> b) Also, are both mysqld binary and libgalera_smm.so unstripped?
>

Revision history for this message
Teemu Ollakka (teemu-ollakka) wrote :

I was able to build working Galera from lp:galera/2.x head, except that there was linking problem with garbd/libboost_program_options.a.

Build command from Galera source root:

./scripts/build.sh -b -o

Patch to apply:

=== modified file 'SConscript'
--- SConscript 2013-08-23 13:12:53 +0000
+++ SConscript 2014-02-19 22:07:58 +0000
@@ -2,8 +2,8 @@
             'gcache/SConscript',
             'gcomm/SConscript',
             'gcs/SConscript',
- 'galera/SConscript',
- 'garb/SConscript'])
+ 'galera/SConscript'])
+# 'garb/SConscript'])

 Import('env', 'sysname')

=== modified file 'galerautils/src/gu_errno.h'
--- galerautils/src/gu_errno.h 2013-08-23 13:12:53 +0000
+++ galerautils/src/gu_errno.h 2014-02-19 21:50:17 +0000
@@ -13,7 +13,9 @@
 # define ENOTUNIQ (ELAST+3)
 # define ERESTART (ELAST+4)
 # if defined(__FreeBSD__)
+# ifndef ENOTRECOVERABLE
 # define ENOTRECOVERABLE (ELAST+5)
+#endif /* !ENOTRECOVERABLE */
 # define ENODATA (ELAST+6)
 # endif
 #endif

=== modified file 'scripts/mysql/build.sh'
--- scripts/mysql/build.sh 2013-12-01 16:23:23 +0000
+++ scripts/mysql/build.sh 2014-02-20 12:12:58 +0000
@@ -627,7 +627,7 @@
     if [ "$SCONS" == "yes" ]
     then
         SCONS_VD=$GALERA_SRC
- cp -P $SCONS_VD/garb/garbd $GALERA_BINS
+# cp -P $SCONS_VD/garb/garbd $GALERA_BINS
         cp -P $SCONS_VD/libgalera_smm.so $GALERA_LIBS
         if [ "$OS" == "Darwin" -a "$DEBUG" == "yes" ]; then
             cp -P -R $SCONS_VD/garb/garbd.dSYM $GALERA_BINS

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Maybe a duplicate of lp:1283100

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

Please retry with the latest HEAD from LP. Should fix that.

Revision history for this message
Alex Yurchenko (ayurchen) wrote :

The latest release should have fixed it. Please complain of not.

Changed in galera:
status: New → Fix Released
Changed in percona-xtradb-cluster:
status: New → Incomplete
Revision history for this message
Shahriyar Rzayev (rzayev-sehriyar) wrote :

Percona now uses JIRA for bug reports so this bug report is migrated to: https://jira.percona.com/browse/PXC-1627

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.