Ubuntu
glusterfs package

Remote glusterfs mounts fail during boot because network is not yet available

Bug #1268064 reported by Louis Zuckerman
18
This bug affects 3 people
Affects Status Importance Assigned to Milestone
glusterfs (Ubuntu)
Fix Released
Undecided
Louis Zuckerman

Bug Description

I'm the upstream package maintainer & I'm filing this bug along with a patch (merge request) for the upcoming trusty release.

Affects: Ubuntu trusty (14.04)

Problem: Upstart tries the glusterfs mounts before the network is available during boot, and they fail.

Solution: An upstart task to block glusterfs mounts during boot until the network is ready

Patch:

The glusterfs 3.4.1 package currently in trusty has outdated upstart jobs. The patch I will be submitting shortly makes two changes: move the mounting-glusterfs blocker job from glusterfs-server to glusterfs-client; change the wait-for-state condition from glusterfs-server to static-network-up.

Background:

When I wrote the old mount blocker, back in early 2012, remote mounts worked fine but localhost glusterfs mounts failed because the local glusterfs-server daemon had not started by the time the mounts were executed. At some point since then we found that remote mounts would also fail during boot, perhaps by better testing or maybe a change in ubuntu since precise, I'm not sure. The new mount blocker job resolves both problems as far as I know. I provide the official glusterfs packages for ubuntu in PPAs (https://launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.4 for example) and many people use those packages and have tested the new mount blocker.

Log file showing the mount fail with the current 3.4.1 package...

[2014-01-11 03:27:38.480966] I [glusterfsd.c:1910:main] 0-/usr/sbin/glusterfs: Started running /usr/sbin/glusterfs version 3.4.1 (/usr/sbin/glusterfs --volfile-id=foo --volfile-server=192.168.192.104 /mnt/foo)
[2014-01-11 03:27:38.489565] I [socket.c:3480:socket_init] 0-glusterfs: SSL support is NOT enabled
[2014-01-11 03:27:38.489610] I [socket.c:3495:socket_init] 0-glusterfs: using system polling thread
[2014-01-11 03:27:38.489691] E [common-utils.c:211:gf_resolve_ip6] 0-resolver: getaddrinfo failed (Name or service not known)
[2014-01-11 03:27:38.489705] E [name.c:249:af_inet_client_get_remote_sockaddr] 0-glusterfs: DNS resolution failed on host 192.168.192.104
[2014-01-11 03:27:38.489747] E [glusterfsd-mgmt.c:1870:mgmt_rpc_notify] 0-glusterfsd-mgmt: failed to connect with remote-host: Success
[2014-01-11 03:27:38.489759] I [glusterfsd-mgmt.c:1873:mgmt_rpc_notify] 0-glusterfsd-mgmt: -1 connect attempts left
[2014-01-11 03:27:38.490749] W [glusterfsd.c:1002:cleanup_and_exit] (-->/usr/sbin/glusterfs(glusterfs_mgmt_init+0x15d) [0x7f7127d2d89d] (-->/usr/lib/x86_64-linux-gnu/libgfrpc.so.0(rpc_clnt_start+0x12) [0x7f7127675342] (-->/usr/sbin/glusterfs(+0xd1fa) [0x7f7127d2e1fa]))) 0-: received signum (1), shutting down
[2014-01-11 03:27:38.490776] I [fuse-bridge.c:5260:fini] 0-fuse: Unmounting '/mnt/foo'.

Related branches

lp:~semiosis/ubuntu/trusty/glusterfs/fix-for-1268064
James Page: Pending requested
Diff: 17170 lines (+7414/-7666)
93 files modified
.pc/.quilt_patches (+0/-1)
.pc/.quilt_series (+0/-1)
.pc/.version (+0/-1)
.pc/01-spelling-error.diff/rpc/rpc-transport/rdma/src/rdma.c (+0/-4578)
.pc/01-spelling-error.diff/xlators/mgmt/glusterd/src/glusterd-store.c (+0/-2530)
.pc/applied-patches (+0/-1)
ChangeLog (+1978/-4)
api/Makefile.am (+1/-1)
api/Makefile.in (+1/-1)
api/examples/Makefile.am (+6/-0)
api/examples/Makefile.in (+555/-0)
api/examples/README (+19/-0)
api/examples/gfapi.py (+402/-0)
api/examples/glfsxmp.c (+1601/-0)
api/examples/setup.py.in (+29/-0)
api/src/Makefile.am (+3/-2)
api/src/Makefile.in (+15/-3)
api/src/glfs-fops.c (+91/-32)
api/src/glfs-handleops.c (+1278/-0)
api/src/glfs-handles.h (+143/-0)
api/src/glfs-internal.h (+59/-0)
api/src/glfs-mem-types.h (+2/-1)
api/src/glfs-resolve.c (+70/-3)
api/src/glfs.c (+22/-4)
api/src/glfs.h (+26/-0)
cli/src/cli-cmd-parser.c (+36/-4)
cli/src/cli-cmd-volume.c (+68/-3)
cli/src/cli-rpc-ops.c (+22/-14)
cli/src/cli-xml-output.c (+81/-4)
configure (+13/-11)
configure.ac (+3/-1)
debian/changelog (+16/-0)
debian/glusterfs-client.mounting-glusterfs.upstart (+9/-0)
debian/glusterfs-server.mounting-glusterfs.upstart (+0/-7)
debian/rules (+3/-2)
debian/source.lintian-overrides (+1/-0)
doc/glusterd.vol (+1/-0)
glusterfs.spec (+41/-16)
glusterfs.spec.in (+33/-8)
glusterfsd/src/glusterfsd-mgmt.c (+5/-0)
libglusterfs/src/globals.c (+56/-1)
libglusterfs/src/globals.h (+4/-0)
libglusterfs/src/mem-types.h (+2/-1)
libglusterfs/src/syncop.c (+154/-0)
libglusterfs/src/syncop.h (+63/-19)
rpc/rpc-transport/rdma/src/rdma.c (+4/-4)
xlators/cluster/afr/src/afr-inode-read.c (+1/-1)
xlators/cluster/afr/src/afr.c (+10/-0)
xlators/cluster/dht/src/dht-common.c (+16/-6)
xlators/cluster/dht/src/dht-common.h (+1/-0)
xlators/cluster/dht/src/dht-inode-read.c (+4/-0)
xlators/cluster/dht/src/dht-inode-write.c (+2/-0)
xlators/cluster/dht/src/dht-layout.c (+11/-3)
xlators/cluster/dht/src/dht-rebalance.c (+29/-14)
xlators/cluster/dht/src/dht-selfheal.c (+7/-0)
xlators/mgmt/glusterd/src/glusterd-brick-ops.c (+9/-3)
xlators/mgmt/glusterd/src/glusterd-op-sm.c (+71/-4)
xlators/mgmt/glusterd/src/glusterd-pmap.c (+5/-5)
xlators/mgmt/glusterd/src/glusterd-replace-brick.c (+8/-3)
xlators/mgmt/glusterd/src/glusterd-store.c (+3/-4)
xlators/mgmt/glusterd/src/glusterd-utils.c (+148/-14)
xlators/mgmt/glusterd/src/glusterd-utils.h (+7/-2)
xlators/mgmt/glusterd/src/glusterd-volume-ops.c (+12/-4)
xlators/mgmt/glusterd/src/glusterd.c (+10/-2)
xlators/mgmt/glusterd/src/glusterd.h (+1/-0)
xlators/mount/fuse/src/fuse-bridge.h (+14/-7)
xlators/nfs/server/src/mount3.c (+5/-14)
xlators/nfs/server/src/mount3.h (+4/-13)
xlators/nfs/server/src/mount3udp_svc.c (+4/-13)
xlators/nfs/server/src/nfs-common.c (+4/-13)
xlators/nfs/server/src/nfs-common.h (+4/-13)
xlators/nfs/server/src/nfs-fops.c (+4/-13)
xlators/nfs/server/src/nfs-fops.h (+4/-13)
xlators/nfs/server/src/nfs-generics.c (+4/-13)
xlators/nfs/server/src/nfs-generics.h (+4/-13)
xlators/nfs/server/src/nfs-inodes.c (+4/-13)
xlators/nfs/server/src/nfs-inodes.h (+4/-13)
xlators/nfs/server/src/nfs-mem-types.h (+4/-13)
xlators/nfs/server/src/nfs.c (+4/-13)
xlators/nfs/server/src/nfs.h (+4/-13)
xlators/nfs/server/src/nfs3-fh.c (+4/-13)
xlators/nfs/server/src/nfs3-fh.h (+4/-13)
xlators/nfs/server/src/nfs3-helpers.c (+4/-13)
xlators/nfs/server/src/nfs3-helpers.h (+4/-13)
xlators/nfs/server/src/nfs3.c (+4/-13)
xlators/nfs/server/src/nfs3.h (+4/-13)
xlators/nfs/server/src/nlm4.c (+4/-13)
xlators/nfs/server/src/nlm4.h (+4/-13)
xlators/nfs/server/src/nlmcbk_svc.c (+4/-13)
xlators/protocol/auth/addr/src/addr.c (+4/-13)
xlators/protocol/auth/login/src/login.c (+4/-13)
xlators/protocol/client/src/client-handshake.c (+1/-0)
xlators/storage/posix/src/posix.c (+27/-10)
lp:ubuntu/trusty-proposed/glusterfs
Louis Zuckerman (semiosis)
Changed in glusterfs (Ubuntu):
assignee: nobody → semiosis (semiosis)
status: New → In Progress
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package glusterfs - 3.4.2-1ubuntu1

---------------
glusterfs (3.4.2-1ubuntu1) trusty; urgency=low

  * Merge from Debian unstable. Remaining changes:
    - Updated mount blocking upstart job to wait until network is up for
      all glusterfs mounts (LP: #1268064)

glusterfs (3.4.2-1) unstable; urgency=low

  * New upstream release.
  * Overwrite debian-watch-may-check-gpg-signature warning. There is no GPG
    signature available.
 -- Louis Zuckerman <email address hidden> Fri, 10 Jan 2014 21:40:13 -0500

Changed in glusterfs (Ubuntu):
status: In Progress → Fix Released
Revision history for this message
Bruce Weirdan (weirdan) wrote :

I'm having this problem on saucy. Will this fix be backported to 13.10?

Revision history for this message
Louis Zuckerman (semiosis) wrote :

Bruce,

Saucy has a very old version of GlusterFS. You should look into upgrading to 3.4.3, or if you're adventurous, 3.5.0 which was just released this week.

You should use the GlusterFS PPAs, which have packages for Saucy...

launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.4
launchpad.net/~semiosis/+archive/ubuntu-glusterfs-3.5

Please join #gluster on Freenode IRC for further support.

Thanks!

Revision history for this message
Bruce Weirdan (weirdan) wrote :

Thanks, I'll try PPA's tomorrow.

Revision history for this message
Jason Woods (devel-k) wrote :

Hi Louis,

I've noticed this problem with the 3.6 and 3.7 PPA packages at:
https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.6
https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.7

Do you know if there are plans to include this change in those too?

Thanks!

Jason

Revision history for this message
Kaleb S. KEITHLEY (kkeithleatredhat) wrote :

> I've noticed this problem with the 3.6 and 3.7 PPA packages at:
> https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.6
> https://launchpad.net/~gluster/+archive/ubuntu/glusterfs-3.7
>
> Do you know if there are plans to include this change in those too?

I'm not an expert at Debian packages, but looking at the packaging bits in git that were used to create the 3.6 .debs -- both in semiosis' old git repo and the new one at github.com:gluster/glusterfs-debian.git -- it looks to me like the upstart bits are in the 3.6 .debs but not in the 3.7 .debs. Here's an excerpt of the vivid 3.6 rules file, same as 3.5:

...
override_dh_installinit:
        dh_installinit -pglusterfs-client --upstart-only --no-start --name=mounting-glusterfs
        dh_installinit -pglusterfs-server --upstart-only
...

I'll add them to the 3.7 builds. How they were not included is certainly a mystery to me.

Danny Hammo (dan-hammo)
Changed in glusterfs (Ubuntu):
milestone: none → trusty-updates
milestone: trusty-updates → none
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.
Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.