rabbitmq-server fails to start on a IPv6-Only environment/epmd is not IPv6 enabled

Bug #1312507 reported by Thiago Martins
32
This bug affects 6 people
Affects Status Importance Assigned to Milestone
erlang (Ubuntu)
Fix Released
High
James Page
Trusty
Fix Released
High
Unassigned
Utopic
Fix Released
High
James Page
rabbitmq-server (Ubuntu)
Invalid
Undecided
Unassigned
Trusty
Invalid
Undecided
Unassigned
Utopic
Invalid
Undecided
Unassigned

Bug Description

[Impact]
Users of RabbitMQ in IPv6 environments are unable to stop/start/restart rabbitmq.

[Test Case]
On and IPv6 only server:
sudo apt-get install rabbitmq-server
...
 * Starting message broker rabbitmq-server
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
   ...fail!

[Regression Potential]
The fix for this issue is in erlang itself; enabling IPv6 support in epmd is a compile time option so there is scope for regression. Testing should ensure that existing IPv4 deployments of erlang tools do not regress with this change (validated RMQ already against PPA packages).

[Original Bug Report]
Guys,

I'm trying to install rabbitmq-server on Ubuntu 14.04 on a IPv6-Only network and it fails, look:

---
root@server-1:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up rabbitmq-server (3.2.4-1) ...
 * Starting message broker rabbitmq-server
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
   ...fail!
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
dpkg: error processing package rabbitmq-server (--configure):
 subprocess installed post-installation script returned error exit status 1
Errors were encountered while processing:
 rabbitmq-server
E: Sub-process /usr/bin/dpkg returned an error code (1)
---

The log:
---
root@server-1:~# cat /var/log/rabbitmq/startup_log
ERROR: epmd error for host "server-1": address (cannot connect to host/port)
---

The connectivity (resolved via /etc/hosts):

---
root@server-1:~# ping6 server-1
PING server-1(server-1.domain.com.br) 56 data bytes
64 bytes from server-1.domain.com.br: icmp_seq=1 ttl=64 time=0.019 ms
64 bytes from server-1.domain.com.br: icmp_seq=2 ttl=64 time=0.044 ms
---

However, if I comment out the "nameserver 2001:4860:4860::8844" entries out from /etc/resolv.conf, then, rabbitmq-server starts! Very weird...

Nothing on /etc/resolv.conf now...:

---
root@server-1:~# apt-get -f install
Reading package lists... Done
Building dependency tree
Reading state information... Done
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Setting up rabbitmq-server (3.2.4-1) ...
 * Starting message broker rabbitmq-server
   ...done.
---

I found the following topics about this on the net:

* Related:

http://rabbitmq.1065348.n5.nabble.com/Error-starting-rabbitmq-in-IPv6-environment-td32001.html

* Not sure if it is related:

http://blog.domb.net/?p=290

http://hg.rabbitmq.com/rabbitmq-erlang-client/rev/7f53561c0c11

Best!
Thiago

Revision history for this message
Thiago Martins (martinx) wrote :

Guys,

Is that any workaround for this problem?

It is very annoying to manually edit the /etc/resolv.conf file just to be able to restart RabbirMQ... :-/

Tks in advance!

Thiago

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in rabbitmq-server (Ubuntu):
status: New → Confirmed
Revision history for this message
James Page (james-page) wrote :

I think the issue is that the epmd by default will only listen on IPv4 addresses; this appears to be a compile time option in erlang.

Revision history for this message
James Page (james-page) wrote :

Thiago

If you want to test I have a patched erlang in:

   ppa:james-page/rabbitmq

I think it forces the right behaviour in epmd:

tcp6 0 0 :::5672 :::* LISTEN
tcp6 0 0 :::4369 :::* LISTEN

Changed in rabbitmq-server (Ubuntu Utopic):
status: Confirmed → Invalid
Changed in rabbitmq-server (Ubuntu Trusty):
status: New → Invalid
Changed in erlang (Ubuntu Utopic):
status: New → In Progress
importance: Undecided → High
assignee: nobody → James Page (james-page)
Changed in erlang (Ubuntu Trusty):
status: New → Triaged
importance: Undecided → High
James Page (james-page)
summary: - rabbitmq-server fails to start on a IPv6-Only environment
+ rabbitmq-server fails to start on a IPv6-Only environment/epmd is not
+ IPv6 enabled
James Page (james-page)
description: updated
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package erlang - 1:17.1-dfsg-4ubuntu2

---------------
erlang (1:17.1-dfsg-4ubuntu2) utopic; urgency=medium

  * Enable IPv6 support in epmd (LP: #1312507), supporting use in IPv6
    only environments.
    - d/rules: Use "-DEPMD6" in CPPFLAGS and ensure this is passed to
      relevant bits of the build process.
 -- James Page <email address hidden> Tue, 12 Aug 2014 14:51:44 +0100

Changed in erlang (Ubuntu Utopic):
status: In Progress → Fix Released
Revision history for this message
Thiago Martins (martinx) wrote :

Awesome James! I'll test it...

Listen, is it possible to backport this fix to Trusty ?

Thank you!

Best,
Thiago

Revision history for this message
James Page (james-page) wrote :

Thiago

I've uploaded to the trusty-proposed queue for SRU team review.

Revision history for this message
Chris J Arges (arges) wrote : Please test proposed package

Hello Thiago, or anyone else affected,

Accepted erlang into trusty-proposed. The package will build now and be available at http://launchpad.net/ubuntu/+source/erlang/1:16.b.3-dfsg-1ubuntu2.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in erlang (Ubuntu Trusty):
status: Triaged → Fix Committed
tags: added: verification-needed
Revision history for this message
Thiago Martins (martinx) wrote :

Hey guys! Awesome job!

Problem fixed, packages: "erlang-* 1:16.b.3-dfsg-1ubuntu2.1".

No more IPv4 at my OpenStack Controller Node, YAY! :-)

Have a nice weekend!

Thanks!

tags: added: verification-done
removed: verification-needed
Revision history for this message
Xiang Hui (xianghui) wrote :

Verified, works for me, thanks.

Revision history for this message
Chris J Arges (arges) wrote : Update Released

The verification of the Stable Release Update for erlang has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package erlang - 1:16.b.3-dfsg-1ubuntu2.1

---------------
erlang (1:16.b.3-dfsg-1ubuntu2.1) trusty; urgency=medium

  * Enable IPv6 support in epmd (LP: #1312507), supporting use in IPv6
    only environments:
    - d/rules: Use "-DEPMD6" in CPPFLAGS and ensure this is passed to
      relevant bits of the build process.
 -- James Page <email address hidden> Tue, 12 Aug 2014 17:29:24 +0100

Changed in erlang (Ubuntu Trusty):
status: Fix Committed → Fix Released
Revision history for this message
Thiago Martins (martinx) wrote :

For some reason, one of controllers (small isolated region that I have for tests in lab), it still doesn't work... I'll investigate it more.

Version "1:16.b.3-dfsg-1ubuntu2.1":

---
root@controller-1:~# dpkg -l | grep erlang-base
ii erlang-base 1:16.b.3-dfsg-1ubuntu2.1 amd64 Erlang/OTP virtual machine and base applications

root@controller-1:~# service rabbitmq-server restart
 * Restarting message broker rabbitmq-server
   ...fail!
---

Then, I removed the IPv6 "nameservers" entries out from /etc/resolv.conf...:

---
root@controller-1:~# service rabbitmq-server restart
 * Restarting message broker rabbitmq-server
   ...done.
---

It seems that there are still something wrong with it...

I'll post it here if I can figure it out.

Best,
Thiago

Revision history for this message
s10 (vlad-esten) wrote :

After upgrading erlang to 1:16.b.3-dfsg-1ubuntu2.1, rabbitmq-server have been broken. Now it can't starts with epdm environment variable like "export ERL_EPMD_ADDRESS=10.0.0.1" in /etc/rabbitmq/rabbitmq-env.conf

Revision history for this message
David Hedberg (david-hedberg-t) wrote :

I suspect that this fix also broke rabbitmq on one of my servers. For unimportant reasons this particular server had ipv6 disabled through the kernel command line ( GRUB_CMDLINE_LINUX="ipv6.disable=1" ). I ran strace on epmd and this is, I assume, the relevant section:

---
socket(PF_INET6, SOCK_STREAM, IPPROTO_IP) = -1 EAFNOSUPPORT (Address family not supported by protocol)
open("/etc/localtime", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=1892, ...}) = 0
fstat(3, {st_mode=S_IFREG|0644, st_size=1892, ...}) = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f9e056a0000
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\4\0\0\0\4\0\0\0\0"..., 4096) = 1892
lseek(3, -1217, SEEK_CUR) = 675
read(3, "TZif2\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\6\0\0\0\6\0\0\0\0"..., 4096) = 1217
close(3) = 0
munmap(0x7f9e056a0000, 4096) = 0
write(2, "epmd: Mon Sep 29 15:09:23 2014: "..., 103epmd: Mon Sep 29 15:09:23 2014: error opening stream socket: Address family not supported by protocol
) = 103
exit_group(1) = ?
+++ exited with 1 +++

Re-enabling ipv6 fixes the problem.

Revision history for this message
hexes (ivan-s-borisov) wrote :

cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=14.04
DISTRIB_CODENAME=trusty
DISTRIB_DESCRIPTION="Ubuntu 14.04.2 LTS"

uname -a
Linux hexes-book 3.13.0-46-generic #77-Ubuntu SMP Mon Mar 2 18:23:39 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

sudo dpkg --configure -a
Настраивается пакет rabbitmq-server (3.2.4-1) …
 * Starting message broker rabbitmq-server * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
                                                                         [fail]
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
dpkg: error processing package rabbitmq-server (--configure):
 подпроцесс установлен сценарий post-installation возвратил код ошибки 1
При обработке следующих пакетов произошли ошибки:
 rabbitmq-server

erl -sname foo
{error_logger,{{2015,3,3},{21,43,40}},"Protocol: ~tp: register/listen error: ~tp~n",["inet_tcp",econnrefused]}

cat /etc/hosts
127.0.0.1 hexes-book localhost

hostname
hexes-book

Revision history for this message
True-night (hlpimfalling) wrote :

I am having the same issue. With ipv6 disabled, epmd will not start...

Revision history for this message
Thiago Martins (martinx) wrote : Re: [Bug 1312507] Re: rabbitmq-server fails to start on a IPv6-Only environment/epmd is not IPv6 enabled

Sorry to ask but, why disable IPv6? That's odd, I'm disabling IPv4 these
days... ;-)

Let it enabled... For God's sake... It doesn't hurt... =P

On 14 April 2015 at 15:14, True-night <email address hidden> wrote:

> I am having the same issue. With ipv6 disabled, epmd will not start...
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1312507
>
> Title:
> rabbitmq-server fails to start on a IPv6-Only environment/epmd is not
> IPv6 enabled
>
> To manage notifications about this bug go to:
>
> https://bugs.launchpad.net/ubuntu/+source/erlang/+bug/1312507/+subscriptions
>

Revision history for this message
John Carter (johncarter2679) wrote :

If your Router server fails and you can't able to connect to the internet. And also you are facing the IP address problem and DNS follow Dlink Support and visit https://www.routertechnicalsupportnumbers.com/d-link-router-support/ for help. I was also following that to solve the problems of router server fails and also DNS error.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.