TestRabbitMessaging breaks in lander

Bug #1225980 reported by Jeroen T. Vermeulen
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
MAAS
Fix Released
Critical
Gavin Panella
rabbitfixture
Fix Released
Critical
Gavin Panella

Bug Description

We hit this strange test failure in Jenkins:

http://10.189.74.2:8080/view/MAAS/job/maas-merger-trunk/label=lp-saucy-server-amd64/87/console

Disabling the test for the time being.

Tags: tests

Related branches

Revision history for this message
Gavin Panella (allenap) wrote :

RabbitMQ seems to be saying it's not able to start because the address is already in use:

  ... {cannot_listen,{0,0,0,0,0,0,0,0},48258,eaddrinuse}

This might be an avenue for reproducing the bug.

Gavin Panella (allenap)
Changed in maas:
assignee: nobody → Gavin Panella (allenap)
Gavin Panella (allenap)
Changed in rabbitfixture:
importance: Undecided → Critical
status: New → Triaged
assignee: nobody → Gavin Panella (allenap)
Revision history for this message
Gavin Panella (allenap) wrote :

The problem is tangled, afaict. In rabbitfixture:

- RabbitMQ is started on all addresses; the hostname attribute on
  RabbitServerResources doesn't actually seem to have an effect.

- RabbitServerResources.fq_hostname always uses socket.gethostname()
  instead of the hostname attribute. I think this was required because
  the server was starting up on all addresses rather than one.

- allocate_ports() only binds to localhost, not the address on which
  the RabbitMQ server is going to be brought up on. There's nothing
  preventing the port from being in use on any other address.

I think the fix will look like:

- Only start RabbitMQ on the specified address. The environment
  variable RABBITMQ_NODE_IP_ADDRESS can be used to do this.

- Change fq_hostname to use the configured hostname.

- Change allocate_ports() to take an address parameter, and pass in
  the configured hostname when using it.

Gavin Panella (allenap)
Changed in rabbitfixture:
status: Triaged → Fix Released
Gavin Panella (allenap)
Changed in maas:
status: Triaged → In Progress
Revision history for this message
Gavin Panella (allenap) wrote :

I've just tested lp:~allenap/maas/rabbitmq-fixture-robustness on Julian's landing host, and it appears to fix the issue.

Gavin Panella (allenap)
Changed in maas:
status: In Progress → Fix Committed
Revision history for this message
Gavin Panella (allenap) wrote :

Released; it's a change that affects the test suite only.

Changed in maas:
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.