rax-nova-agent does not drain the event queue, before starting

Bug #1765057 reported by Dimitri John Ledkov
6
This bug affects 1 person
Affects Status Importance Assigned to Milestone
rax-nova-agent (Ubuntu)
Fix Released
Undecided
Unassigned

Bug Description

A single iteration of event processing is not sufficient to drain all of the events on firstboot provisioning in Xen based clouds.

Specifically, instead of receiving a list of 7 events, one might only receive 1 event at a time, spaced 1 or 2 seconds apart, until all initial events are processed.

To work around the fact that supervisor is not batching all events together, do the following:

* process events until there are none
* ensure that there were at least three iterations without any events
* then notify that daemon as ready, as no more early boot events are expected
* continue polling any interactive triggered events going forward

Revision history for this message
Dimitri John Ledkov (xnox) wrote :
Download full text (3.3 KiB)

Tested on a self-created image:

2018-04-18 14:43:48,129 [INFO ] Skipping os.fork as directed by arguments
2018-04-18 14:43:48,129 [INFO ] Starting actions for novaagent.libs.debian
2018-04-18 14:43:48,130 [INFO ] Checking for existence of /dev/xen/xenbus
2018-04-18 14:43:48,131 [INFO ] Provider: Rackspace
2018-04-18 14:43:48,144 [INFO ] Event: 77bc84f1-1810-4089-946e-668ccb3d1e86 -> version
2018-04-18 14:43:48,145 [INFO ] Returning {"message": "2.1.13", "returncode": "0"}
2018-04-18 14:43:49,148 [INFO ] Event: 5fd73e82-fe51-4463-be63-e73928cf05be -> features
2018-04-18 14:43:49,149 [INFO ] Returning {"message": "kmsactivate,resetnetwork,version,keyinit,features,password,injectfile", "returncode": "0"}
2018-04-18 14:43:50,151 [INFO ] Event: b0f171aa-c1ab-416d-a51c-0a72a18ee4ea -> injectfile
2018-04-18 14:43:50,153 [INFO ] Returning {"message": "", "returncode": "0"}
2018-04-18 14:43:51,155 [INFO ] Event: ce18fecb-128d-4ed3-b88f-79ea05bb2c50 -> keyinit
2018-04-18 14:43:51,156 [INFO ] Returning {"message": "63332560236465421491477949819793", "returncode": "D0"}
2018-04-18 14:43:52,160 [INFO ] Event: 8a591e8d-b24f-48a8-9bc0-e75c32a947b8 -> password
2018-04-18 14:43:52,201 [INFO ] Returning {"message": "", "returncode": "0"}
2018-04-18 14:43:53,204 [INFO ] Event: d9edad5f-737d-482f-8744-c48384a1e89d -> resetnetwork
2018-04-18 14:43:53,205 [INFO ] Hostname: xnox-test-counter
2018-04-18 14:43:53,205 [INFO ] Backing up -> /etc/hostname (1524062633.2054577.bak)
2018-04-18 14:43:53,212 [ERROR] Error using hostnamectl: b'Failed to create bus connection: No such file or directory\n'
2018-04-18 14:43:53,218 [INFO ] Interface BC764E20680D: {'label': 'private', 'broadcast': '10.209.95.255', 'ips': [{'ip': '10.209.64.42', 'netmask': '255.255.224.0', 'enabled': '1', 'gateway': None}], 'mac': 'BC:76:4E:20:68:0D', 'routes': [{'route': '10.208.0.0', 'netmask': '255.240.0.0', 'gateway': '10.209.64.1'}, {'route': '10.176.0.0', 'netmask': '255.240.0.0', 'gateway': '10.209.64.1'}], 'gateway': None}
2018-04-18 14:43:53,219 [INFO ] Interface BC764E206804: {'ip6s': [{'ip': '2001:4802:7805:101:be76:4eff:fe20:6804', 'netmask': 64, 'enabled': '1', 'gateway': 'fe80::def'}], 'label': 'public', 'broadcast': '104.239.175.255', 'ips': [{'ip': '104.239.175.86', 'netmask': '255.255.255.0', 'enabled': '1', 'gateway': '104.239.175.1'}], 'mac': 'BC:76:4E:20:68:04', 'gateway_v6': 'fe80::def', 'dns': ['69.20.0.164', '69.20.0.196'], 'gateway': '104.239.175.1'}
2018-04-18 14:43:53,444 [INFO ] Returning {"message": "", "returncode": "0"}
2018-04-18 14:43:54,446 [INFO ] No event received
2018-04-18 14:43:55,448 [INFO ] No event received
2018-04-18 14:43:56,450 [INFO ] No event received

and

Apr 18 14:43:47 ubuntu systemd[1]: Starting Nova Agent for xenstore...
Apr 18 14:43:52 ubuntu passwd[729]: pam_unix(passwd:chauthtok): password changed for root
Apr 18 14:43:57 xnox-test-counter systemd[1]: Started Nova Agent for xenstore.
Apr 18 14:43:57 xnox-test-counter systemd[1]: Starting Initial cloud-init job (pre-networking)...
Apr 18 14:43:58 xnox-test-counter cloud-init[763]: Cloud-init v. 18.2 running 'init-local' at Wed, 18 Apr 2018 14:43:58 +0000. Up 16.60 seconds.
Apr 18 14:43:58 xnox-...

Read more...

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package rax-nova-agent - 2.1.13-0ubuntu3

---------------
rax-nova-agent (2.1.13-0ubuntu3) bionic; urgency=medium

  * Drain the events queue, until there are no more events, for at least 3
    iterations, before sending started notification. LP: #1765057

 -- Dimitri John Ledkov <email address hidden> Wed, 18 Apr 2018 14:59:45 +0100

Changed in rax-nova-agent (Ubuntu):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.