SRU runs with plainbox only show 5 instead of ~50 devices

I tried to submit my own system using plainbox (trunk version) + plainbox transport.

The device count is normal:
https://certification.canonical.com/hardware/201202-10584/submission/91490/

But a sunmission from the same day still show 5 devices only:
https://certification.canonical.com/hardware/201101-6965/submission/91492/

I used to following cmd line:
plainbox run -i package -i uname -i lsb -i cpuinfo -i dpkg -i dmi_attachment -i sysfs_attachment -i udev_attachment -i usb/detect -f xml -o /tmp/sub.xml -t certification --transport-options secure_id=a00D000000LVGLt --transport-where https://certification.canonical.com/submissions/submit/

According to https://certification.canonical.com/hardware/201202-10584/submission/91497/devices/

it seems that the 5 devices are extracted from the dmi attachement, not the udev/device attachement hence the fixed count of devices.

I've been able to reproduce it using the xml saved on this system (https://certification.canonical.com/hardware/201202-10584/).
The submission I got is here (posted to my test system):
https://certification.canonical.com/hardware/201202-10584/submission/91497/

Awesome! OK, so I downloaded the hexr source and ran the submission parser script. This basically uses the checkbox submission parser module, parses the xml and produces a json file with a representation of all the stuff like devices, packages and so on.

- bzr branch lp:hexr
- Apply the below patch
- Ensure you have a checkbox source tree somewhere
- Run the parser like this:
PYTHONPATH=$PATH_TO_YOUR_CHECKBOX_TRUNK python apps/uploads/checkbox_parser.py submission.xml -v >/tmp/output.json

Looking at both the output.json and the log.log file produced by the parser, what's happening is that the plainbox submission results in two "devices" dictionaries existing. One is taken from the dmi devices (these are the first 5 devices that plainbox submissions have). The other one contains most devices and is taken from the udev element.

On the other hand, the checkbox-produced submission results in just one "devices" dictionary.

This is helpful in reproducing the problem locally and getting closer to the root cause of the problem. Next I'll look at the ordering of attachments in plainbox vs. checkbox submissions and see if we can do a workaround there. The real fix for this would be having the parser *not* create two "devices" elements, but updating the parser in c4 is slightly more complex.

=== modified file 'apps/uploads/checkbox_parser.py'
--- apps/uploads/checkbox_parser.py 2012-11-21 16:58:07 +0000
+++ apps/uploads/checkbox_parser.py 2013-05-23 19:07:54 +0000
@@ -131,11 +131,13 @@
     parser.add_argument("-v", "--verbose",
         action="store_true",
         help="Turn-on verbosity to get more info on what's going on.")
+ parser.add_argument("-l", "--logfile", default="log.log",
+ help="Log file")
     args = parser.parse_args()

     # Set up logger to console
     format = '%(asctime)s %(levelname)-8s %(message)s'
- console_handler = logger.StreamHandler()
+ console_handler = logging.StreamHandler()
     console_handler.setFormatter(logging.Formatter(format))

     # Set up the overall logger
@@ -145,7 +147,7 @@
     root_logger.setLevel(logging.DEBUG)

     if args.verbose:
- verbose_handler = logging.FileHandler(logfile)
+ verbose_handler = logging.FileHandler(args.logfile)
         verbose_handler.setLevel(logging.DEBUG)
         verbose_handler.setFormatter(logging.Formatter(format))
         root_logger.addHandler(verbose_handler)

tl;dr: A workaround to this problem is updating plainbox so the xml "summary" block has the "architecture" element before the "distroseries" element. The story is long but hopefully interesting, see below....

Yay! So I found a difference between the plainbox and checkbox XML files which is triggering this problem.

It will be worth continuing to look at the parser until we understand this problem; I'm uneasy about just magically changing something and then having things work without understanding at least why the parser behaves like this.

In the summary section (at the very end of the xml file), checkbox produces this summary block:

<summary>
<architecture value="amd64"/>
<client name="checkbox-qt" version="0.13.7"/>
<contactable value="False"/>
<date_created value="2013-05-08T22:27:35"/>
<distribution value="Ubuntu"/>
<distroseries value="12.04"/>
<kernel-release value="3.2.0-41-generic"/>
<live_cd value="False"/>
<private value="False"/>
<system_id value="669b662da410063cc918e0f60cf6cddf"/>
</summary>

Whereas plainbox has the same fields but in different order:

  <summary>
    <client name="plainbox" version="0.4.0"/>
    <date_created value="2013-05-23T07:19:04"/>
    <distribution value="Ubuntu"/>
    <distroseries value="12.04"/>
    <architecture value="amd64"/>
    <kernel-release value="3.5.0-23-generic"/>
    <private value="False"/>
    <contactable value="False"/>
    <live_cd value="False"/>
    <system_id value=""/>
  </summary>

As it turns out, if the architecture field comes *after* distroseries, the parser starts acting strange. For instance, I was able to reproduce the weird "dmi devices first, then a block of packages that breaks things, then a second block of udev devices" behavior, using a checkbox submission, and simply moving the architecture field to be after distroseries.

In my investigations of the parser I found that the separate blocks are there "by design": if the parser starts finding devices, it will only add them to the existing "devices" block if the last block (or message, in checkbox parlance) is another "devices" message. So what happens is:

- Parser starts finding devices from the dmi attachment, creates a devices message, and appends the devices to this one.
- Parser starts processing packages (WHY? this is still a mystery, triggered by the architecture/distroseries problem in the summary).
- Parser finds another batch of devices from the udev attachment, but since the packages "got in the way", it creates a new devices message and puts the devices there.

Code for this is in hexr, in apps/uploads/checkbox_parser.py, in TestRun.addDeviceState (at the beginning of the method).

A quick solution for this would be changing plainbox's xml exporter so that it produces the summary fields in the same order as checkbox (or at least, ensuring architecture comes before distroseries, in my experiments that seems to be enough).

However it may be worth taking a closer look at this on the parser and c4 side. For instance, the checkbox/lib/parsers/submission.xml clearly says:

# Iterate over the root children, "summary" first

So even if the summary is at the end, that's the first thing that's parsed, and I ...

Awesome work Daniel.

That parser seems like a can of rotten worms, let's have a look at it tomorrow to see what we can do.

Excellent, I was ordering the context and hardware part, the next step was ordering the summary. You did it!
Thanks for confirming it works.

We'll see tomorrow if we want to open this box.