Merge lp:~bladernr/checkbox-legacy/fix-network-test-output-fail into lp:checkbox-legacy

[Jeff Lane ]

We're still doing 12.04 certs for server, and this just makes the output on fail a little better as the current output captured by checkbox is useless for debugging the test results.

[Daniel Manrique (roadmr)]

+1, simple enough, I could think of a way to avoid code duplication (from the print statements above) but that will actually be more code than this.

review: Approve

[Daniel Manrique (roadmr)]

Actually, wait a second... which one should I approve, this or https://code.launchpad.net/~bladernr/checkbox/1229435-network-script-output-improvement/+merge/233380 ?

Let me know.

review: Needs Information

[Jeff Lane  (bladernr)]

Daniel, that depends... both? This started because of 12.04.5 testing
(which uses checkbox-certification-server/checkbox). I modified the
script in lp:checkbox and pushed that, then remembered that "checkbox"
checkbox is now build out of the base in checkbox-legacy.

So I modified checkbox-legacy thinking that would get the changes into
the PPA for precise packages, since we're using the following:

14.04+ = checkbox-ng/plainbox providers
12.04 = checkbox/checkbox-certification

I don't expect to really do many changes at all to checkbox-legacy,
but since we're still actively using it and I've seen a couple systems
with questionable results we couldn't debug due to lack of useful
output, here we are.

Out of curiosity, what is the process for updating legacy, in case I'm
doing it wrong? Update lp:checkbox and then cherry-pick into legacy?
or treat them as separate code bases?

Also, per your other comment, there are better ways, but since this is
old code, and the change is innocuous, I just did it the quick and
dirty way.

On Thu, Sep 4, 2014 at 11:44 AM, Daniel Manrique
<email address hidden> wrote:
> Review: Needs Information
>
> Actually, wait a second... which one should I approve, this or https://code.launchpad.net/~bladernr/checkbox/1229435-network-script-output-improvement/+merge/233380 ?
>
> Let me know.
> --
> https://code.launchpad.net/~bladernr/checkbox-legacy/fix-network-test-output-fail/+merge/233379
> You are the owner of lp:~bladernr/checkbox-legacy/fix-network-test-output-fail.

--
"Entropy isn't what it used to be."

Jeff Lane - Server Certification Team Lead, Tools Developer, Warrior
Poet, Lover of Pie
Phone: [phone]
Ubuntu Ham: W4KDH Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417 C466 4ABD 3635 3A14 B2DD

[Daniel Manrique (roadmr)]

Oh right, this is me being dumb and not paying attention to the target branches :/

Thanks!

review: Approve

[Zygmunt Krynicki (zyga)]

Jeff, quick question: why are you still using checkbox-legacy?

[Jeff Lane  (bladernr)]

Because it's the tool in place for 12.04, so mostly because I'm
squeamish about switching the entire testing system mid-stream from
the established one. It's one thing to make a drastic tooling change
between releases, but my preference is to stay consistent across an
LTS.

I could be convinced otherwise, but to be honest, 12.04 certs are so
few for us now that changing all the docs and re-training everyone to
use checkbox-ng wouldn't be worth it, IMO.

On Thu, Sep 4, 2014 at 12:44 PM, Zygmunt Krynicki
<email address hidden> wrote:
> Jeff, quick question: why are you still using checkbox-legacy?
> --
> https://code.launchpad.net/~bladernr/checkbox-legacy/fix-network-test-output-fail/+merge/233379
> You are the owner of lp:~bladernr/checkbox-legacy/fix-network-test-output-fail.

--
"Entropy isn't what it used to be."

Jeff Lane - Server Certification Team Lead, Tools Developer, Warrior
Poet, Lover of Pie
Phone: [phone]
Ubuntu Ham: W4KDH Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417 C466 4ABD 3635 3A14 B2DD

[Zygmunt Krynicki (zyga)]

Thanks for explaining that.

I wonder about training costs given that (I hope) you are also using
checkbox-ng in parallel with other releases. Am I missing something?

On Thu, Sep 4, 2014 at 7:00 PM, Jeff Lane <email address hidden> wrote:
> Because it's the tool in place for 12.04, so mostly because I'm
> squeamish about switching the entire testing system mid-stream from
> the established one. It's one thing to make a drastic tooling change
> between releases, but my preference is to stay consistent across an
> LTS.
>
> I could be convinced otherwise, but to be honest, 12.04 certs are so
> few for us now that changing all the docs and re-training everyone to
> use checkbox-ng wouldn't be worth it, IMO.
>
> On Thu, Sep 4, 2014 at 12:44 PM, Zygmunt Krynicki
> <email address hidden> wrote:
>> Jeff, quick question: why are you still using checkbox-legacy?
>> --
>> https://code.launchpad.net/~bladernr/checkbox-legacy/fix-network-test-output-fail/+merge/233379
>> You are the owner of lp:~bladernr/checkbox-legacy/fix-network-test-output-fail.
>
>
>
> --
> "Entropy isn't what it used to be."
>
> Jeff Lane - Server Certification Team Lead, Tools Developer, Warrior
> Poet, Lover of Pie
> Phone: [phone]
> Ubuntu Ham: W4KDH Freenode IRC: bladernr or bladernr_
> gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417 C466 4ABD 3635 3A14 B2DD
>
> https://code.launchpad.net/~bladernr/checkbox-legacy/fix-network-test-output-fail/+merge/233379
> You are subscribed to branch lp:checkbox-legacy.

[Jeff Lane  (bladernr)]

On Thu, Sep 4, 2014 at 1:06 PM, Zygmunt Krynicki
<email address hidden> wrote:
> Thanks for explaining that.
>
> I wonder about training costs given that (I hope) you are also using
> checkbox-ng in parallel with other releases. Am I missing something?

Well, the upside is that legacy checkbox is ONLY used for 12.04 certs,
of which there are very few, with Cisco and IBM being the only ones
showing interest in that now that 14.04 is out.

For 14.04 and everything beyond, we ONLY use checkbox-ng tooling.

The actual training costs are fairly low because at this point, the
testers have all been testing since pre-14.04, and the only real
training necessary was in using MAAS and in using checkbox-ng stuff.
The ONLY legacy training that I've had to do was for the IBM TSTL lab
in Taiwan because they were completely new to Ubuntu Training. There
MAY be others in the near future, but doubtful because I do not
believe we are pushing or encouraging 12.04 certs for anyone (we're
trying to avoid it, actually, except in cases where paying customers
are demanding it).

>
> On Thu, Sep 4, 2014 at 7:00 PM, Jeff Lane <email address hidden> wrote:
>> Because it's the tool in place for 12.04, so mostly because I'm
>> squeamish about switching the entire testing system mid-stream from
>> the established one. It's one thing to make a drastic tooling change
>> between releases, but my preference is to stay consistent across an
>> LTS.
>>
>> I could be convinced otherwise, but to be honest, 12.04 certs are so
>> few for us now that changing all the docs and re-training everyone to
>> use checkbox-ng wouldn't be worth it, IMO.
>>
>> On Thu, Sep 4, 2014 at 12:44 PM, Zygmunt Krynicki
>> <email address hidden> wrote:
>>> Jeff, quick question: why are you still using checkbox-legacy?
>>> --
>>> https://code.launchpad.net/~bladernr/checkbox-legacy/fix-network-test-output-fail/+merge/233379
>>> You are the owner of lp:~bladernr/checkbox-legacy/fix-network-test-output-fail.
>>
>>
>>
>> --
>> "Entropy isn't what it used to be."
>>
>> Jeff Lane - Server Certification Team Lead, Tools Developer, Warrior
>> Poet, Lover of Pie
>> Phone: [phone]
>> Ubuntu Ham: W4KDH Freenode IRC: bladernr or bladernr_
>> gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417 C466 4ABD 3635 3A14 B2DD
>>
>> https://code.launchpad.net/~bladernr/checkbox-legacy/fix-network-test-output-fail/+merge/233379
>> You are subscribed to branch lp:checkbox-legacy.
>
> --
> https://code.launchpad.net/~bladernr/checkbox-legacy/fix-network-test-output-fail/+merge/233379
> You are the owner of lp:~bladernr/checkbox-legacy/fix-network-test-output-fail.

--
"Entropy isn't what it used to be."

Jeff Lane - Server Certification Team Lead, Tools Developer, Warrior
Poet, Lover of Pie
Phone: [phone]
Ubuntu Ham: W4KDH Freenode IRC: bladernr or bladernr_
gpg: 1024D/3A14B2DD 8C88 B076 0DD7 B404 1417 C466 4ABD 3635 3A14 B2DD

[Daniel Manrique (roadmr)]

The attempt to merge lp:~bladernr/checkbox-legacy/fix-network-test-output-fail into lp:checkbox-legacy failed. Below is the output from the failed tests.

[precise] Bringing VM 'up'
[precise] Unable to 'up' VM!
[precise] stdout: http://paste.ubuntu.com/8459912/
[precise] stderr: http://paste.ubuntu.com/8459913/
[precise] NOTE: unable to execute tests, marked as failed
[precise] Destroying failed VM to reclaim resources
[precise] Forcing shutdown of VM...
[precise] Destroying VM and associated drives...
[precise] Running cleanup tasks for 'shell' provisioner...
[precise] Running cleanup tasks for 'shell' provisioner...
[trusty] Bringing VM 'up'
[trusty] Unable to 'up' VM!
[trusty] stdout: http://paste.ubuntu.com/8459950/
[trusty] stderr: http://paste.ubuntu.com/8459951/
[trusty] NOTE: unable to execute tests, marked as failed
[trusty] Destroying failed VM to reclaim resources
[trusty] Forcing shutdown of VM...
[trusty] Destroying VM and associated drives...
[trusty] Running cleanup tasks for 'shell' provisioner...
[trusty] Running cleanup tasks for 'shell' provisioner...

[Daniel Manrique (roadmr)]

OHH, since we're now doing tarmac tests using lxc, and checkbox-legacy wasn't updated for that, this doesn't work....

[Daniel Manrique (roadmr)]

OK, I fixed the tarmac tester, let's reapprove.

review: Approve

=== modified file 'checkbox-old/scripts/network'
--- checkbox-old/scripts/network	2014-03-14 17:36:52 +0000
+++ checkbox-old/scripts/network	2014-09-04 15:28:37 +0000
@@ -124,6 +124,9 @@
 
             if percent < self.fail_threshold:
                 logging.warn("Poor network performance detected")
+                logging.warn("  Transfer speed: {} {}b/s".format(
+                             throughput, units))
+                logging.warn("  %3.2f%% of " % percent, end="")
                 return 30
 
             logging.debug("Passed benchmark")