Tickets take 4.5 hours to successfully complete

Marking this as medium for now as vila says his tickets take 20 minutes to complete. I'll bump this up to high/critical if we cannot get confirmation from a few other people.

https://launchpad.net/ubuntu/saucy/+source/python-scipy/+builds

Between 41 mins and 1h50 to build depending on arch.

> It looks like it took 94.5846 minutes to complete the tests:

http://d-jenkins.ubuntu-ci:8080/job/saucy-adt-python-scipy/buildTimeTrend

http://d-jenkins.ubuntu-ci:8080/job/saucy-adt-python-scipy/52/testReport

/http://d-jenkins.ubuntu-ci:8080/job/saucy-adt-python-scipy/52/ARCH=amd64,label=adt/

22 minutes.

The hardwares are different and I suspect the processor itself is more relevant than the number of cores (the test runner testbed runs one dep-8 test after the other in both cases).

Let's resume the comparison once we have more logs out of the testbed.

Note that having timestamps in http://d-jenkins.ubuntu-ci:8080/job/saucy-adt-python-scipy/52/ARCH=amd64,label=adt/artifact/results/log will go a long way to make comparisons easier.

One of my tickets for pkg 'adduser' took 1 hour and 8 mins.
                                  bsb- 15 mins,
                                  imagebuilder-47 mins
                                  testrunner-5 mins
The imagebuild-worker which has 1G RAM had taken 21 mins for apt-get clean in the chroot env. http://pastebin.ubuntu.com/7073343/ .

Another ticket of the same package finished in 33 mins with imagebuild-worker taking just 10 mins.

Increasing the RAM size could possibly improve this timing? Still investigating with some more runs altering configs.

We have a RAM issue for the bsbuilder also. Should we just default all our services to use more memory?

Not every service needs it. Apache will run comfortably on a small instance, for example. Our deployment is already pretty big and resource intensive with the number of instances we have, so we should be careful about making it larger without having clear reason to. Equally, if a service needs more memory, we should find out why - if there's a memleak, using a m1.medium will only save us for so long.

So yeah, pick a larger instance size where you need one.

The kernel tests I've been going through take hours, but almost all of that is caught up in the ppa build. I seem to get a builder assigned pretty quickly so I don't think bumping the score would help much in this case. We do have a ram issue on the bsb-worker, but only in some cases (kernel so far). I've bumped it to 4G locally and I'll include that in my patch later today which will hopefully fix some other things too, as soon as I can make it all the way through the build/image and at least see it trying to run the test. I suspect for my case, we'll want to bump up to more cpu/mem on the test runner also though. NB: this would not be the tr-worker, but rather, the instance it starts to actually run the tests on the image we create.

NB: this would not be the tr-worker, but rather, the instance it starts to actually run the tests on the image we create.

/me nods

Until bug #1288617 is fixed, bug #1287955 provided a way to address that by changing the instance flavor in ./juju-deployer/configs/unit_config.yaml.tmpl

Still working on this, but from what I see, bsb, imagebuilder and test runner are the time consuming stages and we've already increased the bsb memory to 4G.

Along with increasing the memory/ cores/ scores we also need to see if we should impose timeouts for each stage. For now at least, it is impossible to see all possible reasons for a stage to wait/hang forever. As of rev365 I don't see any timeouts at least on ppa-assigner stage.

Increasing polling time from 60s-> 15s won't save us a lot in my view especially when we have stages taking time in minutes.

I am running consecutive tickets for the same package to compare the timing variations with the default configs of the deployment, not sure how many tickets we need to run to really spot a memory leak, but will keep running and later with some increased config values.

> Increasing polling time from 60s-> 15s won't save us a lot in my view
> especially when we have stages taking time in minutes.

If in the optimal case a ticket is taking us a few minutes, shaving 45 seconds off here and there would make a noticeable difference.

Ev,

Do you have a package that runs through the system in a few minutes, and how many minutes is few minutes? :)

The least I saw is 18 minutes in my runs.

https://docs.google.com/a/canonical.com/spreadsheet/ccc?key=0AvOb-Wd__AD8dEhkeGRWUzBnSmZ5SWhOT0RlQ0tyWmc&usp=drive_web#gid=0

Contains the list of timings in about 15 successful runs and here is a summary:
+ 's
1. testrunner takes almost half the time when I used standard.medium instead of standard.small (6-9 mins -> 3 mins)
2. Imagebuilder completes from 8-9 mins -> 3 mins when 4G RAM is used

-'s
1. bsb does not conclusively gain anything by increasing either RAM nor cpu cores
2. ppa-assigner stays waiting for ever because launchad taking too long to clean the 'dirty' ppa's. I've had to force make the ppa's available to continue on this.

I am going to do an MP for imagebuilder and test runner as a first step.

Attached a branch with improved capabilities to imaegbuilder and testbed instances. Looking into polling time improvements now

psivaa has merged some code that should help this out quite a bit. Lets keep an eye on ticket times over the next few days before we say this is "fixed"

The ticket for python-scipy took 2hr 2min for me to complete with failed tests: http://paste.ubuntu.com/7086926/

Testrunner took 1 hr 4 min on a "standard.2xlarge" (30GB RAM | 8 VCPU | 30.0GB Disk) testbed instance and logging into the testbed during the test showed the tests themselves consuming 99+ % CPU.

On alderamin as Vincent points out this takes less time. This is definitely package specific and making a single selection for all the packages is overkill.

Leaving the testbed specification to be configurable by the user, probably via the cli appears to be the best solution for me, may be post phase-0.

Another data point I discovered. The image-builder is super slow for me tonight:

23:59:30 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'Adding requested packages to the image...'}
00:30:51 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'updating chroot to be more compressible...'}

So it took 30 minutes to run "apt-get install phablet-tools". I logged in while it was running and "apt-get install" seemed to be waiting on 2 instances of /usr/lib/apt/methods/http

This seems to not have anything todo with capacity but with network i/o or something. What's really odd, is I ran that command by hand and it worked in <60seconds. Not sure what this means.

@Andy: Only difference that I can think of why it took significantly different times is that they could have probably used different archive locations: http://az3.clouds.archive.ubuntu.com vs http://archive.ubuntu.com/ubuntu/ and the latter might have been too slow at the time. The apt-get update within the chroot is hitting the latter but on the instance it's hitting the az3.cloud one.

Also there is a zerofile creation/removal step that's used to speed up the compression in the image-builder and this used to make the process slow but after increasing the RAM I have not seen it. From the logs it's not clear whether this was the case.

Just to double check to see if there is anything in phablet-tools I ran a ticket to see if it has anything specific in it for the delay. That only took 3 mins for the apt-get install with rev388 of the trunk.
17:59:23 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'Adding requested packages to the image...'}
18:02:06 INFO PROGRESS_TRIGGER: STATUS: {u'message': u'Shrinking the converted image...'}

Will run it a couple of more times to see if it happens by chance.

On 03/14/2014 02:22 PM, Parameswaran Sivatharman wrote:
> @Andy: Only difference that I can think of why it took significantly
> different times is that they could have probably used different archive
> locations: http://az3.clouds.archive.ubuntu.com vs
> http://archive.ubuntu.com/ubuntu/ and the latter might have been too
> slow at the time. The apt-get update within the chroot is hitting the
> latter but on the instance it's hitting the az3.cloud one.

Its not happening anymore, so I think it was just temporary LP insanity

> Also there is a zerofile creation/removal step that's used to speed up
> the compression in the image-builder and this used to make the process
> slow but after increasing the RAM I have not seen it. From the logs
> it's not clear whether this was the case.

This was purely in apt-get-install. The compression step ran pretty fast.

i'm seeing package installation going really slow again in the image builder. I've hacked mine up where i can easily follow the download speed while trying to grab the stuff required to install my kernel. I'm going at about 50kB/s and its telling me it will take 40 minutes to download.

One thing we need to be cognizant of is that hpcloud seems to be much slower on grabbing stuff from our archives. Kind of makes sense I suppose.

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 15/03/14 04:56, Andy Doan wrote:
> I'm going at about 50kB/s and its telling me it will take 40
> minutes to download.
I was able to reproduce this and it took 24 mins for me to install
phablet-tools. Had a chat with the hp support and according to them
there was apparently nothing wrong in the imagebuilder instance and
that "It could be the nature of the way the load balancer handles
duplicate requests" in archive.u.c.

>
> One thing we need to be cognizant of is that hpcloud seems to be
> much slower on grabbing stuff from our archives. Kind of makes
> sense I suppose.
The issue does appear to be on archve.u.c side. Download speed on the
image builder instance from archive.ubuntu.com is 55.9 KB/s as opposed
to 11.9 MB/s to download an iso from cdimage.u.c. At the same point in
time this almost matches to the timings on my local machine. Hence I
don't think there is any issue in the instance itself or hp side. It
could be archive.u.c is heavily loaded during this time.

I'll poke at the IS about this.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.15 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTJNqDAAoJEA4SEg8jPuwAL+4H/0Cxx55eoP058xqUBXFf4rzy
d2ZlPdBzA7M5hRtSedx5fY4EsPogXw3CQWR8dEUK8DrDRE1TDbit3aKcS7fSWH35
XTo1xFex3SvoJUl2M+sickwuoVs2i8m1D8ODPGUYrGXHGi1LTDcYBmOJKPS0qiUR
Xh1goCn15hr4hRcUMJ6/MKy4tOizcPu8/IDz0kgVnSz+efNkJZwYQF7N0beikajR
9Xdb2NQuVEU1p7LQp4RsCuwikM5bStCMhP2UW55N5DlHhanynqhGuxmOaR8fvEGc
MB8NIskP849TD/OIgo/z7yAwdqNbxWU+Gq0cGV199/cDRIMooem9aWoZDLeotEA=
=Qtds
-----END PGP SIGNATURE-----

RT 68480 is reported on IS to look into this.

https://app.asana.com/0/9548224959691/11003358517617 is created to update the images to use the suitable archive mirrors