Merge lp:~flacoste/ubuntu-archive-tools/copy-build-scheduler into lp:ubuntu-archive-tools

[Francis J. Lacoste]

This branch adds the copy-build-scheduler.py script. It can be use by a buildd administrator to process copy archive rebuilds in a timely manner.

Normally, copy builds have the lowest priority and are only being processed when the build farm is otherwise idle. That's fine for some cases, but to feed back the results of the rebuild into the normal Ubuntu QA cycle, the rebuild must be processed in a more timely manner.

This script loops until all the builds in the archive are completed. It will remove the copy archive penalty applied to some of the archive builds. But only a few at a time as not to take over the build farm. The default ratio is 0.25, meaning that up to (builders * 0.25) builds will be rescored. That should make sure that the rebuilds progress.

This should fix bug #805634.

[Robert Collins (lifeless)]

I have two specific comments.

Firstly the queue capacity changes over time; so that should be refreshed periodically - and it can change -fast- when QA nab machines for testing.

Secondly, I think 1 minute is too fast a polling time, something like 5 minutes would be a bit gentler. (I wish we had webhooks :P)

[lp:~flacoste/ubuntu-archive-tools/copy-build-scheduler]

280. By Francis J. Lacoste on 2011-07-27

Increase time between schedule run, and recompute capacity every 5 schedule iteration.

281. By Francis J. Lacoste on 2011-07-27

Typo.

[Francis J. Lacoste (flacoste)]

The queue capacity is actually pretty constant, except when HW enablement grabs machine for testing which shouldn't happen during copy rebuilds which require timely processing. (As they are now supposed to coordinate with platform to make sure they don't reduce capacity when it's most needed.)

Nonetheless, I did move the capacity computing within the loop and it's going to be recomputed every 25 minutes.

I also increase the time between schedule run to 5 minutes. The time we should use is probably around the time it takes for short build to complete. But 5 minutes will probably work fine any way.

Thanks for the review.

[Julian Edwards (julian-edwards)]

120 + archive = distribution.getArchive(name=args[0])
121 + if archive is None:
122 + parser.error('unknown archive: %s' % args[0])

It might be worth adding extra validation here that the selected archive's is_copy property is True.

142 + for build in pending_builds:
143 + if builds_to_rescore <= 0:
144 + break
145 +
146 + if build.arch_tag != processor:
147 + continue

You're going to iterate over the batch once for each architecture here, that'll create a lot of extra API requests won't it? It might be worth sorting the batch into a dict of builds lists indexed on archtag first perhaps?

Rob said:
> Firstly the queue capacity changes over time; so that should be refreshed periodically - and it can change -fast- when QA nab machines for testing.

Well spotted :)

review: Needs Fixing

[Robert Collins (lifeless)]

This should be landable now, no?

[Francis J. Lacoste (flacoste)]

I think so, but I've got no news from the ubuntu-archive-tools maintainers (and they are the only one who can land the change).

[Jamie Strandboge (jdstrand)]

I'll merge this-- it is a simple leaf application that shouldn't affect any other tools in ubuntu-archive-tools. I can only give a (very) high level review and things seem ok. Can someone comment on Julian's request for extra error checking?

review: Approve

[Francis J. Lacoste (flacoste)]

Thanks Jamie.

That error check cannot be added because is_copy isn't exported in the API.

=== added file 'copy-build-scheduler.py'
--- copy-build-scheduler.py	1970-01-01 00:00:00 +0000
+++ copy-build-scheduler.py	2011-07-27 14:58:35 +0000
@@ -0,0 +1,174 @@
+#! /usr/bin/python
+# Copyright 2011 Canonical Ltd.
+#
+# This script can be used to reschedule some of the copy archives
+# builds so that they are processed like regular PPA builds.
+#
+# Copy archives builds have a huge penalty applied to them which means
+# that they are only processed when there is nothing else being processed
+# by the build farm. That's usually fine, but for some rebuilds, we want
+# more timely processing, while at the same time, we do want to continue to 
+# service regular PPA builds.
+#
+# This script will try to have a portion of the build farm processing copy
+# builds. It does that by rescoring builds to the normal build priority
+# range. But will only rescore a few builds at a time, so as not to take ove
+# the build pool. By default, it won't rescore more than 1/4 the number of
+# available builders. So for example, if there are 12 i386 builders, only 
+# 3 builds at a time will have a "normal priority".
+import logging
+import sys
+import time
+
+from launchpadlib.launchpad import Launchpad
+from optparse import OptionParser
+
+
+API_NAME = 'copy-build-scheduler'
+
+NEEDS_BUILDING = 'Needs building'
+BUILDING = 'Currently building'
+COPY_ARCHIVE_SCORE_PENALTY = 2600
+# Number of minutes to wait between schedule run.
+SCHEDULE_PERIOD = 5
+
+def determine_builder_capacity(lp, options):
+    """Find how many builders to use for copy builds by processor."""
+    capacity = {}
+    for processor in options.processors:
+        queue = lp.builders.getBuildersForQueue(
+            processor='/+processors/%s' % processor, virtualized=True)
+        max_capacity = len(queue)
+        capacity[processor] = round(max_capacity * options.builder_ratio)
+        # Make sure at least 1 builders is associated
+        if capacity[processor] == 0:
+            capacity[processor] = 1
+        logging.info(
+            'Will use %d out of %d %s builders', capacity[processor],
+            max_capacity, processor)
+    return capacity
+
+
+def get_archive_used_builders_capacity(archive):
+    """Return the number of builds currently being done for the archive."""
+    capacity = {}
+    building = archive.getBuildRecords(build_state=BUILDING)
+    for build in building:
+        capacity.setdefault(build.arch_tag, 0)
+        capacity[build.arch_tag] += 1
+    return capacity
+
+
+def main(argv):
+    parser = OptionParser(
+        usage="%prog [options] copy-archive-name")
+    parser.add_option(
+        '--lp-instance', default='production', dest='lp_instance',
+        help="Select the Launchapd instance to run against. Defaults to "
+        "'production'")
+    parser.add_option(
+        '-v', '--verbose', default=0, action='count', dest='verbose',
+        help="Increase verbosity of the script. -v prints info messages"
+        "-vv will print debug messages.")
+    parser.add_option(
+        '-c', '--credentials', default=None, action='store',
+        dest='credentials',
+        help="Use the OAuth credentials in FILE instead of the desktop "
+        "one.", metavar='FILE')
+    parser.add_option(
+        '-d', '--distribution', default='ubuntu', action='store',
+        dest='distribution',
+        help="The archive distribution. Defaults to 'ubuntu'.")
+    parser.add_option(
+        '-p', '--processor', default=('i386', 'amd64'), action='append',
+        dest='processors',
+        help="The processor for which to schedule builds. "
+        "Default to i386 and amd64.")
+    parser.add_option(
+        '-r', '--ratio', default=0.25, action='store', type='float',
+        dest='builder_ratio',
+        help="The ratio of builders that you want to use for the copy "
+        "builds.  Default to 25% of the available builders.")
+    options, args = parser.parse_args(argv[1:])
+    if len(args) != 1:
+        parser.error('Missing archive name.')
+
+    if options.verbose >= 2:
+        log_level = logging.DEBUG
+    elif options.verbose == 1:
+        log_level = logging.INFO
+    else:
+        log_level = logging.WARNING
+    logging.basicConfig(level=log_level)
+
+    if options.builder_ratio >= 1 or options.builder_ratio < 0:
+        parser.error(
+            'ratio should be a float between 0 and 1: %s' %
+            options.builder_ratio)
+
+    lp = Launchpad.login_with(
+        API_NAME, options.lp_instance,
+        credentials_file=options.credentials,
+        version='devel')
+
+    try:
+        distribution = lp.distributions[options.distribution]
+    except KeyError:
+        parser.error('unknown distribution: %s' % options.distribution)
+
+    archive = distribution.getArchive(name=args[0])
+    if archive is None:
+        parser.error('unknown archive: %s' % args[0])
+
+    iteration = 0
+    while True:
+        # Every 5 schedules run - and on the first - compute available
+        # capacity.
+        if (iteration % 5) == 0:
+            capacity = determine_builder_capacity(lp, options)
+        iteration += 1
+
+        pending_builds = archive.getBuildRecords(build_state=NEEDS_BUILDING)
+        logging.debug('Found %d pending builds.' % len(pending_builds))
+        if len(pending_builds) == 0:
+            logging.info('No more builds pending. We are done.')
+            break
+
+        used_capacity = get_archive_used_builders_capacity(archive)
+
+        # For each processor, rescore up as many builds as we have
+        # capacity for.
+        for processor in options.processors:
+            builds_to_rescore = (
+                capacity[processor] - used_capacity.get(processor, 0))
+            logging.debug(
+                'Will try to rescore %d %s builds', builds_to_rescore,
+                processor)
+            for build in pending_builds:
+                if builds_to_rescore <= 0:
+                    break
+
+                if build.arch_tag != processor:
+                    continue
+
+                if build.score < 0:
+                    # Only rescore builds that look like the negative
+                    # copy archive modified have been applied.
+                    logging.info('Rescoring %s' % build.title)
+                    # This should make them considered like a regular build.
+                    build.rescore(
+                        score=build.score + COPY_ARCHIVE_SCORE_PENALTY)
+                else:
+                    logging.debug('%s already rescored', build.title)
+
+                # If the score was already above 0, it was probably
+                # rescored already, count it against our limit anyway.
+                builds_to_rescore -= 1
+
+        # Reschedule in a while.
+        logging.debug('Sleeping for %d minutes.', SCHEDULE_PERIOD)
+        time.sleep(SCHEDULE_PERIOD * 60)
+
+
+if __name__ == '__main__':
+    main(sys.argv)