Merge lp:~brian-murray/ubuntu-archive-tools/override-rate into lp:ubuntu-archive-tools

[Brian Murray]

There was no support in the phased-updater for overriding an increased rate of crashes about a package, this adds that. Additionally, the phased-updater does not look to see what the highest phased-update-percentage was set to so we could have a case where a phased update was set to 80%, stopped and then restarted at 10% which doesn't make a whole lot of sense. This fixes that by looking at the binary package publishing history to see what the maximum phased-update-percentage was.

[Colin Watson (cjwatson)]

18:10 <cjwatson> what happens if the package isn't built on i386 at all?
18:11 <cjwatson> maybe my proof of concept was a bit naive - you could search without distro_arch_series= and then go through the results looking for things where .distro_arch_series.distro_series (or however it's spelled) is release
18:11 <cjwatson> I'm also concerned that this appears to go through the entire history
18:12 <cjwatson> you should get it back in ordered form, so I think you would be better off stopping at the most recent publication that matches your criteria
18:12 <cjwatson> (i.e. you should get it most recent first IIRC)
18:13 <bdmurray> right and most recent is zero, so just use [1]?
18:15 <cjwatson> I'd use a for loop and break once you've found a publication you like

review: Needs Fixing

[lp:~brian-murray/ubuntu-archive-tools/override-rate]

768. By Brian Murray on 2013-08-08

updated based on reviewer feedback

769. By Brian Murray on 2013-08-12

be more verbose about why an email address was looked up and skipped

770. By Brian Murray on 2013-08-19

always set the rate_increase to false so that phasing is not stopped because of it

=== modified file 'phased-updater'
--- phased-updater	2013-08-07 16:26:41 +0000
+++ phased-updater	2013-08-19 19:39:52 +0000
@@ -240,10 +240,12 @@
                              signer)
                 continue
             signer_email = get_primary_email(signer)
-            if not signer_email:
-                continue
             pkg = spph.source_package_name
             version = spph.source_package_version
+            if not signer_email:
+                logging.info('no signer email address for %s %s %s' %
+                             (rname, pkg, version))
+                continue
             signer_name = signer.name
             changer = spph.package_creator
             if 'buckets' in spph_buckets[rname][spph]:
@@ -403,22 +405,25 @@
             # the fraction of hours that have passed so far in the day)
                 current_amount = data['this_count']
                 increase = current_amount - previous_amount
+            logging.info('[%s/%s] increase: %s, previous_avg: %s' %
+                         (release_version, src_pkg, increase,
+                          previous_amount))
             # 2013-06-20 - Fix errors api to add this
             link = data['web_link'] + '&version=%s' % version
+            logging.info('Details: %s' % link)
             return(increase, link)
 
 
 def main():
     #  TODO: make email code less redundant
-    #  TODO: Determine how to override an increased rate
     #  TODO: modify HTTP_USER_AGENT (both versions of urllib)
     #  TODO: Open bugs for regressions when false positives reduced
-    #  TODO: determine if starting p-u-p over from 0% is an issue
     ubuntu = launchpad.distributions['ubuntu']
     archive = ubuntu.getArchive(name='primary')
     options.archive = archive
 
     overrides = defaultdict(list)
+    rate_overrides = []
     override_file = csv.reader(open(OVERRIDES, 'r'))
     for row in override_file:
         # package, version, problem
@@ -427,7 +432,10 @@
         package = row[0].strip()
         version = row[1].strip()
         problem = row[2].strip()
-        overrides[(package, version)].append(problem)
+        if problem == 'increased-rate':
+            rate_overrides.append((package, version))
+        else:
+            overrides[(package, version)].append(problem)
 
     releases = []
     for series in ubuntu.series:
@@ -467,30 +475,46 @@
                     issues[rname][pub_source] = {}
                     issues[rname][pub_source]['buckets'] = problems
             rate_increase = crash_rate_increase(rvers, src_pkg, version)
+            # Unset the rate_increase until it is more useful
             if rate_increase:
+                logging.info('Disregarding rate increase for: %s' % src_pkg)
+                rate_increase = False
+            if rate_increase and (src_pkg, version) not in rate_overrides:
                 if pub_source not in issues[rname]:
                     issues[rname][pub_source] = {}
                 issues[rname][pub_source]['rate'] = rate_increase
-            pups = [pb.phased_update_percentage
-                    for pb in pub_source.getPublishedBinaries()
-                    if pb.phased_update_percentage]
-            if pups:
+            pbs = [pb for pb in pub_source.getPublishedBinaries()
+                   if pb.phased_update_percentage is not None]
+            if pbs:
+                # the p-u-p is currently the same for all binary packages
+                last_pup = pbs[0].phased_update_percentage
                 if pub_source not in issues[rname]:
                     issues[rname][pub_source] = {}
-                issues[rname][pub_source]['pup'] = pups[0]
+                # phasing has stopped so check what the max value was
+                if last_pup == 0:
+                    for allpb in archive.getPublishedBinaries(
+                            exact_match=True, pocket='Updates',
+                            binary_name=pbs[0].binary_package_name):
+                        if allpb.distro_arch_series.distroseries == release:
+                            if allpb.phased_update_percentage > 0:
+                                max_pup = allpb.phased_update_percentage
+                                break
+                    issues[rname][pub_source]['pup'] = max_pup
+                else:
+                    issues[rname][pub_source]['pup'] = last_pup
             suite = rname + '-updates'
             if pub_source not in issues[rname]:
                 continue
             elif ('rate' not in issues[rname][pub_source] and
                   'buckets' not in issues[rname][pub_source] and
-                  pups):
+                  pbs):
                 # there is not an error so increment the phasing
                 current_pup = issues[rname][pub_source]['pup']
                 new_pup = current_pup + PUP_INCREMENT
                 if not options.no_act:
                     set_pup(current_pup, new_pup, release, suite, src_pkg)
                 issues[rname][pub_source]['pup'] = new_pup
-            elif pups:
+            elif pbs:
                 # there is an error and pup is not None so stop the phasing
                 current_pup = issues[rname][pub_source]['pup']
                 issues[rname][pub_source]['previous_pup'] = current_pup