Merge lp:~vila/udd/717109-create-import-jobs-backlog into lp:udd

[Vincent Ladeuil]

This fixes bug #717109 so we can re-start a package importer from scratch.

This should be enough to seed the importer.

Adding a command-line option may be a nice followup but I don't need it for now.

[James Westby (james-w)]

review: Approve

[John A Meinel (jameinel)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/11/2011 9:35 AM, Vincent Ladeuil wrote:
> Vincent Ladeuil has proposed merging lp:~vila/udd/717109-create-import-jobs-backlog into lp:udd.
>
> Requested reviews:
> Ubuntu Distributed Development Developers (udd)
> Related bugs:
> #717109 lp timeout when trying to start a package importer from scratch
> https://bugs.launchpad.net/bugs/717109
>
> For more details, see:
> https://code.launchpad.net/~vila/udd/717109-create-import-jobs-backlog/+merge/49406
>
> This fixes bug #717109 so we can re-start a package importer from scratch.
>
> This should be enough to seed the importer.
>
> Adding a command-line option may be a nice followup but I don't need it for now.

I know James approved this. But I did send this earlier:
As I mentioned on the bug, this is dangerous because it creates a window
where there can be packages that were uploaded but will never be seen.

The fact that future updates will cause us to go back and import them
means this shouldn't be a data-loss situation, but I'm uncomfortable
with it.

Apparently "status: needs_information" rejects the whole message.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1Vfc8ACgkQJdeBCYSNAAMcgwCffeI5c4Z5rZQ5HYncr9+qsZ2G
uq0AoKv5rmReLo3eUYmw5RpBS4GOz43P
=eFlx
-----END PGP SIGNATURE-----

[Robert Collins (lifeless)]

Did you file an LP bug about the timeout? And include the OOPS?

[Vincent Ladeuil (vila)]

@John: As mentioned in the bug, this script is mostly to seed the p-i, as soon as a package enters the p-i, all its uploads are caught up. From James explanation, I understand that this is how the p-i has been seeded in the past so if the approach is wrong this needs to be addressed but this mp was about unblocking the situation with the *current* approach.

@Robert: I didn't but this is not urgent anymore and the request against lp may not be ideal anyway. I'll track it better the next time I'll work in this area.

=== modified file 'icommon.py'
--- icommon.py	2011-02-08 02:58:52 +0000
+++ icommon.py	2011-02-11 15:35:23 +0000
@@ -344,20 +344,28 @@
     last_known_published = status_db.last_import_time()
     last_published_call = None
     if last_known_published is not None:
-        last_known_published = \
-                last_known_published - datetime.timedelta(0, 600)
-        last_published_call = last_known_published.isoformat()
+        last_known_published = (last_known_published
+                                - datetime.timedelta(0, 600))
+    else:
+        # When starting from scratch, limit the query to the last ten days or
+        # lp timeout (https://bugs.launchpad.net/udd/+bug/717109)
+        last_known_published = (datetime.datetime.now()
+                                - datetime.timedelta(0, 10 * 24 * 3600))
+    last_published_call = last_known_published.isoformat()
     # We want all the new Published records since the last run
     publications = set()
-    for p in iterate_collection(lp_call(call_with_limited_size,
-            lp.distributions['ubuntu'].main_archive.getPublishedSources,
-            status="Published",
-            created_since_date=last_published_call)):
+    collection = lp_call(
+        call_with_limited_size,
+        lp.distributions['ubuntu'].main_archive.getPublishedSources,
+        status="Published",
+        created_since_date=last_published_call)
+    for p in iterate_collection(collection):
         publications.add(p)
-    for p in iterate_collection(lp_call(call_with_limited_size,
+    collection = lp_call(call_with_limited_size,
             lp.distributions['debian'].main_archive.getPublishedSources,
             status="Pending",
-            created_since_date=last_published_call)):
+            created_since_date=last_published_call)
+    for p in iterate_collection(collection):
         publications.add(p)
     import_from_publications(status_db, publications, last_known_published)