lp timeout when trying to start a package importer from scratch

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/11/2011 7:11 AM, Vincent Ladeuil wrote:
> Public bug reported:
>
> icommon.create_import_jobs queries lp for PublishedSources since its
> last run (minus 10 minutes).
>
> This is fine when the db is up to date, but when trying to start a test
> p-i from scratch I ran into timeouts from lp.
>
> This runs from add_import_jobs.py which is invoked by cron every 5
> minutes.
>
> ** Affects: udd
> Importance: Undecided
> Status: Confirmed
>
> ** Changed in: udd
> Status: New => Confirmed
>

Note that some of this is just the shear number of packages it needs to
be informed about. Because of stuff like wheezy being added, etc.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1VPPcACgkQJdeBCYSNAAPm+wCfe1nuom+5a4c1IuBIlkOUzVQI
jrkAoIXzSqJOqomOeyE+Est2y+PRS71k
=TpmC
-----END PGP SIGNATURE-----

> Because of stuff like wheezy being added, etc.

No, I reported the bug today but encountered it before wheezy was released.

I have a branch working around the problem by pretending the job had its last run a couple of days ago.

I'm running a test right now with couple == 10 to ensure it's robust enough.

Since this is used to restart a p-i from scratch this should be enough to seed it (which anyway is better than not being able to seed it at all).

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 2/11/2011 9:03 AM, Vincent Ladeuil wrote:
>> Because of stuff like wheezy being added, etc.
>
> No, I reported the bug today but encountered it before wheezy was
> released.
>
> I have a branch working around the problem by pretending the job had its
> last run a couple of days ago.
>
> I'm running a test right now with couple == 10 to ensure it's robust
> enough.
>
> Since this is used to restart a p-i from scratch this should be enough
> to seed it (which anyway is better than not being able to seed it at
> all).
>

If you pretend the last run was a couple of days ago, then you'll miss
all of the new code drops that happened between the time of the last
successful scan and the time you are pretending.

For example:
 day 1 - process is stopped
 day 2 - package A and B are uploaded
 day 3 - package C and D are uploaded
 day 4-10 - Packages E-Z are uploaded

If you claim the imported had succeeded on day 4, then you'll miss all
of A B C and D.

I think at the moment, that just leads to stale branches, because if you
get bzr-2.1-ubuntu1 uploaded on day 2, and bzr-2.1-ubuntu2 uploaded on
day 10, when it goes to import bzr-2.1-ubuntu2 it will still import
bzr-2.1-ubuntu1.

John
=:->

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Cygwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk1VYJoACgkQJdeBCYSNAAP6mwCfX8cgwcOz3da1DDIW91ipQL0U
onIAoJf31uG2xEc1hYMXR1JjRyhR0Cvy
=laZ6
-----END PGP SIGNATURE-----

AIUI (based on a discussion with james_w), the script is mainly to discover new packages and the rest of the importer will catch up.