Merge lp:~jelmer/launchpad/reduce-svn-revision-limit into lp:launchpad

[Jelmer Vernooij]

Reduce the import size of SVN code imports to 500 revisions.

Reduce the batch size of SVN code imports to 500 revisions.

Whenever importing a single revision that is part of a code import batch fails, the entire batch is cancelled and discarded. Some hosting sites have unreliable SVN servers that cause occasional connection resets. The larger the batch size, the likelier it is to be interrupted by one of these connection resets.

Reducing the batch size from 5000 to 500 should reduce the impact of these connection resets when they occur.

There is some overhead to each batch (fetching the import base branch, and pushing the branch back after the import), so decreasing the batch size means that large branches will slightly some more time. That should be insignificant compared to the overall run time though.

There is more that we can do to improve this situation - such as preserving the revisions that have already been imported when the server hangs up - but I'll leave that for another MP.

[Graham Binns (gmb)]

Hi Jelmer,

Is this something we can test, or is that just wild optimism on my part?

review: Needs Information

[Jelmer Vernooij (jelmer)]

Am 29/11/11 11:53, schrieb Graham Binns:
> Review: Needs Information
>
> Hi Jelmer,
>
> Is this something we can test, or is that just wild optimism on my part?
It's tweaking of the parameters based on the real-life erratic behaviour
of SourceForge. I don't think there's much we can test here, unless we
want to add a handful of svn import runs against the actual SourceForge
server to the testsuite. ;-)

Cheers,

Jelmer

[Graham Binns (gmb)]

Hmm. Well, as tempting as that is, I think we can live without :).

review: Approve (code)

=== modified file 'lib/canonical/config/schema-lazr.conf'
--- lib/canonical/config/schema-lazr.conf	2011-11-28 08:30:05 +0000
+++ lib/canonical/config/schema-lazr.conf	2011-11-28 21:50:24 +0000
@@ -496,7 +496,7 @@
 git_revisions_import_limit: 30000
 
 # Import only this many revisions from svn (via bzr-svn) at once.
-svn_revisions_import_limit: 5000
+svn_revisions_import_limit: 500
 
 # Import only this many revisions from hg at once.
 hg_revisions_import_limit: 5000