Merge lp:~beuno/loggerhead/export-tarball into lp:loggerhead

[Martin Albisetti]

This adds some basic support for exporting tarballs of revisions.
It also allows excluding this feature by running BranchWSGIApp with allow_tarballs=False.

There are 2 more things I can think to add to this, but I've decided to defer that work after this lands:
- Allow specifying the cache dir
- Allow specifying zip or tarballs

[lp:~beuno/loggerhead/export-tarball]

432. By Martin Albisetti on 2011-03-10

Allow conditionally exporting tarballs, now with 300% less cruft.

[Robert Collins (lifeless)]

Cache directory seems like a bad idea to me - will create stuff we have to clean up, and either the tarball is cheap enough to make that its ok to do it live (at all), or its too expensive and shouldn't be enabled on a given server.

review: Needs Fixing

[xaav (xaav)]

@beuno How is this coming along?

review: Needs Information

[Martin Albisetti (beuno)]

I need to re-work this patch. Proper support for this has landed in bzrlib, so now I need to make use of that, and also make sure we either have a fall-back for older versions (unlikely, lots of work and hackish) or only display the option if it's supported.
Hope to get back to this after the Natty release craze dies down.

[xaav (xaav)]

@beuno When (approximately) is this going to be ready?

[xaav (xaav)]

Please see https://answers.launchpad.net/bzr/+question/159205

[Robert Collins (lifeless)]

I'd like us to stream it directly to the client.

Making the tarball even just in-memory, and then serving it adds the generation time to the overall delivery time: our load balancing software will timeout connections for non-trivial projects if we do it that way.

[Martin Albisetti (beuno)]

Geoff, see: https://code.launchpad.net/~jelmer/bzr/export-tgz-711226/+merge/53235

[Robert Collins (lifeless)]

I'm rejecting this branch - xaav's approach will work a lot better.

[Robert Collins (lifeless)]

https://code.launchpad.net/~xaav/loggerhead/export-tarball/+merge/63931 is the other approach.

Unmerged revisions

=== modified file '.bzrignore'
--- .bzrignore	2010-04-22 08:52:59 +0000
+++ .bzrignore	2011-03-10 03:57:10 +0000
@@ -9,3 +9,4 @@
 loggerhead-memprofile
 ./docs/_build/
 tags
+loggerhead/static/downloads/*

=== modified file 'loggerhead/apps/branch.py'
--- loggerhead/apps/branch.py	2011-02-09 22:29:57 +0000
+++ loggerhead/apps/branch.py	2011-03-10 03:57:10 +0000
@@ -33,7 +33,7 @@
 from loggerhead.controllers.atom_ui import AtomUI
 from loggerhead.controllers.changelog_ui import ChangeLogUI
 from loggerhead.controllers.diff_ui import DiffUI
-from loggerhead.controllers.download_ui import DownloadUI
+from loggerhead.controllers.download_ui import DownloadUI, DownloadTarballUI
 from loggerhead.controllers.filediff_ui import FileDiffUI
 from loggerhead.controllers.inventory_ui import InventoryUI
 from loggerhead.controllers.revision_ui import RevisionUI
@@ -49,7 +49,7 @@
 
     def __init__(self, branch, friendly_name=None, config={},
                  graph_cache=None, branch_link=None, is_root=False,
-                 served_url=_DEFAULT, use_cdn=False):
+                 served_url=_DEFAULT, use_cdn=False, export_tarballs=True):
         self.branch = branch
         self._config = config
         self.friendly_name = friendly_name
@@ -61,6 +61,7 @@
         self.is_root = is_root
         self.served_url = served_url
         self.use_cdn = use_cdn
+        self.export_tarballs = export_tarballs
 
     def get_history(self):
         file_cache = None
@@ -126,6 +127,7 @@
         'revision': RevisionUI,
         'search': SearchUI,
         'view': ViewUI,
+        'tarball': DownloadTarballUI,
         }
 
     def last_updated(self):
@@ -171,10 +173,14 @@
         try:
             try:
                 c = cls(self, self.get_history)
-                return c(environ, start_response)
+                to_ret = c(environ, start_response)
             except:
                 environ['exc_info'] = sys.exc_info()
                 environ['branch'] = self
                 raise
+            if type(to_ret) == type(httpexceptions.HTTPSeeOther('/')):
+                raise to_ret
+            else:
+                return to_ret
         finally:
             self.branch.unlock()

=== modified file 'loggerhead/config.py'
--- loggerhead/config.py	2010-04-13 23:35:01 +0000
+++ loggerhead/config.py	2011-03-10 03:57:10 +0000
@@ -36,6 +36,7 @@
         use_cdn=False,
         sql_dir=None,
         allow_writes=False,
+        export_tarballs=True,
         )
     parser.add_option("--user-dirs", action="store_true",
                       help="Serve user directories as ~user.")
@@ -69,6 +70,8 @@
                       help="The directory to place the SQL cache in")
     parser.add_option("--allow-writes", action="store_true",
                       help="Allow writing to the Bazaar server.")
+    parser.add_option("--export-tarballs", action="store_true",
+                      help="Allow exporting revisions to tarballs.")
     return parser
 
 

=== modified file 'loggerhead/controllers/__init__.py'
--- loggerhead/controllers/__init__.py	2011-02-10 17:01:10 +0000
+++ loggerhead/controllers/__init__.py	2011-03-10 03:57:10 +0000
@@ -20,7 +20,7 @@
 import bzrlib.errors
 import time
 
-from paste.httpexceptions import HTTPNotFound
+from paste.httpexceptions import HTTPNotFound, HTTPSeeOther
 from paste.request import path_info_pop, parse_querystring
 
 from loggerhead import util
@@ -93,6 +93,8 @@
         headers = {}
 
         vals.update(self.get_values(path, kwargs, headers))
+        if 'download' in vals:
+            return HTTPSeeOther(vals['download'])
 
         self.log.info('Getting information for %s: %r secs' % (
             self.__class__.__name__, time.time() - z))

=== modified file 'loggerhead/controllers/download_ui.py'
--- loggerhead/controllers/download_ui.py	2010-05-05 19:03:40 +0000
+++ loggerhead/controllers/download_ui.py	2011-03-10 03:57:10 +0000
@@ -19,6 +19,7 @@
 
 import logging
 import mimetypes
+import os
 import urllib
 
 from paste import httpexceptions
@@ -67,3 +68,29 @@
             ]
         start_response('200 OK', headers)
         return [content]
+
+class DownloadTarballUI(TemplatedBranchView):
+    """Download a revno as a tarball or zip file."""
+    ext = 'tar.gz'
+    dest = os.path.join(os.getcwd(), 'loggerhead/static/downloads/')
+    download_dir = '/static/downloads/'
+ 
+    def get_values(self, path, kwargs, headers):
+        """Return a URL to a tarball.
+
+        In the form of: /tarball/revno_or_revid."""
+        history = self._history
+        if len(self.args):
+            revid = history.fix_revid(self.args[0])
+        else:
+            revid = self.get_revid()
+        revno = history.get_revno(revid)
+        filename = '%s_%s.%s' % (history._branch_nick, revno, self.ext)
+        rpath = os.getcwd()+ '/loggerhead/static/downloads/' + filename
+        if not os.path.exists(rpath):
+            history.export(revid, rpath)
+        if self._branch.export_tarballs:
+            return {'download': '/static/downloads/' + filename}
+        else:
+            # redirect to the home page, the user is cheating :)
+            return {'download': '/'}

=== modified file 'loggerhead/controllers/revision_ui.py'
--- loggerhead/controllers/revision_ui.py	2011-03-02 14:07:21 +0000
+++ loggerhead/controllers/revision_ui.py	2011-03-10 03:57:10 +0000
@@ -110,7 +110,7 @@
                 self._branch.friendly_name,
                 self._branch.is_root,
                 'changes'))
-
+        can_export = self._branch.export_tarballs
         return {
             'branch': self._branch,
             'revid': revid,
@@ -132,4 +132,5 @@
             'compare_revid': compare_revid,
             'url': self._branch.context_url,
             'directory_breadcrumbs': directory_breadcrumbs,
+            'can_export': can_export,
         }

=== modified file 'loggerhead/history.py'
--- loggerhead/history.py	2010-05-10 13:48:25 +0000
+++ loggerhead/history.py	2011-03-10 03:57:10 +0000
@@ -38,6 +38,7 @@
 import bzrlib.branch
 import bzrlib.delta
 import bzrlib.errors
+import bzrlib.export
 import bzrlib.foreign
 import bzrlib.revision
 
@@ -760,3 +761,9 @@
             removed=sorted(reporter.removed, key=lambda x: x.filename),
             modified=sorted(reporter.modified, key=lambda x: x.filename),
             text_changes=sorted(reporter.text_changes, key=lambda x: x.filename))
+
+    def export(self, revid, dest):
+        """Export a tarball of a revision."""
+        rev_tree = self._branch.repository.revision_tree(revid)
+        bzrlib.export.export(rev_tree, dest)
+        return dest

=== added directory 'loggerhead/static/downloads'
=== modified file 'loggerhead/templates/revision.pt'
--- loggerhead/templates/revision.pt	2011-02-23 03:01:15 +0000
+++ loggerhead/templates/revision.pt	2011-03-10 03:57:10 +0000
@@ -84,7 +84,10 @@
                tal:attributes="href python:url(['/diff', change.revno], clear=1)">download diff</a>
             <a tal:condition="python:compare_revid is not None"
                tal:attributes="href python:url(['/diff', change.revno, history.get_revno(compare_revid)], clear=1)">download diff</a>
-          </li>
+           </li>
+           <li tal:condition="python:can_export">
+            <a tal:attributes="href python:url(['/tarball', change.revno])">download tarball</a>
+           </li>
           <li id="last"><a tal:attributes="href python:url(['/changes', change.revno]);
                                            title string:view history from revision ${change/revno}"
                            tal:content="string:view history from revision ${change/revno}"></a></li>