Merge ~racb/usd-importer:repo-builder into usd-importer:master

[Robie Basak]

Make jenkins happy.

[Robie Basak (racb)]

Perhaps I should s/Link/Symlink/.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:91e2d9ffacb7ef0a3fc38965201630f5d9dab948
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+merge/333040/+edit-commit-message

https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/190/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/190/rebuild

review: Needs Fixing (continuous-integration)

[Nish Aravamudan (nacc)]

This is not your fault, master is broken due to a typo on my part.

On Tue, Oct 31, 2017 at 11:23 AM, Server Team CI bot
<email address hidden> wrote:
> Review: Needs Fixing continuous-integration
>
> FAILED: Continuous integration, rev:91e2d9ffacb7ef0a3fc38965201630f5d9dab948
> No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
> https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+merge/333040/+edit-commit-message
>
> https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/190/
> Executed test runs:
> SUCCESS: Checkout
> SUCCESS: Style Check
> SUCCESS: Unit Tests
> FAILED: Integration Tests
>
> Click here to trigger a rebuild:
> https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/190/rebuild
>
> --
> https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+merge/333040
> Your team Ubuntu Server Dev import team is requested to review the proposed merge of ~racb/usd-importer:repo-builder into usd-importer:master.

[Robie Basak (racb)]

I intend to squash these two commits up before landing, but I've left them here to allow incremental review.

[Nish Aravamudan (nacc)]

On 31.10.2017 [17:30:59 -0000], Robie Basak wrote:
> Perhaps I should s/Link/Symlink/.

Link is missing some super() / **kwargs-foo.

Otherwise, the code seems really nice and clean. +1 from me on the
direction.

--
Nishanth Aravamudan
Ubuntu Server
Canonical Ltd

[Nish Aravamudan (nacc)]

Oh sorry Robie, I wasn't notified of the changes, just your comment. Weird.

If you rebase to master now, it hsould pass CI.

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:3bb6b1ccbdb0093fa5e614126b9ec039ea09d248
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+merge/333040/+edit-commit-message

https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/191/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/191/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:78b22fae3c034b83a35e47eeabe96ef5a95d85a7
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+merge/333040/+edit-commit-message

https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/192/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/192/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:78b22fae3c034b83a35e47eeabe96ef5a95d85a7
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~racb/usd-importer/+git/usd-importer/+merge/333040/+edit-commit-message

https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/193/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/193/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

FAILED: Continuous integration, rev:333f5306ad7249f8cc4062fb87671708f49f753b
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/199/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    FAILED: Integration Tests

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/199/rebuild

review: Needs Fixing (continuous-integration)

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:333f5306ad7249f8cc4062fb87671708f49f753b
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/200/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/200/rebuild

review: Approve (continuous-integration)

[Nish Aravamudan (nacc)]

Robie,

Do you want me to land this?

[Nish Aravamudan (nacc)]

review: Approve

[Server Team CI bot (server-team-bot)]

PASSED: Continuous integration, rev:80b7dc41bcfab583a03367d02df113ac8fdcbf4e
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/209/
Executed test runs:
    SUCCESS: Checkout
    SUCCESS: Style Check
    SUCCESS: Unit Tests
    SUCCESS: Integration Tests
    IN_PROGRESS: Declarative: Post Actions

Click here to trigger a rebuild:
https://jenkins.ubuntu.com/server/job/git-ubuntu-ci/209/rebuild

review: Approve (continuous-integration)

diff --git a/gitubuntu/git_repository.py b/gitubuntu/git_repository.py
index 86d1a26..c598c10 100644
--- a/gitubuntu/git_repository.py
+++ b/gitubuntu/git_repository.py
@@ -73,12 +73,11 @@ def _follow_symlinks_to_blob(repo, top_tree_object, search_path,
     elif entry.type == 'blob' and entry.filemode == pygit2.GIT_FILEMODE_LINK:
         # Found a symlink. Start again from the top with adjustment for symlink
         # following
+        target_tail = [decode_binary(repo.get(entry.id).data)]
+        if tail is not None:
+            target_tail.append(tail)
         search_path = posixpath.normpath(
-            posixpath.join(
-                _rel_path,
-                decode_binary(repo.get(entry.id).data),
-                tail,
-            )
+            posixpath.join(_rel_path, *target_tail)
         )
         return _follow_symlinks_to_blob(
             repo=repo,
diff --git a/gitubuntu/repo_builder.py b/gitubuntu/repo_builder.py
new file mode 100644
index 0000000..cb93278
--- /dev/null
+++ b/gitubuntu/repo_builder.py
@@ -0,0 +1,250 @@
+import functools
+import tempfile
+import unittest
+
+import pygit2
+import pytest
+
+
+"""Build test git repositories as data structures
+
+Represent a git repository as a data structure, nesting using Tree objects as
+required. Name objects in a flat namespace using the name keyword object to
+these constructors.
+
+Reference the same object with a Placeholder giving the name of the target to
+its constructor. This allows a build in a single data structure instead of
+having to use multiple statements to grab references to other parts of the same
+structure.
+
+Write any object (including any children as necessary) using the write method,
+which takes a single pygit2.Repository parameter. This returns the pygit2.Oid
+of the top level written object.
+
+Objects must be treated as immutable. The result of mutating an object after it has been constructed is undefined.
+"""
+
+
+__all__ = [
+    'Blob',
+    'Commit',
+    'ExecutableBlob',
+    'Symlink',
+    'Placeholder',
+    'Tree',
+]
+
+# General notes
+
+# Sometimes tests need to check an object hash matches something in a supplied
+# builder repo tree. In this case, it seems redundant and messy to have to
+# write that object again to get its hash. Perhaps objects should cache their
+# hashes or something?
+
+
+class NamedMixin:
+    def __init__(self, name=None):
+        self.name = name
+
+
+class WriteMixin:
+    def write(self, repo, record=None):
+        record = record or dict()
+        try:
+            return record[self]
+        except KeyError:
+            record[self] = self._obj_to_oid(repo, record)
+            return record[self]
+
+
+class Placeholder:
+    def __init__(self, target_name):
+        self.target_name = target_name
+
+    def walk(self, parent=None):
+        yield parent, self
+
+
+class Blob(NamedMixin, WriteMixin):
+    GIT_TREE_ATTR = pygit2.GIT_FILEMODE_BLOB
+
+    def __init__(self, content, **kwargs):
+        """Construct a Blob
+
+        @param content: a bytes object of the desired blob contents
+        """
+        super().__init__(**kwargs)
+        self.content = content
+
+    def walk(self, parent=None):
+        yield parent, self
+
+    def _obj_to_oid(self, repo, record):
+        return repo.create_blob(self.content)
+
+
+class ExecutableBlob(Blob):
+    GIT_TREE_ATTR = pygit2.GIT_FILEMODE_BLOB_EXECUTABLE
+
+
+class Symlink(NamedMixin, WriteMixin):
+    GIT_TREE_ATTR = pygit2.GIT_FILEMODE_LINK
+
+    def __init__(self, target, **kwargs):
+        """Construct a symlink
+
+        @param target: a bytes object containing the target
+
+        target is not interpreted; it is exactly equivalent to the first
+        parameter to ln(1).
+        """
+        super().__init__(**kwargs)
+        self.target = target
+
+    def walk(self, parent=None):
+        yield parent, self
+
+    def _obj_to_oid(self, repo, record):
+        return repo.create_blob(self.target)
+
+
+class Tree(NamedMixin, WriteMixin):
+    GIT_TREE_ATTR = pygit2.GIT_FILEMODE_TREE
+
+    def __init__(self, entries, **kwargs):
+        """Construct a tree
+
+        A tree is equivalent to a directory and contains zero or more entries
+
+        @param entries: a dictionary whose values are other objects from this
+                        module
+        """
+        super().__init__(**kwargs)
+        self.entries = entries
+
+    def __getitem__(self, k):
+        if k == self.name:
+            return self
+
+        # XXX inefficient
+        for entry in self.entries.values():
+            if isinstance(entry, Placeholder):
+                # As an exception, Placeholders don't have names so cannot be
+                # located using this method.
+                continue
+            if entry.name == k:
+                return entry
+            elif isinstance(entry, type(self)):
+                try:
+                    return entry[k]
+                except KeyError:
+                    pass
+        raise KeyError('Object named %r not found' % k)
+
+    def replace(self, old, new):
+        # XXX inefficient
+        for name, entry in self.entries.items():
+            if entry is old:
+                self.entries[name] = new
+                return
+        raise KeyError("Cannot find %r in entries" % old)
+
+    def walk(self, parent=None):
+        for entry in self.entries.values():
+            yield from entry.walk(parent=self)
+        yield parent, self
+
+    def _obj_to_oid(self, repo, record):
+        tree_builder = repo.TreeBuilder()
+        for name, entry in self.entries.items():
+            tree_builder.insert(
+                name,  # name
+                entry.write(repo, record=record),  # oid
+                entry.GIT_TREE_ATTR,  # attr
+            )
+        return tree_builder.write()
+
+
+class Commit(NamedMixin, WriteMixin):
+    def __init__(self, tree, **kwargs):
+        super().__init__(**kwargs)
+        self.tree = tree
+
+    def walk(self, parent=None):
+        yield parent, self
+
+    def _obj_to_oid(self, repo, record=None):
+        return repo.create_commit(
+            None,
+            pygit2.Signature(name='Test Builder', email='[e-mail]'),
+            pygit2.Signature(name='Test Builder', email='[e-mail]'),
+            'Test commit',
+            self.tree.write(repo),
+            [],
+        )
+
+    def __getitem__(self, k):
+        return self if self.name == k else self.tree[k]
+
+
+def replace_placeholders(top):
+    for parent, obj in top.walk():
+        if isinstance(obj, Placeholder):
+            parent.replace(obj, top[obj.target_name])
+
+
+class TestObjectCreation(unittest.TestCase):
+    def setUp(self):
+        self.git_dir = tempfile.TemporaryDirectory()
+        self.repo = pygit2.init_repository(self.git_dir.name)
+
+    def tearDown(self):
+        self.repo = None
+        self.git_dir.cleanup()
+
+    def testBlobCreation(self):
+        ref = Blob(b'foo').write(self.repo)
+        assert self.repo.get(ref).data == b'foo'
+
+    def testExecutableBlobTreeCreation(self):
+        tree_ref = Tree({'foo': ExecutableBlob(b'bar')}).write(self.repo)
+        tree_entry = self.repo.get(tree_ref)['foo']
+        assert self.repo.get(tree_entry.id).data == b'bar'
+        assert tree_entry.filemode == pygit2.GIT_FILEMODE_BLOB_EXECUTABLE
+
+    def testSymlinkTreeCreation(self):
+        tree_ref = Tree({'foo': Symlink(b'bar')}).write(self.repo)
+        tree_entry = self.repo.get(tree_ref)['foo']
+        assert tree_entry.filemode == pygit2.GIT_FILEMODE_LINK
+
+    def testCommitCreation(self):
+        tree = Tree({'foo': Blob(b'bar')})
+        tree_ref = tree.write(self.repo)
+        commit_ref = Commit(tree).write(self.repo)
+        commit = self.repo.get(commit_ref)
+        assert commit.tree_id == tree_ref
+
+    def testNameSearch(self):
+        inner_tree = Tree({'foo': Blob(b'bar', name='blob')}, name='inner')
+        outer_tree = Tree({'baz': inner_tree}, name='outer')
+        commit = Commit(outer_tree, name='commit')
+        assert commit['commit'] is commit
+        assert commit['inner'] is inner_tree
+        assert commit['outer'] is outer_tree
+        assert isinstance(commit['blob'], Blob)
+        with pytest.raises(KeyError):
+            commit['absent']
+
+
[e-mail]('cls', [
+    functools.partial(Blob, b'qux'),
+    functools.partial(ExecutableBlob, b'qux'),
+    functools.partial(Symlink, 'target'),
+    functools.partial(Tree, {}),
+])
+def test_placeholder(cls):
+    common_blob = cls(name='name')
+    top = Tree({'foo': common_blob, 'bar': Placeholder('name')})
+    assert top.entries['foo'] is not top.entries['bar']
+    replace_placeholders(top)
+    assert top.entries['foo'] is top.entries['bar']
diff --git a/gitubuntu/test_git_repository.py b/gitubuntu/test_git_repository.py
new file mode 100644
index 0000000..640dc16
--- /dev/null
+++ b/gitubuntu/test_git_repository.py
@@ -0,0 +1,85 @@
+import tempfile
+import unittest
+
+import pygit2
+import pytest
+
+from gitubuntu.git_repository import follow_symlinks_to_blob
+from gitubuntu.repo_builder import (
+    Blob,
+    Symlink,
+    Tree,
+)
+
+
[e-mail]
+def repo():
+    """A empty pygit2 repository fixture in a temporary directory"""
+    with tempfile.TemporaryDirectory() as tmp_dir:
+        yield pygit2.init_repository(tmp_dir)
+
+
[e-mail]('tree_func', [
+    # The tree_func parameter is a function that accepts a mock Blob that is to
+    # represent the changelog blob itself and returns a mock Tree with the mock
+    # Blob embedded somewhere within it. The test function can then ensure that
+    # follow_symlinks_to_blob can correctly find the changelog Blob given the
+    # Tree.
+
+    # Of course this is only expected to work if, after checking out the Tree,
+    # "cat debian/changelog" would work. But this allows us to test the various
+    # permutations of symlink following in Trees that _are_ valid.
+
+    # Simple case
+    lambda b: Tree({
+        'debian': Tree({'changelog': b}),
+    }),
+
+    # Symlink in debian/
+    lambda b: Tree({
+        'debian': Tree({
+                    'changelog.real': b,
+                    'changelog': Symlink('changelog.real'),
+                }),
+    }),
+
+    # Symlink to parent directory
+    lambda b: Tree({
+        'changelog': b,
+        'debian': Tree({
+            'changelog': Symlink('../changelog'),
+        })
+    }),
+
+    # Symlink to subdirectory
+    lambda b: Tree({
+        'debian': Tree({
+            'changelog': Symlink('subdirectory/changelog'),
+            'subdirectory': Tree({'changelog': b}),
+        })
+    }),
+
+    # debian/ itself is a symlink to a different directory
+    lambda b: Tree({
+        'pkg': Tree({'changelog': b}),
+        'debian': Symlink('pkg'),
+    })
+])
+def test_follow_symlinks_to_blob(repo, tree_func):
+    blob = Blob(b'')
+    blob_id = blob.write(repo)
+    tree = repo.get(tree_func(blob).write(repo))
+    result_blob = follow_symlinks_to_blob(repo, tree, 'debian/changelog')
+    assert result_blob.id == blob_id
+
+
[e-mail]('tree', [
+    Tree({}),
+    Tree({'debian': Tree({})}),
+    Tree({'debian': Tree({'changelog': Symlink('other')})}),
+    Tree({'debian': Tree({'changelog': Symlink('../other')})}),
+])
+def test_follow_symlinks_to_blob_not_found(repo, tree):
+    pygit2_tree = repo.get(tree.write(repo))
+    with pytest.raises(KeyError):
+        follow_symlinks_to_blob(repo, pygit2_tree, 'debian/changelog')