Merge ~apw/+git/kteam-tools:cranky/checkout--add-roles-for-prepare-and-apply into ~canonical-kernel/+git/kteam-tools:master

[Andy Whitcroft]

This patch stack adds support to `cranky checkout` for separation of
roles for kernel preparation and patch application. This adds new
command line options to checkout: --prepare-kernels and --apply-patches.
Defaulting to --prepare-kernels for compatibility with current
semantics.

This patch set changes the local branch namespace for cranky generated
branches to `cranky-{prepare,apply}/<kernel-suffix>`. This is done
to disambiguate clearly which function the local branch fills. It also
simplifies branch destination selection for push.

This patch set changes a number of things:

1) it adds the concept of local_branch to our Handle objects allowing us
to reuse the existing overlap handling used for remotes,
2) it adds the concept of local_branch to our GitHandle object allowing
us to feed roll and source information into the branch selection,
3) adds some extra summary information indicating where the branch
came from (and should push to), and
4) automatically keeps the git concept of upstream branch updated.

This last feature in concert with a simple addition to the users basic
git config allows us to simply `git push` on any `cranky-*/*` branch and
have it go back to whence it came. It is hoped this will simplify or
eliminate the need for a `cranky push`.

8d0c7f2... by Andy Whitcroft on 2021-04-07

cranky: docs/cranking-the-kernel -- add patch application instructions

Add patch application instructions. As we now need slightly more git
configuration also add a new section on that which lists all the main
configuration you should need to change.

Signed-off-by: Andy Whitcroft <email address hidden>

08c1545... by Andy Whitcroft on 2021-04-07

cranky: crl/git -- default role to prepare when not specified

Signed-off-by: Andy Whitcroft <email address hidden>

Unmerged commits

08c1545... by Andy Whitcroft on 2021-04-07

cranky: crl/git -- default role to prepare when not specified

Signed-off-by: Andy Whitcroft <email address hidden>

8d0c7f2... by Andy Whitcroft on 2021-04-07

cranky: docs/cranking-the-kernel -- add patch application instructions

Add patch application instructions. As we now need slightly more git
configuration also add a new section on that which lists all the main
configuration you should need to change.

Signed-off-by: Andy Whitcroft <email address hidden>

feeb2e1... by Andy Whitcroft on 2021-03-31

cranky: crl/git -- ensure that the git upstream pointer is correct

When we clone/checkout/reset a branch to update it ensure we set
the upstream pointer to the upstream remote/branch so that we can
push the branch back to whence it came.

With this in place, we can add the following confiuration to our
.gitconfig file:

    [includeIf "onbranch:cranky-*/*"]
        path=~/.gitconfig--cranky

Adding this as ~/.gitconfig--cranky:

    [push]
        default=upstream

Then a simple `git push` on a checked out `cranky-prepare/*` or
`cranky-apply/*` branch will push to the branch from which `cranky
checkout` pulled it.

Signed-off-by: Andy Whitcroft <email address hidden>

c11e5b5... by Andy Whitcroft on 2021-03-31

cranky: checkout -- add remote and branch to summary

We report a lot of information about the local branch on which we are
checking out. Include the remote name and remote branch in the summary
data.

Signed-off-by: Andy Whitcroft <email address hidden>

72665b2... by Andy Whitcroft on 2021-03-31

cranky: checkout -- add prepare and apply roles

Add --prepare-kernel (alias --prepare) and --apply-patches (alias
--apply) options to indicate the role for which you want to checkout.
Use this to form the branch name on which we checkout the destination so
that we can better track where the branch should go when pushed.

Signed-off-by: Andy Whitcroft <email address hidden>

79b0fa0... by Andy Whitcroft on 2021-03-31

cranky: crl/git -- get_cranky_branch_name add role information

In preparation for splitting the crank and apply roles in `cranky
checkout` start taking the role when forming the local branch names.

Signed-off-by: Andy Whitcroft <email address hidden>

4516239... by Andy Whitcroft on 2021-03-31

cranky: crl/git -- add local_branch and use for cranky

Add local_branch property which consumes the Handle's local_branch
information and forms a `cranky/` branch from it.

Signed-off-by: Andy Whitcroft <email address hidden>

c47c0aa... by Andy Whitcroft on 2021-03-31

cranky: crl/handle -- HandleTree add local_branch

Work out the local branch name for a branch from the associated package.
Use the same mixing detection logic we use for the remote name to
prevent collisions in the cranky/ branch namespace.

Signed-off-by: Andy Whitcroft <email address hidden>

diff --git a/cranky/cmds.d/checkout b/cranky/cmds.d/checkout
index 301c5a6..6dfbfd7 100755
--- a/cranky/cmds.d/checkout
+++ b/cranky/cmds.d/checkout
@@ -14,7 +14,7 @@ from crl.handle import HandleError
 from ktl.log import cerror, cnotice
 
 
-def checkout_repos(handle, reference=None, dissociate=False):
+def checkout_repos(handle, reference=None, dissociate=False, role=None):
     """
     Clone the repos that make up the set of kernel packages
     """
@@ -47,19 +47,18 @@ def checkout_repos(handle, reference=None, dissociate=False):
         pkg        = gh.package
         remote     = gh.remote
         rmt_branch = gh.remote_branch
-        branch     = gh.get_cranky_branch_name()
+        branch     = gh.get_cranky_branch_name(role)
 
         # Add info for the summary
         summary.append({"name": pkg.name, "dir": gh.directory, "remote": remote,
-                        "branch": branch})
+                        "branch": branch, "remote-branch": rmt_branch})
 
         gh.checkout(remote, rmt_branch, branch)
 
     # Print the summary
     cnotice("Summary:")
     for s in summary:
-        cnotice("  Repo '{}' in directory '{}' (branch '{}')".format(
-                s["name"], s["dir"], s["branch"]))
+        cnotice("  Repo '{name}' in directory '{dir}' (branch '{branch}' remote '{remote} {remote-branch})".format(**s))
 
 
 def expanduser(path):
@@ -143,6 +142,16 @@ Borrow the objects from the referenced local repository only to reduce network
 traffic. This is a git clone option, check 'git help clone' for more details.
 """
 
+    help_apply = """
+Check out the application branch for this repository.
+"""
+
+    help_prepare = """
+Check out the preparation branch for this repository.
+"""
+
+    default_branch = config_cmd.get('default-branch', 'prepare')
+
     formatter = argparse.RawDescriptionHelpFormatter
     parser = argparse.ArgumentParser(description=desc,
                                      formatter_class=formatter,
@@ -155,6 +164,13 @@ traffic. This is a git clone option, check 'git help clone' for more details.
     parser.add_argument("-d", "--dissociate", action="store_true",
                         default=config_cmd.get('dissociate', False),
                         help=help_dissociate)
+    parser.set_defaults(role=default_branch)
+    parser.add_argument("--apply-patches", "--apply", dest="role", action="store_const",
+                        const="apply",
+                        help=help_apply)
+    parser.add_argument("--prepare-kernel", "--prepare", dest="role", action="store_const",
+                        const="prepare",
+                        help=help_prepare)
 
     try:
         checkout_repos(**vars(parser.parse_args()))
diff --git a/cranky/crl/git.py b/cranky/crl/git.py
index 63d55bc..4c9dd8f 100644
--- a/cranky/crl/git.py
+++ b/cranky/crl/git.py
@@ -54,6 +54,10 @@ class GitHandle():
         return self.__ht.remote
 
     @property
+    def local_branch(self):
+        return self.__ht.local_branch
+
+    @property
     def remote_branch(self):
         pkg = self.__ht.package
 
@@ -96,20 +100,11 @@ class GitHandle():
 
         return sha1
 
-    def get_cranky_branch_name(self):
+    def get_cranky_branch_name(self, role='prepare'):
         '''
         Return the local branch name which is used by cranky.
         '''
-        pkg    = self.package
-        remote = self.remote
-
-        # Cranky branch is the real branch name if we are checking out against 'origin',
-        # the derivative's source name otherwise.
-        branch_suffix = pkg.repo.branch if pkg.repo.branch else 'master'
-        if remote != 'origin':
-            branch_suffix = pkg.source.name.replace('linux-', '')
-
-        return 'cranky/' + branch_suffix
+        return 'cranky-' + role + '/' + self.local_branch
 
     def get_url(self):
         '''
@@ -167,6 +162,11 @@ class GitHandle():
             if result.returncode != 0:
                 raise GitError("unable to reset to new tip tip rc={}".format(result.returncode))
 
+        # Ensure the upstream branch reference points to where we are resrtting to.
+        result = run(["git", "branch", "-q", "--set-upstream-to=" + rmt_ref], cwd=self.directory)
+        if result.returncode != 0:
+            raise GitError("unable to set upstream to remote branch rc={}".format(result.returncode))
+
         # Validate whether the right branch is checked out.
         validate_hdl = Handle().lookup_tree(self.directory)
         if self.package != validate_hdl.package:
diff --git a/cranky/crl/handle.py b/cranky/crl/handle.py
index 9c261f3..f50fd85 100755
--- a/cranky/crl/handle.py
+++ b/cranky/crl/handle.py
@@ -153,8 +153,7 @@ class HandleTree(HandleCore):
 
         self.directory = directory
 
-    @property
-    def remote(self):
+    def _evaluate_cross(self):
         cross_series = False
         cross_source = False
         cross_type = False
@@ -187,9 +186,32 @@ class HandleTree(HandleCore):
             bits.append(primary_package.source.name.replace('linux-', ''))
         if cross_type and primary_package.type:
             bits.append(primary_package.type)
-
         remote = '-'.join(bits)
-        return remote if remote != '' else 'origin'
+        if remote == '':
+            remote = 'origin'
+
+        bits = []
+        if cross_series:
+            bits.append(self.package.series.codename)
+        if cross_source and self.package.source.name != 'linux':
+            bits.append(self.package.source.name.replace('linux-', ''))
+        if cross_type and self.package.type:
+            bits.append(self.package.type)
+        branch = '-'.join(bits)
+        if branch == '':
+            branch = 'linux'
+
+        return remote, branch
+
+    @property
+    def remote(self):
+        remote, branch = self._evaluate_cross()
+        return remote
+
+    @property
+    def local_branch(self):
+        remote, branch = self._evaluate_cross()
+        return branch
 
 
 class HandleSet(HandleCore):
diff --git a/cranky/docs/cranking-the-kernel.md b/cranky/docs/cranking-the-kernel.md
index a638619..17840b5 100644
--- a/cranky/docs/cranking-the-kernel.md
+++ b/cranky/docs/cranking-the-kernel.md
@@ -63,9 +63,40 @@ cranky chroot create-base xenial:linux
 cranky chroot create-session configs xenial:linux
 ```
 
+### Setup git configuration
+
+You should teach git your real name and email address via your
+`~/.gitconfig`:
+
+```
+git config --global user.name 'Your Name'
+git config --global user.email '[e-mail]'
+```
+
+In order for a simple `git push` to work on your `cranky` branches
+we need to teach `git` about those branches.  The easiest way to do
+this is to run the following to add appropriate configuration to your
+`~/.gitconfig`:
+
+```
+git config --file ~/.gitconfig--cranky push.default upstream
+git config --global 'includeIf.onbranch:cranky-*/*.path' '~/.gitconfig--cranky'
+```
+
+**Note**: Private kernel repositories (including security) will get
+checked out using a `git+ssh://` URL. If your local username and your
+Launchpad ID are not the same, then the following additional mapping
+is needed in your `~/.gitconfig` file:
+
+```
+git config --global \
+	'url.git+ssh://<your-launchpad-id>@git.launchpad.net/.insteadof' \
+	'git+ssh://git.launchpad.net/'
+```
+
 ## Build
 
-### Clone and checkout the kernel repository - `cranky checkout`
+### Clone and checkout the kernel repository (to prepare) - `cranky checkout`
 <!--cheatsheet-->
 ```
 cranky checkout RELEASE:KERNEL
@@ -80,16 +111,6 @@ cd canonical/kernel/ubuntu
 cranky checkout xenial:linux-oracle
 ```
 
-**Note**: Private kernel repositories (including security) will get
-checked out using a `git+ssh://` URL. If your local username and your
-Launchpad ID are not the same, then the following mapping is needed
-in your `~/.gitconfig` file:
-
-```
-[url "git+ssh://<your-launchpad-id>@git.launchpad.net/"]
-	insteadof = "git+ssh://git.launchpad.net/"
-```
-
 ### dkms package update stage - `update-version-dkms`
 <!--cheatsheet-->
 ```
@@ -516,6 +537,20 @@ $ cd linux-oracle
 $ cranky build-sources --build-opts "main:-v4.15.0-1021.23~16.04.1" --build-opts "meta:-v4.15.0.1021.15" --build-opts "signed:-v4.15.0-1021.23~16.04.1"
 ```
 
+## Patching
+
+### Clone and checkout the kernel repository (to apply patches) - `cranky checkout --apply-patches`
+<!--cheatsheet-->
+```
+cranky checkout --apply-patches RELEASE:KERNEL
+```
+<!--/cheatsheet-->
+
+Use `cranky checkout --apply-patches` to get the kernel that you want to apply patches to.
+
+Apply patches in the normal way with `git am -s`.  Once complete you should be able
+to push the updated branch to its original branch using a simple `git push`.
+
 ## Review
 
 ### Reviewing - `cranky review`