Merge lp:~mwhudson/ubuntu/xenial/juju/use-dh-golang-proper into lp:~juju-qa/ubuntu/xenial/juju/xenial-2.0-beta6

[Michael Hudson-Doyle]

Convert to using dh-golang more conventionally.

As observed in https://bugs.launchpad.net/ubuntu/+source/juju-core/+bug/1578550 juju-core ftbfs in yakkety due to assumptions in dh_golang. This branch, therefore, arranges that the build tree looks much more like what dh_golang expects, via a terrifying perl script.

[lp:~mwhudson/ubuntu/xenial/juju/use-dh-golang-proper]

121. By Michael Hudson-Doyle on 2016-05-06

Use dh_golang to build the package.

This fixes the ftbfs on yakkety.

dh_golang's "configure" step is replaced by a completely out of control perl script.

[Michael Hudson-Doyle (mwhudson)]

It does seem to work, for all the perl: https://launchpad.net/~mwhudson/+archive/ubuntu/devirt/+packages?field.name_filter=juju-core

[Michael Hudson-Doyle (mwhudson)]

One last comment for now: I could rewrite the perl script in python. I'm not sure how to do the Dpkg::Deps stuff in Python, I could always shell out to a perl one-liner from Python :-)

[Martin Packman (gz)]

Thanks Michael!

Some specific questions as inline comments, but the big thing to answer is do we want a juju-specific answer here or something generic that this distro can use for other go projects?

The current script is more terrifying than I imagined when we talked about the solution, but does seem to cover the edge cases well enough to be nearly reusable elsewhere. I think we could simplify it a fair by using dependencies.tsv from the github.com/juju/juju tree if we decide this is going to remain a juju-only packaging solution.

I think deb822 in python-debian does the dependency parsing bits we'd need, but I can live with perl.

Anyway, I'm down with trying this approach for juju out in yakkety, but think we need to point in out and ask for feedback from a wider audience of people who may care about how we do golang packaging.

review: Needs Information

[lp:~mwhudson/ubuntu/xenial/juju/use-dh-golang-proper]

122. By Michael Hudson-Doyle on 2016-05-09

use dependencies.tsv instead

123. By Michael Hudson-Doyle on 2016-05-09

review comments

124. By Michael Hudson-Doyle on 2016-05-09

kill the perl

125. By Michael Hudson-Doyle on 2016-05-09

fix so it actually works

[Michael Hudson-Doyle (mwhudson)]

> Thanks Michael!
>
> Some specific questions as inline comments, but the big thing to answer is do
> we want a juju-specific answer here or something generic that this distro can
> use for other go projects?
>
> The current script is more terrifying than I imagined when we talked about the
> solution,

:-)

> but does seem to cover the edge cases well enough to be nearly
> reusable elsewhere. I think we could simplify it a fair by using
> dependencies.tsv from the github.com/juju/juju tree if we decide this is going
> to remain a juju-only packaging solution.

Oh man, I forgot about that. That makes things so much easier that I've replaced that perl script with a simple Python script instead. (I can always get the perl out of branch history if I want it again).

My new Python has the issue that having extra packages installed might affect the build, but that's not new.

> I think deb822 in python-debian does the dependency parsing bits we'd need,
> but I can live with perl.
>
> Anyway, I'm down with trying this approach for juju out in yakkety, but think
> we need to point in out and ask for feedback from a wider audience of people
> who may care about how we do golang packaging.

I'll ask in Debian what they think about the general problem (although they'll probably just say "don't bundle dependencies"...)

[lp:~mwhudson/ubuntu/xenial/juju/use-dh-golang-proper]

126. By Michael Hudson-Doyle on 2016-05-09

add comment

127. By Michael Hudson-Doyle on 2016-05-09

edit comment

128. By Michael Hudson-Doyle on 2016-05-09

finally tidying of comment

129. By Michael Hudson-Doyle on 2016-05-09

revert changelog, set DH_GOPKG as well for easier backporting to trusty

[Martin Packman (gz)]

Thanks for addressing all the comments! Lets go with this and do a new upload for yakkety.

review: Approve

=== modified file 'debian/control'
--- debian/control	2016-04-20 17:22:29 +0000
+++ debian/control	2016-05-09 04:09:18 +0000
@@ -23,6 +23,7 @@
                python
 Standards-Version: 3.9.7
 Homepage: http://launchpad.net/juju-core
+Xs-Go-Import-Path: github.com/juju/juju
 Testsuite: autopkgtest
 
 Package: juju-2.0

=== added directory 'debian/helpers'
=== added file 'debian/helpers/setup-build-directory.py'
--- debian/helpers/setup-build-directory.py	1970-01-01 00:00:00 +0000
+++ debian/helpers/setup-build-directory.py	2016-05-09 04:09:18 +0000
@@ -0,0 +1,42 @@
+#!/usr/bin/python
+
+# This script sets up the _build directory to be suitable to have
+# GOPATH pointed at it. The output looks much like what:
+#
+#     dh_auto_configure --buildsystem=golang --builddirectory=golang
+#
+# would produce for a normal package build despite the fact that this
+# package is not at all out laid in a fashion dh_golang thinks of as
+# normal.
+#
+# Dependencies installed from .deb packages are used in favour of
+# packages from ./src.
+
+import csv
+import errno
+import os
+import shutil
+
+def ensure_directory(path):
+    try:
+        os.makedirs(path)
+    except OSError as exception:
+        if exception.errno != errno.EEXIST:
+            raise
+
+reader = csv.reader(open("src/github.com/juju/juju/dependencies.tsv"), delimiter="\t")
+
+def link_pkg(pkgpath):
+    target = os.path.join("_build/src", pkgpath)
+    ensure_directory(os.path.dirname(target))
+    for prefix in ['/usr/share/gocode/src', 'src']:
+        srcpath = os.path.abspath(os.path.join(prefix, pkgpath))
+        if os.path.exists(srcpath):
+            os.symlink(srcpath, target)
+            return
+    raise Exception("%s not found", pkgpath)
+
+for record in reader:
+    link_pkg(record[0])
+
+shutil.copytree("src/github.com/juju/juju", "_build/src/github.com/juju/juju", symlinks=True)

=== modified file 'debian/rules'
--- debian/rules	2016-04-19 09:21:40 +0000
+++ debian/rules	2016-05-09 04:09:18 +0000
@@ -1,14 +1,11 @@
 #!/usr/bin/make -f
 # -*- makefile -*-
 
-# Uncomment this to turn on verbose mode.
 #export DH_VERBOSE=1
-# NOTE: debian/gocode/src will contain symlinks to system provided
-#       dev packages, which will be used over those in the upstream
-#       juju-core codebase.
 
-export GOPATH:=$(CURDIR)/debian/gocode:$(CURDIR)
-export PATH:=$(CURDIR)/bin:$(PATH)
+export DH_GOPKG:=github.com/juju/juju
+export GOPATH:=$(CURDIR)/_build
+export PATH:=$(CURDIR)/_build/bin:$(PATH)
 
 PKGDIR:=debian/juju
 VERSION:=$(shell sed -n 's/^const version = "\(2\.[0-9]\+\)[\.-].*"/\1/p' $(CURDIR)/src/github.com/juju/juju/version/version.go)
@@ -23,9 +20,9 @@
 endif
 
 %:
-	dh $@ --with=golang
+	dh $@ --with=golang --buildsystem=golang --builddirectory=_build
 
-COMMON_FLAGS:= -x -v -work
+COMMON_FLAGS:= -x -v
 golang_archs:= amd64 i386 armhf s390x ppc64el arm64
 ifeq (,$(filter $(DEB_HOST_ARCH), $(golang_archs)))
 # NOTE(james-page) statically link libgo for the jujud binary for gccgo
@@ -37,28 +34,32 @@
 # NOTE: ensure /usr/share/gocode is use in preference to any
 #       embedded source for dependencies.
 override_dh_auto_configure:
-	dh_auto_configure
-	mkdir -p debian/gocode/src
-	ln -s /usr/share/gocode/src/* debian/gocode/src
+	./debian/helpers/setup-build-directory.py
+
+override_dh_auto_build:
+	dh_auto_build -- $(COMMON_FLAGS)
+	rm _build/bin/jujud
+	# re-link jujud with specific options
+	go install $(COMMON_FLAGS) $(JUJUD_FLAGS) github.com/juju/juju/cmd/jujud
+
+# Don't run the tests -- the juju unit tests are too heavyweight to run during
+# package build.
+override_dh_auto_test:
+	:
 
 override_dh_auto_install:
-	go install $(COMMON_FLAGS) github.com/juju/juju/cmd/juju
-	go install $(COMMON_FLAGS) github.com/juju/juju/cmd/plugins/juju-metadata
-	go install $(COMMON_FLAGS) github.com/juju/juju/cmd/plugins/juju-upgrade-mongo
-	go install $(COMMON_FLAGS) $(JUJUD_FLAGS) github.com/juju/juju/cmd/jujud
-	echo '#!/bin/sh\nexport PATH=/usr/lib/juju-$(VERSION)/bin:"$$PATH"\nexec juju "$$@"' > bin/juju-$(VERSION)
-	chmod 755 bin/juju-$(VERSION)
+	echo '#!/bin/sh\nexport PATH=/usr/lib/juju-$(VERSION)/bin:"$$PATH"\nexec juju "$$@"' > _build/bin/juju-$(VERSION)
+	chmod 755 _build/bin/juju-$(VERSION)
 	mkdir -p debian/home
 	HOME=debian/home $(CURDIR)/src/github.com/juju/juju/scripts/generate-docs.py man -o juju.1
-	dh_install bin/juju usr/lib/juju-$(VERSION)/bin
-	dh_install bin/juju-metadata usr/lib/juju-$(VERSION)/bin
-	dh_install bin/juju-upgrade-mongo usr/lib/juju-$(VERSION)/bin
-	dh_install bin/jujud usr/lib/juju-$(VERSION)/bin
-	dh_install bin/juju-$(VERSION) usr/bin
+	dh_install _build/bin/juju usr/lib/juju-$(VERSION)/bin
+	dh_install _build/bin/juju-metadata usr/lib/juju-$(VERSION)/bin
+	dh_install _build/bin/juju-upgrade-mongo usr/lib/juju-$(VERSION)/bin
+	dh_install _build/bin/jujud usr/lib/juju-$(VERSION)/bin
+	dh_install _build/bin/juju-$(VERSION) usr/bin
 	dh_install juju.1 usr/lib/juju-$(VERSION)/man/man1
 	dh_install src/github.com/juju/juju/etc/bash_completion.d/juju2 \
 	           usr/share/bash-completion/completions
-	dh_auto_install
 
 override_dh_link:
 	dh_link -pjuju-2.0 usr/bin/juju-$(VERSION) usr/bin/juju