Merge lp:~matsubara/auto-package-testing/setup-hook into lp:auto-package-testing

[Diogo Matsubara]

Add a --post-init-script option to run-adt-test so a script can be run after the testbed is initialised. This is required for MAAS, so we can do additional configuration of the testbed network config.

[lp:~matsubara/auto-package-testing/setup-hook]

146. By Diogo Matsubara on 2013-01-23

fix initialised spelling

147. By Diogo Matsubara on 2013-01-25

merge from trunk

148. By Diogo Matsubara on 2013-01-28

inherit env vars from parent shell

149. By Diogo Matsubara on 2013-01-28

add quotes to check if there's a post initi script

150. By Diogo Matsubara on 2013-01-28

remove ./ from post init script

[Martin Pitt (pitti)]

Why does the script get run from $BINDIR? I would expect the --post-init-script argument to be a valid relative or absolute path already ?

review: Needs Information

[lp:~matsubara/auto-package-testing/setup-hook]

151. By Diogo Matsubara on 2013-01-31

remove BINDIR from POST_INIT_SCRIPT variable.

[Diogo Matsubara (matsubara)]

> Why does the script get run from $BINDIR? I would expect the --post-init-
> script argument to be a valid relative or absolute path already ?

Yep, makes sense. I removed the $BINDIR from the script call.

[Martin Pitt (pitti)]

This still needs some quoting around the variable, and the line break in the --help output looks a bit ugly, but that's easy to fix during merge.

My other question is, how is this going to be used on Jenkins? We will most certainly not have a separate mapping of package names to "post-init" scripts in Jenkins. We only test source packages in Ubuntu, so you can always get to debian/tests/ of e. g. maas; and from then on, there is nothing that such a post-init script can do which debian/tests/mytest can't -- so why not put the initialization into the actual test instead? (This might need an advisory "Restrictions: breaks-testbed")

review: Needs Information

[lp:~matsubara/auto-package-testing/setup-hook]

152. By Diogo Matsubara on 2013-02-01

fix line break in --help

[Diogo Matsubara (matsubara)]

Hi Martin, thanks for the review.

I fixed the line break in the --help. The POST_INIT_SCRIPT is quoted on line 37. Do you mean line 38 should be quoted as well or something else?

post-init refers to the testbed initialisation. For example, the post-init-script I'm running on MAAS[1][2] copies some files/scripts to the testbed, restart services and config the proxy. That way, when the package installation through adt-run starts, I have all my network interfaces configured and apt can take advantage of the proxy.

Doing so inside debian/tests/mytests means I'd have the maas package installed without using the proxy and without the proper network interfaces configured. Also, on MAAS, we're using this for integration tests, and this reproduces more closely what users would be actually doing when installing the maas package.

I understand this patch and the others I've been proposing are very specific for our use case, so I'm happy to make changes to this and the other patches to make them more generally useful (or at least don't get in the way of how auto-package-testing is designed to be used). What do you think?

[1] https://pastebin.canonical.com/83602/
[2] https://pastebin.canonical.com/83606/

[Diogo Matsubara (matsubara)]

Another thing that occurred to me that this patch would help solve (but we're not using atm on MAAS but I think we'll soon) is to copy non-ubuntu packages (e.g. pypi packages) to the testbed and have them installed before the adt tests start.
Our QA lab is a restricted environment and can't access the internet to download packages directly from pypi. Keeping those non-ubuntu dependencies in a local place that can be uploaded to the testbed would be useful for us.

[Martin Pitt (pitti)]

> I fixed the line break in the --help. The POST_INIT_SCRIPT is quoted on line
> 37. Do you mean line 38 should be quoted as well or something else?

Lines 28 and 39, yes.

> post-init refers to the testbed initialisation. For example, the post-init-
> script I'm running on MAAS[1][2] copies some files/scripts to the testbed,
> restart services and config the proxy. That way, when the package installation
> through adt-run starts, I have all my network interfaces configured and apt
> can take advantage of the proxy.

Ah, so the maas package requires you to set things up even before apt installs the packagage? That sounds a little weird, it means you could never install the package by default?

> Doing so inside debian/tests/mytests means I'd have the maas package installed
> without using the proxy and without the proper network interfaces configured.

Not necessarily. If you cannot configure the maas package after installation, you can always drop it from the test dependencies and install it from within your tests. So again, debian/tests/ can do anything such a post-install script can do, with the added benefits that (1) you keep all the knowledge in one place, and (2) you will actually be able to run this in our QA lab at https://jenkins.qa.ubuntu.com/view/Raring/view/AutoPkgTest/ and also make stuff reproducible for users.

[lp:~matsubara/auto-package-testing/setup-hook]

153. By Diogo Matsubara on 2013-02-01

quoted POST_INIT_SCRIPT var

[Diogo Matsubara (matsubara)]

> > I fixed the line break in the --help. The POST_INIT_SCRIPT is quoted on line
> > 37. Do you mean line 38 should be quoted as well or something else?
>
> Lines 28 and 39, yes.

Quoted.

>
> > post-init refers to the testbed initialisation. For example, the post-init-
> > script I'm running on MAAS[1][2] copies some files/scripts to the testbed,
> > restart services and config the proxy. That way, when the package
> installation
> > through adt-run starts, I have all my network interfaces configured and apt
> > can take advantage of the proxy.
>
> Ah, so the maas package requires you to set things up even before apt installs
> the packagage? That sounds a little weird, it means you could never install
> the package by default?

Not the maas package itself, but rather the QA lab and its VMs. If we use the defaults provided by the package, it wouldn't work in the MAAS QA lab.

>
> > Doing so inside debian/tests/mytests means I'd have the maas package
> installed
> > without using the proxy and without the proper network interfaces
> configured.
>
> Not necessarily. If you cannot configure the maas package after installation,
> you can always drop it from the test dependencies and install it from within
> your tests. So again, debian/tests/ can do anything such a post-install script
> can do, with the added benefits that (1) you keep all the knowledge in one
> place, and (2) you will actually be able to run this in our QA lab at
> https://jenkins.qa.ubuntu.com/view/Raring/view/AutoPkgTest/ and also make
> stuff reproducible for users.

I'd rather not drop maas from the test dependencies. Installing it through adt is one of the things I really want to keep to re-use the infrastructure as much as possible. We can't use the Ubuntu QA lab as we want the MAAS nodes to use bare metal rather than VMs. The VMs are only for the region and cluster controllers. That's what make our use case trickier than other packages.

[Martin Pitt (pitti)]

Hello Diogo,

Diogo Matsubara [2013-02-01 16:55 -0000]:
> Not the maas package itself, but rather the QA lab and its VMs. If
> we use the defaults provided by the package, it wouldn't work in the
> MAAS QA lab.

So it sounds like it would never work anyway in the QA lab or in
developer provided VMs? Would it perhaps be possible to drop the
XS-Testsuite: field from debian/control, so that we stop picking it up
in Jenkins and thus cause an eternal failure?

> I'd rather not drop maas from the test dependencies. Installing it
> through adt is one of the things I really want to keep to re-use the
> infrastructure as much as possible.

It's not really that different, it's just calling apt-get install
either way.

> We can't use the Ubuntu QA lab as we want the MAAS nodes to use bare
> metal rather than VMs.

OK, thanks for the heads-up. Out of interest, what doesn't work in
VMs? AFAIK they can PXE boot just as well as real iron?

[Diogo Matsubara (matsubara)]

> Hello Diogo,
>
> Diogo Matsubara [2013-02-01 16:55 -0000]:
> > Not the maas package itself, but rather the QA lab and its VMs. If
> > we use the defaults provided by the package, it wouldn't work in the
> > MAAS QA lab.
>
> So it sounds like it would never work anyway in the QA lab or in
> developer provided VMs? Would it perhaps be possible to drop the
> XS-Testsuite: field from debian/control, so that we stop picking it up
> in Jenkins and thus cause an eternal failure?

Yes, I'll remove that and the tests from the sourcepackage since we're downloading the QA lab tests during the test run.

>
> > I'd rather not drop maas from the test dependencies. Installing it
> > through adt is one of the things I really want to keep to re-use the
> > infrastructure as much as possible.
>
> It's not really that different, it's just calling apt-get install
> either way.
>
> > We can't use the Ubuntu QA lab as we want the MAAS nodes to use bare
> > metal rather than VMs.
>
> OK, thanks for the heads-up. Out of interest, what doesn't work in
> VMs? AFAIK they can PXE boot just as well as real iron?

It works with VMs but we want to test with bare metal, since MAAS is about managing bare metal.
The lab was built for this.

Unmerged revisions

153. By Diogo Matsubara on 2013-02-01

quoted POST_INIT_SCRIPT var

152. By Diogo Matsubara on 2013-02-01

fix line break in --help

151. By Diogo Matsubara on 2013-01-31

remove BINDIR from POST_INIT_SCRIPT variable.

150. By Diogo Matsubara on 2013-01-28

remove ./ from post init script

149. By Diogo Matsubara on 2013-01-28

add quotes to check if there's a post initi script

148. By Diogo Matsubara on 2013-01-28

inherit env vars from parent shell

147. By Diogo Matsubara on 2013-01-25

merge from trunk

146. By Diogo Matsubara on 2013-01-23

fix initialised spelling

145. By Diogo Matsubara on 2013-01-23

add a --post-init-script option to run-adt-test so a script can be run once the testbed is initialised.

=== modified file 'bin/run-adt-test'
--- bin/run-adt-test	2013-01-23 17:03:02 +0000
+++ bin/run-adt-test	2013-02-01 16:54:21 +0000
@@ -116,6 +116,9 @@
   -p,--ppa     Include a PPA to search packages
   -b,--bzr     Use bazaar branch BRANCH. It cannot be used
                simultaneouslty with -S (Deprecated, use -S instead)
+  -i,--post-init-script
+               Use this option to run a script after the testbed is
+               initialised.
 
 
 EOF
@@ -123,7 +126,7 @@
 }
 
 
-TEMP=$(getopt -o ha:b:r:p:PdklsS: --long help,arch:,bzr:,release:,ppa:,proposed,debug,keep,login,use-shm,source: -- "$@")
+TEMP=$(getopt -o ha:b:r:p:PdklsS:i: --long help,arch:,bzr:,release:,ppa:,proposed,debug,keep,login,use-shm,source:,post-init-script: -- "$@")
 eval set -- "$TEMP"
 
 exec 2>&1
@@ -181,6 +184,9 @@
             SRC_LOCATION=$2
             RUN_OPTS="$RUN_OPTS -S \"$SRC_LOCATION\""
             shift 2;;
+        -i|--post-init-script)
+            POST_INIT_SCRIPT=$2
+            shift 2;;
         --) shift ; break ;;
         *) usage;;
     esac
@@ -274,6 +280,11 @@
     LOOP=$(( LOOP + 1 ))
 done
 
+# Run a post init script once the testbed is initialised.
+if [ -n "$POST_INIT_SCRIPT" ]; then
+    . "$POST_INIT_SCRIPT"
+fi
+
 # Upload local source directory to sandbox
 if [ "$( echo $SRC_LOCATION | cut -c-7)" = "file://" ]; then
     LOCALDIR="$(echo $SRC_LOCATION | cut -c8-)"