Merge lp:~hduran-8/juju-core/add_create_stream_script into lp:~go-bot/juju-core/trunk

[Horacio Durán]

Added script to create and upload a stream

The script create_stream.bash is an utility that
creates and uploads a stream to a given server, using
the current jujud build.
This is useful when trying to reproduce production
envs with a dev version, without using --upload-tools,
which produces an env closer to reality

https://codereview.appspot.com/83370043/

[Horacio Durán (hduran-8)]

Reviewers: mp+213749_code.launchpad.net,

Message:
Please take a look.

Description:
Added script to create and upload a stream

The scipt create_stream.bash is an utility that
creates and uploads a stream to a given server, using
the current jujud build.
This is useful when trying to reproduce production
envs with a dev ersion and without using --upload-tools
which gives a testing env closer to reality

https://code.launchpad.net/~hduran-8/juju-core/add_create_stream_script/+merge/213749

(do not edit description out of merge proposal)

Please review this at https://codereview.appspot.com/83370043/

Affected files (+74, -0 lines):
   A [revision details]
   A scripts/create_stream.bash

Index: [revision details]
=== added file '[revision details]'
--- [revision details] 2012-01-01 00:00:00 +0000
+++ [revision details] 2012-01-01 00:00:00 +0000
@@ -0,0 +1,2 @@
+Old revision: tarmac-20140401161742-2uwpewh7i9cnwx7n
+New revision: <email address hidden>

Index: scripts/create_stream.bash
=== added file 'scripts/create_stream.bash'
--- scripts/create_stream.bash 1970-01-01 00:00:00 +0000
+++ scripts/create_stream.bash 2014-04-02 00:20:52 +0000
@@ -0,0 +1,72 @@
+#!/bin/bash
+BUILD_FOLDER=/tmp/build_juju
+INNER_STRUCT_RELEASES=${BUILD_FOLDER}/juju-dist/tools/releases/
+INNER_STRUCT_STREAMS=${BUILD_FOLDER}/juju-dist/
+# Input arguments
+TARGET_HOST=$1
+TARGET_FOLDER=$2
+ARCH=$3
+# I asume you are uploading this to your home on a server
+: ${TARGET_FOLDER:="public_html"}
+# A decent default for the purpose of creation of this script
+: ${ARCH:="amd64"}
+
+usage() {
+ echo "usage: $0 <target_host> [target_host_folder] [arch] "
+ echo " target host will be used for scp, add user@ if required"
+ exit 1
+}
+
+
+get_version() {
+ # Defines $version. $version can be different than $RELEASE used to
+ # match the packages in the archives.
+ control_version=$1
+ version=$(echo "$control_version" |
+ sed
-n 's/^\([0-9]\+\).\([0-9]\+\).\([0-9]\+\)[-+][0-9].*/\1.\2.\3/p')
+ if [ "${version}" == "" ] ; then
+ echo "Invalid version: $control_version"
+ exit 3
+ fi
+}
+
+
+create_dir_structure() {
+ if [ -d $BUILD_FOLDER ]; then
+ rm -fr $BUILD_FOLDER;
+ fi
+ mkdir -p $INNER_STRUCT_RELEASES;
+ mkdir -p $INNER_STRUCT_STREAMS;
+
+ cd $INNER_STRUCT_RELEASES;
+
+ control_version=$(sed -n 's/^const version = "\(.*\)"/\1/p'
${GOPATH}/src/launchpad.net/juju-core/version/version.go)
+ get_version ${control_version}-0ubuntu4
+ series="precise"
+ tool="juju-${version}-${series}-${ARCH}.tgz"
+
+ if ! [ -f ${GOPATH}/bin/jujud ]; then
+ echo "Please run install on jujud, it must be in GOPATH/bin"
+ exit 1
+ fi
+ echo "Building package, contents:"
+ tar cvfz $tool -C ${GOPATH}/bin jujud
+ echo "Packed Version: "
+ ${GOPATH}/bin/jujud --version
+
+ cd $BUILD_FOLDER
+ JUJU_HOME=~/.juju PATH=$GOPATH/bin:$PATH \
+ juju metadata generate-tools -d ${INNER_STRUCT_STREAMS}
+ tar cf - juju-dist | ssh ${TARGET_HOST} tar -xf - -C ${TARGET_FOLDER}
+ ssh ${TARGET_HOST} chmod -R a+r ${TARGET_FOLDER}
+ echo "make sure...

[Dave Cheney (dave-cheney)]

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash
File scripts/create_stream.bash (right):

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode1
scripts/create_stream.bash:1: #!/bin/bash
set -e

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode2
scripts/create_stream.bash:2: BUILD_FOLDER=/tmp/build_juju
should use mktemp

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode44
scripts/create_stream.bash:44: get_version ${control_version}-0ubuntu4
will the suffix always be -0ubuntu4 ?

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode49
scripts/create_stream.bash:49: echo "Please run install on jujud, it
must be in GOPATH/bin"
why not give them the command, go install
launchpad.net/juju-core/cmd/jujud

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode55
scripts/create_stream.bash:55: ${GOPATH}/bin/jujud --version
echo "Packed version: $(${GOPATH}/bin/jujud --version)"

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode69
scripts/create_stream.bash:69: usage
what happens if the other two arguments aren't provided?

https://codereview.appspot.com/83370043/

[Horacio Durán (hduran-8)]

Please take a look.

https://codereview.appspot.com/83370043/

[lp:~hduran-8/juju-core/add_create_stream_script]

2531. By Horacio Durán on 2014-04-02

Applied corrections suggested by dfc
* Set exit on error
* Created temp folder with mktemp
* Corrected suffix generation logic
* Yield useful message on missing jujud
* Improved Packed version message

2532. By Horacio Durán on 2014-04-02

Re-worded output messages to be more useful

[Horacio Durán (hduran-8)]

Please take a look.

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash
File scripts/create_stream.bash (right):

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode1
scripts/create_stream.bash:1: #!/bin/bash
On 2014/04/02 00:42:10, dfc wrote:
> set -e

Done.

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode2
scripts/create_stream.bash:2: BUILD_FOLDER=/tmp/build_juju
On 2014/04/02 00:42:10, dfc wrote:
> should use mktemp

Done.

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode44
scripts/create_stream.bash:44: get_version ${control_version}-0ubuntu4
On 2014/04/02 00:42:10, dfc wrote:
> will the suffix always be -0ubuntu4 ?

That was trailing from the original script (this has been ripped from a
few QA scripts which created the stream from a .deb

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode44
scripts/create_stream.bash:44: get_version ${control_version}-0ubuntu4
On 2014/04/02 00:42:10, dfc wrote:
> will the suffix always be -0ubuntu4 ?

Done.

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode49
scripts/create_stream.bash:49: echo "Please run install on jujud, it
must be in GOPATH/bin"
On 2014/04/02 00:42:10, dfc wrote:
> why not give them the command, go install
launchpad.net/juju-core/cmd/jujud

Done.

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode55
scripts/create_stream.bash:55: ${GOPATH}/bin/jujud --version
On 2014/04/02 00:42:10, dfc wrote:
> echo "Packed version: $(${GOPATH}/bin/jujud --version)"

Done.

https://codereview.appspot.com/83370043/diff/1/scripts/create_stream.bash#newcode69
scripts/create_stream.bash:69: usage
On 2014/04/02 00:42:10, dfc wrote:
> what happens if the other two arguments aren't provided?

There are default values set in lines 10 and 12

https://codereview.appspot.com/83370043/

[William Reade (fwereade)]

So, --upload-tools surely sucks, and we shouldn't be using it... but I'm
not entirely convinced that what we have here is better.

NOT LGTM without a bit of discussion about what use case we're
fulfilling and whether this is really the right way to do it.

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash
File scripts/create_stream.bash (right):

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash#newcode14
scripts/create_stream.bash:14: : ${ARCH:="amd64"}
This *has* to be the arch of the local machine, doesn't it? I see no
cross-compilation...

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash#newcode16
scripts/create_stream.bash:16: : ${TARGET_SERIES:="precise"}
TARGET_SERIES *surely* ought to default to the current local series --
that's where it's been built, and where we can expect it to work.
Assuming that everything built everywhere will work on precise seems a
touch optimistic...

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash#newcode42
scripts/create_stream.bash:42: fi
I don't think it's a good idea to take the version from a source tree
that potentially bears no relation whatsoever to the binary we upload.
Any reason not to build jujud ourselves? (Please just make sure it's
properly isolated -- don't overwrite *anything* in the user's GOPATH.)

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash#newcode48
scripts/create_stream.bash:48: JUJU_HOME=~/.juju PATH=$GOPATH/bin:$PATH
\
why do we set a potentially-differing value for JUJU_HOME?

https://codereview.appspot.com/83370043/

[Curtis Hovey (sinzui)]

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash
File scripts/create_stream.bash (right):

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash#newcode14
scripts/create_stream.bash:14: : ${ARCH:="amd64"}
On 2014/04/02 06:34:58, fwereade wrote:
> This *has* to be the arch of the local machine, doesn't it? I see no
> cross-compilation...

Using local machine is better than hard-coding. Juju CI
scavenges/repurposes the jujud's built on the unit-test machines like
the ppc64el. We exec commands over ssh to get the characteristics of the
build machine. I think we get most of what we want by running "jujud
version" which yields "1.17.7-trusty-amd64" on my machine.

VERSION=$(jujud version | cut -d - -f1)
SERIES=$(jujud version | cut -d - -f2)
ARCH=$(jujud version | cut -d - -f3)

to get the version, series, and arch

https://codereview.appspot.com/83370043/diff/40001/scripts/create_stream.bash#newcode48
scripts/create_stream.bash:48: JUJU_HOME=~/.juju PATH=$GOPATH/bin:$PATH
\
On 2014/04/02 06:34:58, fwereade wrote:
> why do we set a potentially-differing value for JUJU_HOME?

JUJU_HOME isn't doing anything in this case.

This is probably a transcription error from assemble-public-tools.bash.
The script is run on systems without a proper user, without .juju/, and
sometimes without an installed juju. In these cases, the script env is
often sets JUJU_HOME to cloud-city which has a vetted environments.yaml.

https://codereview.appspot.com/83370043/

Unmerged revisions

2532. By Horacio Durán on 2014-04-02

Re-worded output messages to be more useful

2531. By Horacio Durán on 2014-04-02

Applied corrections suggested by dfc
* Set exit on error
* Created temp folder with mktemp
* Corrected suffix generation logic
* Yield useful message on missing jujud
* Improved Packed version message

2530. By Horacio Durán on 2014-04-02

Added Script that produces a stream done with the current build of jujud

=== added file 'scripts/create_stream.bash'
--- scripts/create_stream.bash	1970-01-01 00:00:00 +0000
+++ scripts/create_stream.bash	2014-04-02 01:53:20 +0000
@@ -0,0 +1,64 @@
+#!/bin/bash
+set -e
+BUILD_FOLDER=$(mktemp -d)
+INNER_STRUCT_RELEASES=${BUILD_FOLDER}/juju-dist/tools/releases/
+INNER_STRUCT_STREAMS=${BUILD_FOLDER}/juju-dist/
+# Input arguments
+TARGET_HOST=$1
+TARGET_FOLDER=$2
+ARCH=$3
+TARGET_SERIES=$4
+# I asume you are uploading this to your home on a server
+: ${TARGET_FOLDER:="public_html"}
+# A decent default for the purpose of creation of this script
+: ${ARCH:="amd64"}
+# The current default at present
+: ${TARGET_SERIES:="precise"}
+
+usage() {
+    echo "usage: $0 <target_host> [target_host_folder] [arch] [series] "
+    echo "    target host will be used for scp, add user@ if required  "
+    exit 1
+}
+
+
+create_dir_structure() {
+    if [ -d $BUILD_FOLDER ]; then
+        rm -fr $BUILD_FOLDER;
+    fi
+    mkdir -p $INNER_STRUCT_RELEASES;
+    mkdir -p $INNER_STRUCT_STREAMS;
+
+    cd $INNER_STRUCT_RELEASES;
+
+    version=$(sed -n 's/^const version = "\(.*\)"/\1/p' ${GOPATH}/src/launchpad.net/juju-core/version/version.go)
+    tool="juju-${version}-${TARGET_SERIES}-${ARCH}.tgz"
+
+    if ! [ -f ${GOPATH}/bin/jujud ]; then
+	    echo "\"jujud\" expected in $GOPATH/bin"
+	    echo "Please run:"
+	    echo "\"go install launchpad.net/juju-core/cmd/jujud\""
+	    exit 1
+    fi
+    echo "Building package, contents:"
+    tar cvfz $tool -C ${GOPATH}/bin jujud
+    echo "Packed version: $(${GOPATH}/bin/jujud --version)"
+
+    cd $BUILD_FOLDER
+    JUJU_HOME=~/.juju PATH=$GOPATH/bin:$PATH \
+    juju metadata generate-tools -d ${INNER_STRUCT_STREAMS} 
+    echo "Uploading to folder \"${TARGET_FOLDER}\" of \"${TARGET_HOST}\""
+    tar cf - juju-dist | ssh ${TARGET_HOST} tar -xvf - -C ${TARGET_FOLDER} 
+    echo "Setting remote stream permissions"
+    ssh ${TARGET_HOST} chmod -R a+r ${TARGET_FOLDER}
+    echo "Do not forget to add: "
+    echo "   tools-metadata-url:<yoururl>/<your path to juju-dist/tools>"
+    echo "   to \"~/.juju/environments.yaml\""
+
+}
+
+if ! [ "${TARGET_HOST}" ]; then
+	usage
+fi
+
+create_dir_structure