Merge lp:~marcoceppi/charm-tools/vcs into lp:~charmers/charm-tools/trunk

[Marco Ceppi]

I noticed so far two charms[1][3][2] take wild user input, supposedly a remote repository url, and takes action against that. This merge is designed to streamline the action of identifying what type of scm/vcs the remote source is and pull it down, or update a local repo from the remote source.

Demonstration of potential use cases: http://shelr.tv/records/50372c849660807bfb000098

[1]: http://jujucharms.com/charms/precise/wordpress
[2]: http://jujucharms.com/~mark-mims/oneiric/rails
[3]: http://jujucharms.com/charms/precise/node-app

[Clint Byrum (clint-fewbar)]

I can has tests?! kthxbai!

review: Needs Fixing

[lp:~marcoceppi/charm-tools/vcs]

163. By Marco Ceppi on 2012-08-28

Start of tests for vcs helper

164. By Marco Ceppi on 2012-08-28

* Now with 20% more bash
* Start of test_vcs

165. By Marco Ceppi on 2012-08-28

Deferring update_repo and fetch_repo to a later data as there is no sane way to accurately mock those setups.

[Marco Ceppi (marcoceppi)]

omg u can haz. http://i.imgur.com/9xnK8.gif

* Moved vcs.sh to vcs.bash, as it's a bash script
* Created helpers.bash and update lib.sh
* Test(s) for vcs as best as can be covered

I have no idea what would need to be done to update the charm-helpers packaging to add charm-helpers-bash, so some guidance on that if it's needed would be cool

[lp:~marcoceppi/charm-tools/vcs]

166. By Marco Ceppi on 2012-08-28

Bad logic for CH_VCS_INSTALL_DEPS

[Mark Mims (mark-mims)]

also have the summit charms... I'll link them here when I push the most current versions up to lp (they're still on oneiric).

Summit's just like rails in that it's django and pulls in the app itself from a config branch.
It _also_ pulls in theme branches too as separate config items.

[Clint Byrum (clint-fewbar)]

Thanks for adding the tests Marco, this is awesome.

Approved, with 2 minors to fix:

1) needs to be run from Makefile's "make check" rule so the packaging will be tested.

2) Just comment out the unimplemented tests with a # TODO: add tests. Printing "DEFERRED" there is really confusing.

review: Approve

[lp:~marcoceppi/charm-tools/vcs]

167. By Marco Ceppi on 2012-09-05

Updated makefile to test for bash

168. By Marco Ceppi on 2012-09-05

Hide deferred tests for now, TODO

=== modified file 'Makefile'
--- Makefile	2012-08-28 18:59:42 +0000
+++ Makefile	2012-09-05 18:33:18 +0000
@@ -26,8 +26,11 @@
 
 check:
 	tests/helpers/helpers.sh || sh -x tests/helpers/helpers.sh timeout
+	@echo Test shell helpers with dash
+	bash tests/helpers/helpers.sh || bash -x tests/helpers/helpers.sh timeout
+	tests/helpers/helpers.bash || sh -x tests/helpers/helpers.bash timeout
 	@echo Test shell helpers with bash
-	bash tests/helpers/helpers.sh || bash -x tests/helpers/helpers.sh timeout
+	bash tests/helpers/helpers.bash || bash -x tests/helpers/helpers.bash timeout
 	@echo Test charm proof
 	tests/proof/test.sh
 	tests/create/test.sh

=== added directory 'helpers/bash'
=== added file 'helpers/bash/vcs.bash'
--- helpers/bash/vcs.bash	1970-01-01 00:00:00 +0000
+++ helpers/bash/vcs.bash	2012-09-05 18:33:18 +0000
@@ -0,0 +1,243 @@
+#!/bin/bash
+
+##
+# Copyright 2012 Marco Ceppi <[e-mail]>
+#
+# This file is part of Charm Helpers.
+#
+# Charm Helpers is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# Charm Helpers is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Charm Helpers.  If not, see <http://www.gnu.org/licenses/>.
+##
+
+CH_VCS_INSTALL_DEPS=${CH_VCS_INSTALL_DEPS:-"true"}
+
+##
+# URL Match patterns
+CH_VCS_BZR_URL_PATTERN="^(lp:~?|bzr:|bzr\+ssh:)"
+CH_VCS_GIT_URL_PATTERN="(^git@|^git:|.git$)"
+CH_VCS_SVN_URL_PATTERN="^(svn:|svn\+ssh:)"
+CH_VCS_HG_URL_PATTERN="" #UGH, they only have ssh:// and https?:// SO DIFFICULT.
+# This is a VERY generous pattern and should weed out the majority of malformed URLS
+# Not sure why this didn't work. It works with `egrep -E`, but whatever.
+#CH_VCS_VALID_URL_PATTERN="(lp:|(https?|svn|bzr|git|ssh)(+ssh)?://)[-a-z0-9+&/?=_:.]*[-a-z0-9+&/=_]"
+CH_VCS_VALID_URL_PATTERN="^(lp:~?|https?://|svn://|svn\+ssh://|bzr://|bzr\+ssh://|git://|ssh://)[-a-z0-9\+&/?=_:.@]+*[-a-z0-9\+&/=_]"
+
+##
+# "clone", "branch", "fetch" commands
+CH_VCS_BZR_FETCH_CMD="branch"
+CH_VCS_GIT_FETCH_CMD="clone"
+CH_VCS_SVN_FETCH_CMD="co"
+CH_VCS_HG_FETCH_CMD="clone"
+
+##
+# "pull", "up", updated commands
+CH_VCS_BZR_UPDATE_CMD="pull"
+CH_VCS_GIT_UPDATE_CMD="pull"
+CH_VCS_SVN_UPDATE_CMD="up"
+CH_VCS_HG_UPDATE_CMD="pull"
+
+##
+# File Match patterns
+CH_VCS_BZR_FILE=".bzr"
+CH_VCS_GIT_FILE=".git"
+CH_VCS_SVN_FILE=".svn"
+CH_VCS_HG_FILE=".hg"
+
+##
+# Supported VCS
+CH_VCS_SUPPORTED_TYPES=( bzr git svn hg )
+
+##
+# Make sure all requirements are fullfiled
+if [ "$CH_VCS_INSTALL_DEPS" = "true" ]; then
+	apt-get install -y subversion git-core bzr mercurial
+fi
+
+##
+# Detect VCS
+# Try to determine the Source Control tool used
+#
+# param URL - URL of remote source
+#
+# echo type|null
+#
+# return 0 OK
+# return 1 No REPO_URL
+# return 2 Not a valid VCS
+##
+ch_detect_vcs()
+{
+	local REPO_URL=${1:-""}
+
+	if [ -z "$REPO_URL" ]; then
+		return 1
+	fi
+
+	if [ -d "$REPO_URL" ]; then
+		# This is no URL! It's a path!
+		# Go through each repo type and find out which one we have
+		for type in "${CH_VCS_SUPPORTED_TYPES[@]}"; do
+			local REPO_FILE="CH_VCS_${type^^}_FILE"
+			local REPO_FILE="${!REPO_FILE}"
+
+			if [ -d $REPO_URL/$REPO_FILE ]; then
+				echo $type
+				return 0
+			fi
+		done
+
+		# It's a directory, but we couldn't match a VCS tool. We should bail
+		# because I _doubt_ it'll be a URL.
+		return 2
+	fi
+	# So, it's not a local path? Okay, maybe it's a URL?
+
+	# This should be a loop
+	if [[ $REPO_URL =~ $CH_VCS_BZR_URL_PATTERN ]]; then
+		# It's a BZR repo!
+		echo "bzr"
+		return 0
+	elif [[ $REPO_URL =~ $CH_VCS_GIT_URL_PATTERN ]]; then
+		# It's a Git repo!
+		echo "git"
+		return 0
+	elif [[ $REPO_URL =~ $CH_VCS_SVN_URL_PATTERN ]]; then
+		# It's subversion!
+		echo "svn"
+		return 0
+	elif [[ "$REPO_URL" =~ $CH_VCS_VALID_URL_PATTERN ]]; then
+		for type in "${CH_VCS_SUPPORTED_TYPES[@]}"; do
+			local TMP_REPO=`ch_fetch_repo "$REPO_URL" "$type"`
+			local TMP_RESULT=$?
+			if [ $TMP_RESULT -eq 0 ]; then
+				# It's $type! Clean up and bail
+				rm -rf $TMP_REPO
+				echo "$type"
+				return 0
+			elif [ $TMP_RESULT -le 2 ]; then
+				# Something went wrong, and it shouldn't have.
+				return 128
+			fi
+
+			rm -rf "$TMP_REPO"
+		done
+	fi
+
+	return 2 # Not a valid VCS system
+}
+
+##
+# Fetch Repo
+# Pull down the repository and return it's temporary path
+# param REPO_URL - URL of the remote source
+# param REPO_TYPE optional - The type of VCS to use
+#
+# echo path|null
+#
+# return 0 OK
+# return 1 No REPO_URL
+# return 2 Not a valid URL
+# return 3 Unable to fetch
+##
+ch_fetch_repo()
+{
+	local REPO_URL=${1:-""}
+	local REPO_TYPE=${2:-""}
+
+	# Check for crap
+	if [ -z "$REPO_URL" ]; then
+		# Don't give me crap!
+		return 1
+	fi
+
+	# If we're guessing, then lets guess!
+	if [ -z "$REPO_TYPE" ]; then
+		# FUUUU, got to go figure it out now
+		# Assume nothing, this could be garbage.
+		if [[ $REPO_URL =~ $CH_VCS_VALID_URL_PATTERN ]]; then
+			local REPO_TYPE=`ch_detect_vcs $REPO_URL`
+
+			if [ $? -gt 0 ] || [ -z "$REPO_TYPE" ]; then
+				return 2
+			fi
+		else
+			return 2
+		fi
+	fi
+
+	local REPO_FETCH_CMD="CH_VCS_${REPO_TYPE^^}_FETCH_CMD"
+	local REPO_FETCH_CMD="${!REPO_FETCH_CMD}"
+
+	local REPO_PATH=`mktemp -d /tmp/charm-helper-vcs-$REPO_TYPE.XXXXXX`
+	local REPO_OPTS=""
+
+	# CURE YOU BZR!!!!!
+	if [ $REPO_TYPE == "bzr" ]; then
+		REPO_OPTS="--use-existing-dir"
+	fi
+		
+
+	$REPO_TYPE $REPO_FETCH_CMD $REPO_OPTS $REPO_URL $REPO_PATH > /dev/null 2>&1
+
+	if [ $? -ne 0 ]; then
+		rm -rf $REPO_PATH
+		return 3
+	fi
+
+	echo $REPO_PATH
+}
+
+##
+# Update Repository
+# This will "update" the repository to the latest rev
+#
+# param REPO_PATH - The local path to the repository
+#
+# echo null
+#
+# return 0 OK
+# return 1 No REPO_PATH
+# return 2 Not a valid repo path
+# return 3 Failed to update repo
+##
+ch_update_repo()
+{
+	local REPO_PATH=${1:-""}
+
+	if [ ! -d "$REPO_PATH" ]; then
+		return 1
+	fi
+
+	REPO_TYPE=`ch_detect_vcs "$REPO_PATH"`
+
+	if [ $? -ne 0 ]; then
+		return 3
+	fi
+
+	local REPO_UPDATE="CH_VCS_${REPO_TYPE^^}_UPDATE_CMD"
+	local REPO_UPDATE="${!REPO_UPDATE}"
+	local CWD=`pwd`
+
+	cd $REPO_PATH
+	$REPO_TYPE $REPO_UPDATE > /dev/null 2>&1
+	local SUCC=$?
+	cd $CWD
+
+	if [ $SUCC -eq 0 ]; then
+		return 0
+	else
+		return 3
+	fi
+
+	return 2
+}

=== modified file 'helpers/sh/net.sh'
--- helpers/sh/net.sh	2012-03-12 13:06:08 +0000
+++ helpers/sh/net.sh	2012-09-05 18:33:18 +0000
@@ -19,8 +19,8 @@
 # along with Charm Helpers.  If not, see <http://www.gnu.org/licenses/>.
 ##
 
-## 
-# Globally overridable settings. Make sure to set them before sourcing 
+##
+# Globally overridable settings. Make sure to set them before sourcing
 # this file.
 CH_WGET_ARGS=${CH_WGET_ARGS:-"-q --content-disposition"}
 
@@ -37,7 +37,7 @@
 {
 	local FILE=${1:-""}
 	local HASH=${2:-""}
-    local CH_DOWNLOAD_DIR=${CH_DOWNLOAD_DIR:-"`mktemp -d /tmp/ch-downloads.XXXXXX`"}
+	local CH_DOWNLOAD_DIR=${CH_DOWNLOAD_DIR:-"`mktemp -d /tmp/ch-downloads.XXXXXX`"}
 
 	if [ `ch_is_url "$FILE"` ]; then
 		wget $CH_WGET_ARGS --directory-prefix="$CH_DOWNLOAD_DIR/" "$FILE"
@@ -48,7 +48,7 @@
 		return 2
 	fi
 
-    if [ -z "$HASH" ];then
+	if [ -z "$HASH" ];then
 		#echo "Warning, no has specified. The file will be downloaded but not cryptographically checked!" > 2
 		echo "$FILE"
 		return 0
@@ -181,4 +181,3 @@
 	echo $CHECK_IP
 	return 0
 }
-

=== added file 'tests/helpers/helpers.bash'
--- tests/helpers/helpers.bash	1970-01-01 00:00:00 +0000
+++ tests/helpers/helpers.bash	2012-09-05 18:33:18 +0000
@@ -0,0 +1,18 @@
+#!/bin/bash
+
+set -ue
+
+test_home=`dirname $0`
+test_home=`readlink -f $test_home`
+
+# Should set 60 second timeout
+if [ -z ${1:-""} ] ; then
+    exec $test_home/run_with_timeout.py $0 timeout
+fi
+
+[ "${LIB_SOURCED:-''}" = "1" ] || . $test_home/lib.sh
+
+for i in $test_home/test_*.bash ; do
+  . $i
+done
+

=== modified file 'tests/helpers/lib.sh'
--- tests/helpers/lib.sh	2012-03-12 13:06:08 +0000
+++ tests/helpers/lib.sh	2012-09-05 18:33:18 +0000
@@ -10,6 +10,8 @@
     echo -n `date`: Testing $*
 }
 
-HELPERS_HOME=${HELPERS_HOME:-"$test_home/../../helpers/sh"}
+shell_type=`echo $0 | awk -F. '{print $NF}'`
+
+HELPERS_HOME=${HELPERS_HOME:-"$test_home/../../helpers/$shell_type"}
 
 LIB_SOURCED=1

=== added file 'tests/helpers/test_vcs.bash'
--- tests/helpers/test_vcs.bash	1970-01-01 00:00:00 +0000
+++ tests/helpers/test_vcs.bash	2012-09-05 18:33:18 +0000
@@ -0,0 +1,140 @@
+#!/bin/bash
+
+if [ -z "$test_home" ] ; then
+    test_home=`dirname $0`
+    test_home=`readlink -f $test_home`
+fi
+
+[ "$LIB_SOURCED" = "1" ] || . $test_home/lib.sh
+
+set -ue
+
+MOCK_NET=${MOCK_NET:-"true"}
+MOCK_VCS=${MOCK_VCS:-"true"}
+
+mock_host()
+{
+    if echo $* | grep -q "[_#]" ; then return 1 ; fi
+    echo nothost.local has address 127.0.0.1
+    echo nothost.local has address 127.0.1.1
+}
+
+mock_dig()
+{
+    if echo $* | grep -q "[_#]" ; then return 1 ; fi
+    echo 127.0.0.1
+    echo 127.0.1.1
+}
+
+mock_git()
+{
+	if [ "$1" = "clone" ] && [ $# -eq 3 ]; then
+		return 0
+	fi
+
+	if [ "$1" = "pull" ] && [ $# -eq 1 ]; then
+		return 0
+	fi
+
+	return 1
+}
+
+mock_bzr()
+{
+	if [ "$1" = "branch" ] && [ $# -eq 3 ]; then
+		return 0
+	fi
+
+	if [ "$1" = "pull" ] && [ $# -eq 1 ]; then
+		return 0
+	fi
+
+	return 1
+}
+
+mock_svn()
+{
+	if [ "$1" = "co" ] && [ $# -eq 4 ]; then
+		return 0
+	fi
+
+	if [ "$1" = "up" ] && [ $# -eq 1 ]; then
+		return 0
+	fi
+
+	return 1
+}
+
+mock_apt()
+{
+	return 0
+}
+
+if [ "$MOCK_NET" = "true" ]; then
+    alias host=mock_host
+    alias dig=mock_dig
+fi
+
+## At this time, this doesn't work, but one day it will - or at least should
+## Assuming that ch_detect_vcs works, ch_fetch_repo and ch_update_repo
+## should run without issue.
+if [ "$MOCK_VCS" = "true" ]; then
+	alias git=mock_git
+	alias bzr=mock_bzr
+	alias svn=mock_svn
+	alias hg=mock_git
+	alias apt-get=mock_apt
+fi
+
+CH_VCS_INSTALL_DEPS="false"
+
+. $HELPERS_HOME/vcs.bash
+
+cleanup_repos()
+{
+	output Cleaning up fake repos...
+	rm -rf /tmp/charm-helper-vcs-*
+}
+
+trap cleanup_repos EXIT
+
+output Creating fake local repos...
+temp_no_vcs=`mktemp -d /tmp/charm-helper-vcs-lol.XXXXXX`
+temp_git_dir=`mktemp -d /tmp/charm-helper-vcs-git.XXXXXX`
+temp_bzr_dir=`mktemp -d /tmp/charm-helper-vcs-bzr.XXXXXX`
+temp_svn_dir=`mktemp -d /tmp/charm-helper-vcs-svn.XXXXXX`
+temp_hg_dir=`mktemp -d /tmp/charm-helper-vcs-hg.XXXXXX`
+
+mkdir -p $temp_git_dir/.git
+mkdir -p $temp_bzr_dir/.bzr
+mkdir -p $temp_svn_dir/.svn
+mkdir -p $temp_hg_dir/.hg
+
+start_test ch_detect_vcs...
+ch_detect_vcs "notvalid" && return 1 || :
+[ "`ch_detect_vcs 'lp:charm-tools'`" = "bzr" ]
+[ "`ch_detect_vcs 'lp:~charmers/charm-tools/trunk'`" = "bzr" ]
+[ "`ch_detect_vcs 'bzr+ssh://example.com/~charmers/charm-tools/trunk'`" = "bzr" ]
+[ "`ch_detect_vcs 'bzr+ssh://example.com/~charmers/charm-tools/trunk'`" = "bzr" ]
+[ "`ch_detect_vcs $temp_bzr_dir`" = "bzr" ]
+[ "`ch_detect_vcs 'https://example.com/charm-tools.git'`" = "git" ]
+[ "`ch_detect_vcs 'http://example.com/charm-tools.git'`" = "git" ]
+[ "`ch_detect_vcs 'git://example.com/charm-tools.git'`" = "git" ]
+[ "`ch_detect_vcs '[e-mail]/charm-tools.git'`" = "git" ]
+[ "`ch_detect_vcs $temp_git_dir`" = "git" ]
+[ "`ch_detect_vcs 'svn+ssh://example.com/charm-tools/trunk'`" = "svn" ]
+[ "`ch_detect_vcs 'svn://example.com/charm-tools/trunk'`" = "svn" ]
+[ "`ch_detect_vcs $temp_svn_dir`" = "svn" ]
+[ "`ch_detect_vcs $temp_hg_dir`" = "hg" ]
+echo PASS
+
+# TODO
+#start_test ch_fetch_repo...
+#echo DEFERRED
+
+# TODO
+#start_test ch_update_repo...
+#echo DEFERRED
+
+trap - EXIT
+cleanup_repos