Merge lp:~cprov/uci-engine/webui-static-assets into lp:uci-engine

[Celso Providelo]

Modifying bin/webui-collectstatic.py to collect static files also from external django applications (primarily 'admin').

Modifying bin/webui-collectstatic.py to collect static files also from external django applications (primarily 'admin').

From now on, building the webui charm requires development venv to be activated (for django dependencies).

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:776
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1367/
Executed test runs:

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/uci-engine-ci/1367/rebuild

review: Approve (continuous-integration)

[Joe Talbott (joetalbott)]

review: Approve

[Ubuntu CI Bot (uci-bot)]

The attempt to merge lp:~cprov/uci-engine/webui-static-assets into lp:uci-engine failed. Below is the output from the failed tests.

Running cm...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
uploading webui-content.tgz to swift
Updating source dependencies...
Updating source dependencies...
Updating source dependencies...
Checking juju status
Private PPAs: disabled
Preparing local branch upload...
Uploading local branch, fingerprint e413e8a2b1d087ec9587ada3d08259791b1bd9dc
Building charm: lander
Building charm: wsgi-app
Building charm: rabbitmq-worker
Building charm: webui
Building charm: key-secret-subordinate
Building charm: system-image-server
Building charm: chroot-builder
Installing keys from bzr+ssh://bazaar.launchpad.net/~ci-engineering-private/+junk/ci-airline-dev-keys/
Running juju-deployer -v -c /tmp/tmpXIEYkW/deployer/branch-source-builder.yaml -c /tmp/tmpXIEYkW/deployer/gatekeeper.yaml -c /tmp/tmpXIEYkW/deployer/image-builder.yaml -c /tmp/tmpXIEYkW/deployer/lander.yaml -c /tmp/tmpXIEYkW/deployer/nf-stats-service.yaml -c /tmp/tmpXIEYkW/deployer/ppa-creator.yaml -c /tmp/tmpXIEYkW/deployer/publisher.yaml -c /tmp/tmpXIEYkW/deployer/relations.yaml -c /tmp/tmpXIEYkW/deployer/test-runner.yaml -c /tmp/tmpXIEYkW/deployer/ticket-system.yaml -c /tmp/tmpXIEYkW/deployer/validator.yaml -c /tmp/tmpXIEYkW/deployer/webui.yaml ci-airline
Tests running...
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testConnectFailed ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testProcessQueue ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testRunForever ... OK (0.101 secs)
ci-utils.ci_utils.tests.test_amqp.TestAMQP.testSent ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp.TestProgressTrigger.testProgress ... OK (0.001 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testCancel ... OK (0.103 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoQueue ... OK (0.004 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testNoTicket ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageCalledProcessError ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageFail ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageKilled ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageSimple ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testOnMessageUnexpected ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestAMQPWorker.testSaveLastRun ... OK (0.002 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCBRuns ... OK (0.021 secs)
ci-utils.ci_utils.tests.test_amqp_worker.TestTimer.testCanCancel ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_invalid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreConfig.test_valid_auth_config ... OK (0.000 secs)
ci-utils.ci_utils.tests.test_data_store.TestDataStoreFileName.test_get_file_name ... OK (0.000 secs)
ci-utils.ci_utils.tes...

[lp:~cprov/uci-engine/webui-static-assets]

777. By Celso Providelo on 2014-09-08

Test fixing and more informative errors.

=== modified file 'bin/webui-collectstatic.py'
--- bin/webui-collectstatic.py	2014-05-21 20:56:42 +0000
+++ bin/webui-collectstatic.py	2014-09-08 20:26:21 +0000
@@ -14,49 +14,52 @@
 # You should have received a copy of the GNU Affero General Public License
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
-import fnmatch
+import atexit
 import logging
 import os
 import subprocess
+import shutil
+import sys
 import tempfile
 
-from distutils import dir_util
+try:
+    from webui import settings
+except ImportError:
+    logging.error(
+        'Collect static files requires access to the "webui" module. '
+        ' Please run this script inside the project venv.')
+    sys.exit(1)
 
 
 def main():
-    logging.info("Searching for static directories...")
-    # Directory tree traversal code
-    # taken from http://stackoverflow.com/a/2186565/729970
-    matches = []
-    working_dir = os.path.abspath(os.path.dirname(os.path.abspath(__file__)))
-    path = os.path.abspath(os.path.join(working_dir, '..', 'webui'))
-    for root, dirnames, filenames in os.walk(path):
-        for dirname in fnmatch.filter(dirnames, 'static'):
-            # skip the top level static dir since where django collects static
-            # files.
-            skip_path = os.path.join(path, 'static')
-            dir_path = os.path.join(root, dirname)
-
-            if dir_path == skip_path:
-                continue
-
-            matches.append(dir_path)
-
-    logging.info("matches: %s", matches)
-
     tmp = tempfile.mkdtemp()
-    tmp_path = os.path.join(tmp, 'static')
-    for dirname in matches:
-        dir_util.copy_tree(dirname, tmp_path, update=True)
-
-    tarfile = '{}/webui_content.tgz'.format(tmp)
-    subprocess.check_output(['tar', 'czvf', tarfile, '-C', tmp, 'static'])
-
-    # Change to the temporary directory so swift doesn't add path bits to the
-    # filename
-    os.chdir(tmp)
-    subprocess.check_output(
-        ['swift', 'upload', 'charmblobs', os.path.basename(tarfile)]
+    atexit.register(shutil.rmtree, tmp)
+
+    logging.info("Collecting static assets in {}".format(tmp))
+
+    bin_dir = os.path.dirname(os.path.abspath(__file__))
+    working_dir = os.path.abspath(os.path.join(bin_dir, os.pardir))
+    manager_path = os.path.join(working_dir, 'webui', 'manage.py')
+
+    # Pass explict --settings using shell because 'DJANGO_SETTINGS_MODULE'
+    # variable might be populated by other django project (ticket-system).
+    subprocess.check_output(
+        '{} collectstatic --noinput --settings=webui.settings'.format(
+            manager_path),
+        cwd=tmp, shell=True,
+    )
+
+    tarfile = 'webui_content.tgz'
+    logging.info("Building {}".format(tarfile))
+    subprocess.check_output(
+        ['tar', 'czvf', tarfile, settings.STATIC_ROOT],
+        cwd=tmp,
+    )
+
+    logging.info("Uploading {} to charmblobs".format(tarfile))
+    subprocess.check_output(
+        ['swift', 'upload', 'charmblobs', os.path.basename(tarfile)],
+        cwd=tmp,
     )