Merge lp:~jtv/maas-test/var-cache into lp:maas-test

[Jeroen T. Vermeulen]

Move ~/.config/maas-test into /var/cache/.

As agreed with Julian. This is not the complete change: we also agreed to move the pidfiles for polipo into /run. It all follows from the fact that we now run maas-test in its entirety as root, instead of sudo'ing all over the place: maas-test's state is now effectively global system state. This also resolves the nastiness of having ~/.config/maas-test, in your own home directory, owned by root. As an added bonus, we no longer run the risk of test runs accidentally creating the directory.

What you see here is not the full story; we also agreed to move the pidfiles into /var/run. I'm leaving that for my next branch. Some of the other items might fit in better with /var/lib, but we'll ignore that for now — the important thing is that you can delete /var/cache/maas-test and start over without past runs coming back to haunt you.

Some of the changes you'll see in the diff are:
 * DEFAULT_STATE_DIR now provides a single shared, global default location.
 * xdg is no longer needed.
 * Documentation no longer says this stuff is in ~/.maas-test — we never updated that.
 * Generated man page has been re-generated. Don't bother reviewing that part.

Jeroen

[Julian Edwards (julian-edwards)]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

 review: approve

Lovely, thank you. Minor gripes inline for things I thought of after
our call. Sorry ...

>
> === modified file 'docs/man/maas-test.8.rst' ---
> docs/man/maas-test.8.rst 2014-01-08 07:33:48 +0000 +++
> docs/man/maas-test.8.rst 2014-01-09 05:13:48 +0000 @@ -108,8 +108,8
> @@ developers, by filing a Launchpad bug which includes the test
> results as an attachment.
>
> -By defaults the results will also be written to timestamped log
> files -under `~/.maas-test/logs`. +By default, the results are also
> written to timestamped log files under
> +`/var/cache/maas-test/logs`.

We should put them in /var/log/ Sorry I neglected to mention this before.

>
>
> Options @@ -188,7 +188,7 @@ test runs, but also caches a large
> amount of data on the testing system's filesystem. The proxy
> software used is `polipo`. The cache will be stored under -
> `~/.maas-test`. + `/var/cache/maas-test`.
>
> .. _`reporting options`:
>
> @@ -199,7 +199,7 @@ --log-results-only Write test results to a
> file, but don't upload them to Launchpad. Results will be written
> to a timestamped log file under - ~/.maas-test/logs. +
> `/var/cache/maas-test/logs`.

/var/log/

> + :param ssh_key_dir: Optional directory in which to store
> the SSH keys + for connecting to this KVM instance.
> Defaults to the default state + directory.

This is a bit awkward. "Defaults to the default..." Can you be more
specific? I think it's ok to mention /var/cache.

> if log_dir is None: - log_dir = os.path.join( -
> BaseDirectory.save_config_path('maas-test'), 'logs') +
> log_dir = DEFAULT_STATE_DIR

/var/log/maas-test

Lastly, check the packaging branch, you might be able to remove
python-xdg as a dependency.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlLOODgACgkQWhGlTF8G/HdX2gCdHeAsEo+7rzLkZ1eMFoMTcbb/
zg8AnRUI4zs3807YNAhMdASSk3AFf1cp
=ZO+r
-----END PGP SIGNATURE-----

review: Approve

[lp:~jtv/maas-test/var-cache]

106. By Jeroen T. Vermeulen on 2014-01-09

Review change in docstring.

[Jeroen T. Vermeulen (jtv)]

> > -By defaults the results will also be written to timestamped log
> > files -under `~/.maas-test/logs`. +By default, the results are also
> > written to timestamped log files under
> > +`/var/cache/maas-test/logs`.
>
> We should put them in /var/log/ Sorry I neglected to mention this before.

Told you /var/cache wasn't right for everything. :) I'll do that part in a separate branch because there will be testing repercussions. We don't want tests to try and create /var/log/maas-test (so it needs to be patched), and conversely we do want a production run to create it (so it needs to be tested).

> > + :param ssh_key_dir: Optional directory in which to store
> > the SSH keys + for connecting to this KVM instance.
> > Defaults to the default state + directory.
>
> This is a bit awkward. "Defaults to the default..." Can you be more
> specific? I think it's ok to mention /var/cache.

OK, done.

=== modified file 'docs/man/maas-test.8.rst'
--- docs/man/maas-test.8.rst	2014-01-08 07:33:48 +0000
+++ docs/man/maas-test.8.rst	2014-01-09 05:58:44 +0000
@@ -108,8 +108,8 @@
 developers, by filing a Launchpad bug which includes the test results as an
 attachment.
 
-By defaults the results will also be written to timestamped log files
-under `~/.maas-test/logs`.
+By default, the results are also written to timestamped log files under
+`/var/cache/maas-test/logs`.
 
 
 Options
@@ -188,7 +188,7 @@
   test runs, but also caches a large amount of data on the testing system's
   filesystem.
   The proxy software used is `polipo`.  The cache will be stored under
-  `~/.maas-test`.
+  `/var/cache/maas-test`.
 
 .. _`reporting options`:
 
@@ -199,7 +199,7 @@
 --log-results-only
   Write test results to a file, but don't upload them to Launchpad.
   Results will be written to a timestamped log file under
-  ~/.maas-test/logs.
+  `/var/cache/maas-test/logs`.
 
 
 Reporting bugs
@@ -226,9 +226,8 @@
 Files
 =====
 
-State and configuration for `maas-test` is stored in the user's home directory,
-under `~/.maas-test`.  This includes an ssh key pair for communicating with the
-virtual machine.
+State and configuration for `maas-test` is stored in `/var/cache/maas-test`.
+This includes an ssh key pair for communicating with the virtual machine.
 
 If you choose to run a local proxy, downloaded data will also be cached there.
 It can quickly grow to gigabyte sizes.

=== modified file 'maastest/kvmfixture.py'
--- maastest/kvmfixture.py	2014-01-08 17:31:43 +0000
+++ maastest/kvmfixture.py	2014-01-09 05:58:44 +0000
@@ -26,6 +26,7 @@
 from fixtures import Fixture
 from lxml import etree
 from maastest.utils import (
+    DEFAULT_STATE_DIR,
     extract_mac_ip_mapping,
     retries,
     run_command,
@@ -33,7 +34,6 @@
 import netaddr
 from six import text_type
 from testtools.content import text_content
-from xdg import BaseDirectory
 
 # Maximum wait for the virtual machine to complete its cloud-init
 # initialization.  It can take quite a while, because it performs an
@@ -110,9 +110,9 @@
         :param archives: An optional list of repository names.  If provided,
             each repository will be added onto the VM.
         :type archives: list
-        :param ssh_key_dir: The directory in which to store the SSH keys
-            for connecting to this KVM instance. If None, this defaults
-            to the default XDG config directory.
+        :param ssh_key_dir: Optional directory in which to store the SSH keys
+            for connecting to this KVM instance.  Defaults to
+            `/var/cache/maas-test`.
         :type ssh_key_dir: string.
         """
         super(KVMFixture, self).__init__()
@@ -136,7 +136,7 @@
             name = text_type(uuid1())
         self.name = name
         if ssh_key_dir is None:
-            ssh_key_dir = BaseDirectory.save_config_path('maas-test')
+            ssh_key_dir = DEFAULT_STATE_DIR
         self.ssh_private_key_file = os.path.join(
             ssh_key_dir, 'vm_ssh_id_rsa')
         self.running = False
@@ -205,8 +205,8 @@
     def generate_ssh_key(self):
         """Set up an ssh key pair for accessing the virtual machine.
 
-        The key pair will be in a directory `~/.maas-test/`.  If this did not
-        exist yet, it will be created.
+        The key pair will be stored in the SSH key directory `ssh_key_dir`.
+        If this did not exist yet, it will be created.
         """
         key_dir = os.path.dirname(self.ssh_private_key_file)
         if not os.path.exists(key_dir):

=== modified file 'maastest/proxyfixture.py'
--- maastest/proxyfixture.py	2014-01-08 17:31:43 +0000
+++ maastest/proxyfixture.py	2014-01-09 05:58:44 +0000
@@ -23,7 +23,6 @@
 from maastest import utils
 from netaddr import IPNetwork
 import netifaces
-from xdg import BaseDirectory
 
 
 DISK_CACHE_ROOT = '%(config_dir)s/proxy_cache/'
@@ -75,7 +74,7 @@
         super(LocalProxyFixture, self).__init__()
         self.port = port
         if config_dir is None:
-            config_dir = BaseDirectory.save_config_path('maas-test')
+            config_dir = utils.DEFAULT_STATE_DIR
         self.config_dir = config_dir
         self.pid_file = os.path.join(self.config_dir, 'proxy.pid')
         self._network_address = None

=== modified file 'maastest/report.py'
--- maastest/report.py	2014-01-08 10:56:59 +0000
+++ maastest/report.py	2014-01-09 05:58:44 +0000
@@ -28,7 +28,7 @@
     )
 
 from apiclient import multipart
-from xdg import BaseDirectory
+from maastest.utils import DEFAULT_STATE_DIR
 
 
 # This is largely cargo-culted from apiclient.multipart because
@@ -128,8 +128,7 @@
     :param log_dir: The directory under which to create `log_file_name`.
     """
     if log_dir is None:
-        log_dir = os.path.join(
-            BaseDirectory.save_config_path('maas-test'), 'logs')
+        log_dir = DEFAULT_STATE_DIR
     if not os.path.exists(log_dir):
         os.makedirs(log_dir)
 

=== modified file 'maastest/tests/test_kvmfixture.py'
--- maastest/tests/test_kvmfixture.py	2014-01-08 17:31:43 +0000
+++ maastest/tests/test_kvmfixture.py	2014-01-09 05:58:44 +0000
@@ -41,10 +41,10 @@
         super(TestKVMFixture, self).setUp()
         self.logger = self.useFixture(fixtures.FakeLogger())
         self.patch(kvmfixture, 'sleep', mock.MagicMock())
-        # KVMFixture.generate_ssh_key() generates an ssh key in ~/.maas-test,
-        # creating that directory if necessary.  Set up a fake home directory
-        # so it can do what it needs to.
-        self.home_dir = self.useFixture(fixtures.TempDir()).path
+        # KVMFixture.generate_ssh_key() generates an ssh key in maas-test's
+        # configuration directory, creating that directory if necessary.  Set
+        # up a temporary directory so it can do what it needs to.
+        self.state_dir = self.useFixture(fixtures.TempDir()).path
 
     def patch_run_command(self, return_value=0, stdout=b'', stderr=b''):
         """Patch out `kvmfixture` calls to `run_command`.
@@ -77,7 +77,7 @@
             series, architecture, name=name,
             direct_interface=direct_interface,
             direct_network=direct_network, archives=archives,
-            proxy_url=proxy_url, ssh_key_dir=self.home_dir)
+            proxy_url=proxy_url, ssh_key_dir=self.state_dir)
         # Hack to get fixture.addDetail to work without actually calling
         # fixture.setUp().
         fixture._details = {}
@@ -290,7 +290,7 @@
         self.assertThat(fixture.ssh_private_key_file, FileExists())
         self.assertThat(fixture.get_ssh_public_key_file(), FileExists())
         self.assertThat(
-            fixture.ssh_private_key_file, StartsWith(self.home_dir + '/'))
+            fixture.ssh_private_key_file, StartsWith(self.state_dir + '/'))
 
         self.assertThat(
             fixture.ssh_private_key_file,

=== modified file 'maastest/utils.py'
--- maastest/utils.py	2014-01-08 17:31:43 +0000
+++ maastest/utils.py	2014-01-09 05:58:44 +0000
@@ -1,4 +1,4 @@
-# Copyright 2013 Canonical Ltd.  This software is licensed under the
+# Copyright 2013-2014 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 """maas-test utilities."""
@@ -13,6 +13,7 @@
 __all__ = [
     "BINARY",
     "binary_content",
+    "DEFAULT_STATE_DIR",
     "determine_vm_series",
     "determine_vm_architecture",
     "get_uri",
@@ -45,6 +46,11 @@
 from testtools.content_type import ContentType
 
 
+# Default location for maas-test state, such as SSH keys for the virtual
+# machine, and the http proxy cache.
+DEFAULT_STATE_DIR = '/var/cache/maas-test'
+
+
 def read_file(path):
     """Return a given file's contents, as `bytes`."""
     with open(path, "rb") as f:

=== modified file 'man/maas-test.8'
--- man/maas-test.8	2014-01-08 07:33:58 +0000
+++ man/maas-test.8	2014-01-09 05:58:44 +0000
@@ -121,8 +121,8 @@
 developers, by filing a Launchpad bug which includes the test results as an
 attachment.
 .sp
-By defaults the results will also be written to timestamped log files
-under \fI~/.maas\-test/logs\fP\&.
+By default, the results are also written to timestamped log files under
+\fI/var/cache/maas\-test/logs\fP\&.
 .SH OPTIONS
 .INDENT 0.0
 .TP
@@ -199,7 +199,7 @@
 test runs, but also caches a large amount of data on the testing system\(aqs
 filesystem.
 The proxy software used is \fIpolipo\fP\&.  The cache will be stored under
-\fI~/.maas\-test\fP\&.
+\fI/var/cache/maas\-test\fP\&.
 .UNINDENT
 .INDENT 0.0
 .TP
@@ -210,7 +210,7 @@
 .B \-\-log\-results\-only
 Write test results to a file, but don\(aqt upload them to Launchpad.
 Results will be written to a timestamped log file under
-~/.maas\-test/logs.
+\fI/var/cache/maas\-test/logs\fP\&.
 .UNINDENT
 .SH REPORTING BUGS
 .sp
@@ -228,9 +228,8 @@
 Ubuntu virtualization tools (uvtool): \fI\%https://launchpad.net/uvtool\fP
 .SH FILES
 .sp
-State and configuration for \fImaas\-test\fP is stored in the user\(aqs home directory,
-under \fI~/.maas\-test\fP\&.  This includes an ssh key pair for communicating with the
-virtual machine.
+State and configuration for \fImaas\-test\fP is stored in \fI/var/cache/maas\-test\fP\&.
+This includes an ssh key pair for communicating with the virtual machine.
 .sp
 If you choose to run a local proxy, downloaded data will also be cached there.
 It can quickly grow to gigabyte sizes.

=== modified file 'packages.txt'
--- packages.txt	2013-12-06 07:02:08 +0000
+++ packages.txt	2014-01-09 05:58:44 +0000
@@ -11,7 +11,6 @@
 python-testresources
 python-testtools
 python-virtualenv
-python-xdg
 python3-fixtures
 python3-lxml
 python3-mock
@@ -19,7 +18,6 @@
 python3-six
 python3-testresources
 python3-testtools
-python3-xdg
 qemu-kvm
 uvtool
 uvtool-libvirt