Merge lp:~allenap/maas/remove-maas into lp:~maas-committers/maas/trunk

[Gavin Panella]

Remove src/maas.

This fails in one test: http://paste.ubuntu.com/23460489/

I haven't been able to figure it out yet, but I wanted to put this up in case someone else <cough>Blake<cough> would have an idea.

[Blake Rouse (blake-rouse)]

I will have to dig into this to see why its failing. It has to be something in the tarball that is used for upgrading pre 2.0.

[Andres Rodriguez (andreserl)]

Why are we removing src/maas ?

review: Needs Information

[Gavin Panella (allenap)]

> Why are we removing src/maas ?

It's not needed as a standalone thing, and hasn't been for a long time now. It also gets in the way of using "maas" as a namespace package, a change Blake has kicked off with the client API.

[Gavin Panella (allenap)]

> I will have to dig into this to see why its failing. It has to be something in
> the tarball that is used for upgrading pre 2.0.

Thanks. I did make one change to the tarball: I added maas19settings.py. You'll get this when pulling this branch, but it won't show up in the diff, so I've pasted it too: http://paste.ubuntu.com/23480843/

[Andres Rodriguez (andreserl)]

This would need a lot of changes on the installation. src/maas gets up installed in /usr/share/maas and would break all installations....

[Gavin Panella (allenap)]

> This would need a lot of changes on the installation. src/maas gets up
> installed in /usr/share/maas and would break all installations....

A packaging change is needed, yes, but /usr/share/maas/maas can be removed. There's nothing in there that needs to be preserved so there isn't any migration needed; configuration has been in /etc/maas for a long time now. The only difficult bit is making it work with migrations from pre-2.0.

[Gavin Panella (allenap)]

Okay, this is working now: I had missed out SOUTH_MIGRATION_MODULES from the maas19settings.py file that's in the South migrations tarball. I'll prep a packaging change.

[Gavin Panella (allenap)]

Packaging change in lp:~allenap/maas/packaging.remove-maas.

[Blake Rouse (blake-rouse)]

This is good work. Glad to see that folder in the code removed.

review: Approve

=== modified file 'HACKING.txt'
--- HACKING.txt	2016-03-28 13:54:47 +0000
+++ HACKING.txt	2016-12-02 09:29:56 +0000
@@ -138,8 +138,9 @@
 regiond.log. To enable logging of all exceptions even exceptions where MAAS
 will return the correct HTTP status code.::
 
-    $ sudo sed -i 's/DEBUG = False/DEBUG = True/g' /usr/share/maas/maas/settings.py
-    $ sudo service maas-regiond restart
+  $ sudo sed -i 's/DEBUG = False/DEBUG = True/g' \
+  >   /usr/lib/python3/dist-packages/maasserver/djangosettings/settings.py
+  $ sudo service maas-regiond restart
 
 Run regiond in foreground
 ^^^^^^^^^^^^^^^^^^^^^^^^^
@@ -149,8 +150,10 @@
 placed a breakpoint into the code you want to inspect you can start the regiond
 process in the foreground.::
 
-    $ sudo service maas-regiond stop
-    $ sudo -u maas -H DJANGO_SETTINGS_MODULE=maas.settings PYTHONPATH=/usr/share/maas twistd3 --nodaemon --pidfile= maas-regiond
+  $ sudo service maas-regiond stop
+  $ sudo -u maas -H \
+  >   DJANGO_SETTINGS_MODULE=maasserver.djangosettings.settings \
+  >   twistd3 --nodaemon --pidfile= maas-regiond
 
 
 .. Note::
@@ -175,7 +178,8 @@
 Development MAAS server setup
 =============================
 
-Access to the database is configured in ``src/maas/development.py``.
+Access to the database is configured in
+``src/maasserver/djangosettings/development.py``.
 
 The ``Makefile`` or the test suite sets up a development database
 cluster inside your branch. It lives in the ``db`` directory, which

=== modified file 'Makefile'
--- Makefile	2016-10-17 06:38:56 +0000
+++ Makefile	2016-12-02 09:29:56 +0000
@@ -427,7 +427,8 @@
 	$(warning 'distclean' is deprecated; use 'clean')
 
 harness: bin/maas-region bin/database
-	$(dbrun) bin/maas-region shell --settings=maas.demo
+	$(dbrun) bin/maas-region shell \
+	  --settings=maasserver.djangosettings.demo
 
 dbharness: bin/database
 	bin/database --preserve shell

=== modified file 'buildout.cfg'
--- buildout.cfg	2016-10-12 15:26:17 +0000
+++ buildout.cfg	2016-12-02 09:29:56 +0000
@@ -104,7 +104,7 @@
   twistd.region=twisted.scripts.twistd:run
 initialization =
   ${common:initialization}
-  environ.setdefault("DJANGO_SETTINGS_MODULE", "maas.development")
+  environ.setdefault("DJANGO_SETTINGS_MODULE", "maasserver.djangosettings.development")
 scripts =
   maas-region
   twistd.region
@@ -129,7 +129,6 @@
     # "--with-resources",
     "--with-scenarios",
     "--with-select",
-    "--select-dir=src/maas",
     "--select-dir=src/maasserver",
     "--select-dir=src/metadataserver",
     "--cover-package=maas,maasserver,metadataserver",
@@ -294,7 +293,7 @@
   from os import environ
   environ.setdefault("MAAS_RACK_DEVELOP", "TRUE")
   environ.setdefault("MAAS_ROOT", "${buildout:directory}/run-e2e")
-  environ.setdefault("DJANGO_SETTINGS_MODULE", "maas.development")
+  environ.setdefault("DJANGO_SETTINGS_MODULE", "maasserver.djangosettings.development")
   environ.setdefault("DEV_DB_NAME", "test_maas_e2e")
   environ.setdefault("MAAS_PREVENT_MIGRATIONS", "1")
 

=== modified file 'docs/conf.py'
--- docs/conf.py	2016-03-28 13:54:47 +0000
+++ docs/conf.py	2016-12-02 09:29:56 +0000
@@ -24,7 +24,8 @@
 from pytz import UTC
 
 # Configure MAAS's settings.
-environ.setdefault("DJANGO_SETTINGS_MODULE", "maas.settings")
+environ.setdefault(
+    "DJANGO_SETTINGS_MODULE", "maasserver.djangosettings.settings")
 
 # If extensions (or modules to document with autodoc) are in another directory,
 # add these directories to sys.path here. If the directory is relative to the

=== modified file 'docs/troubleshooting.rst'
--- docs/troubleshooting.rst	2014-09-10 16:20:31 +0000
+++ docs/troubleshooting.rst	2016-12-02 09:29:56 +0000
@@ -111,7 +111,7 @@
      always point at the local server.
   #. If you are still getting "404 - Page not found" errors, check that the MAAS
      web interface has been installed in the right place. There should be a file
-     present called /usr/share/maas/maas/urls.py
+     called ``urls.py`` in ``/usr/lib/python3/dist-packages/maasserver/djangosettings/``.
 
 Debugging ephemeral image
 =========================

=== modified file 'media/README'
--- media/README	2012-03-11 21:13:22 +0000
+++ media/README	2016-12-02 09:29:56 +0000
@@ -1,5 +1,5 @@
 This folder contains somewhat ephemeral things: subfolders serve as
-MEDIA_ROOT for maas.demo and maas.development environments. The
-media/demo directory should always exist and not be deleted, though
-its contents can be. The media/development directory should be created
-and destroyed by tests, as needed.
+MEDIA_ROOT for maasserver.djangosettings.demo and .development
+environments. The media/demo directory should always exist and not be
+deleted, though its contents can be. The media/development directory
+should be created and destroyed by tests, as needed.

=== modified file 'services/reloader/run'
--- services/reloader/run	2016-05-11 19:01:48 +0000
+++ services/reloader/run	2016-12-02 09:29:56 +0000
@@ -128,7 +128,7 @@
         exclude_filter=lambda path: (
             "/test/" in path or "/testing/" in path or "/." in path))
     wm.add_watch(
-        ["src/maas*", "src/meta*"], TRIGGER_EVENTS,
+        ["src/maasserver", "src/metadataserver"], TRIGGER_EVENTS,
         proc_fun=handle_maas_change, rec=True, auto_add=True, do_glob=True)
     wm.add_watch(
         ["src/prov*"], TRIGGER_EVENTS, proc_fun=handle_pserv_change,

=== modified file 'src/maascli/cli.py'
--- src/maascli/cli.py	2016-07-30 01:17:54 +0000
+++ src/maascli/cli.py	2016-12-02 09:29:56 +0000
@@ -189,8 +189,8 @@
     # Setup and the allowed django commands into the maascli.
     management = get_django_management()
     if management is not None and is_maasserver_available():
-        os.environ.setdefault("DJANGO_SETTINGS_MODULE", "maas.settings")
-        sys.path.append('/usr/share/maas')
+        os.environ.setdefault(
+            "DJANGO_SETTINGS_MODULE", "maasserver.djangosettings.settings")
         load_regiond_commands(management, parser)
 
 

=== modified file 'src/maasserver/__init__.py'
--- src/maasserver/__init__.py	2016-10-20 14:45:06 +0000
+++ src/maasserver/__init__.py	2016-12-02 09:29:56 +0000
@@ -8,7 +8,7 @@
     'DefaultViewMeta',
     'is_master_process',
     'logger',
-    ]
+]
 
 import logging
 from os import environ

=== renamed directory 'src/maas' => 'src/maasserver/djangosettings'
=== modified file 'src/maasserver/djangosettings/demo.py'
--- src/maas/demo.py	2016-06-07 19:59:49 +0000
+++ src/maasserver/djangosettings/demo.py	2016-12-02 09:29:56 +0000
@@ -5,7 +5,7 @@
 
 from os.path import abspath
 
-from maas import (
+from maasserver.djangosettings import (
     development,
     import_settings,
     settings,

=== modified file 'src/maasserver/djangosettings/development.py'
--- src/maas/development.py	2016-10-18 11:21:26 +0000
+++ src/maasserver/djangosettings/development.py	2016-12-02 09:29:56 +0000
@@ -7,7 +7,7 @@
 from os.path import abspath
 
 from formencode.validators import StringBool
-from maas import (
+from maasserver.djangosettings import (
     fix_up_databases,
     import_settings,
     settings,

=== modified file 'src/maasserver/djangosettings/settings.py'
--- src/maas/settings.py	2016-11-17 11:46:08 +0000
+++ src/maasserver/djangosettings/settings.py	2016-12-02 09:29:56 +0000
@@ -6,9 +6,9 @@
 import os
 
 import django.template.base
-from maas import fix_up_databases
-from maas.monkey import patch_get_script_prefix
 from maasserver.config import RegionConfiguration
+from maasserver.djangosettings import fix_up_databases
+from maasserver.djangosettings.monkey import patch_get_script_prefix
 
 
 def _read_timezone(tzfilename='/etc/timezone'):
@@ -265,7 +265,7 @@
 
 )
 
-ROOT_URLCONF = 'maas.urls'
+ROOT_URLCONF = 'maasserver.djangosettings.urls'
 
 TEMPLATE_DIRS = (
     # Put strings here, like "/home/html/django_templates"

=== renamed file 'src/maas/tests/test_maas.py' => 'src/maasserver/djangosettings/tests/test_settings.py'
--- src/maas/tests/test_maas.py	2016-06-21 10:29:11 +0000
+++ src/maasserver/djangosettings/tests/test_settings.py	2016-12-02 09:29:56 +0000
@@ -10,11 +10,11 @@
 
 from django.conf import settings
 from django.db import connections
-from maas import (
+from maasserver.djangosettings import (
     find_settings,
     import_settings,
 )
-from maas.settings import (
+from maasserver.djangosettings.settings import (
     _get_local_timezone,
     _read_timezone,
 )

=== modified file 'src/maasserver/management/commands/dbupgrade.py'
--- src/maasserver/management/commands/dbupgrade.py	2016-09-04 19:57:50 +0000
+++ src/maasserver/management/commands/dbupgrade.py	2016-12-02 09:29:56 +0000
@@ -9,6 +9,7 @@
 __all__ = []
 
 from importlib import import_module
+import json
 import optparse
 import os
 import shutil
@@ -34,7 +35,7 @@
 # Script that performs the south migrations for MAAS under django 1.6 and
 # python2.7.
 MAAS_UPGRADE_SCRIPT = """\
-# Copyright 2015 Canonical Ltd.  This software is licensed under the
+# Copyright 2015-2016 Canonical Ltd.  This software is licensed under the
 # GNU Affero General Public License version 3 (see the file LICENSE).
 
 from __future__ import (
@@ -46,12 +47,49 @@
 str = None
 
 __metaclass__ = type
-__all__ = [
-    ]
+__all__ = []
 
 import os
 import sys
 
+import django.conf
+
+
+class LazySettings(django.conf.LazySettings):
+    '''Prevent Django from mangling warnings settings.
+
+    At present, Django adds a single filter that surfaces all deprecation
+    warnings, but MAAS handles them differently. Django doesn't appear to give
+    a way to prevent it from doing its thing, so we must undo its changes.
+
+    Deprecation warnings in production environments are not desirable as they
+    are a developer tool, and not something an end user can reasonably do
+    something about. This brings control of warnings back into MAAS's control.
+    '''
+
+    def _configure_logging(self):
+        # This is a copy of *half* of Django's `_configure_logging`, omitting
+        # the problematic bits.
+        if self.LOGGING_CONFIG:
+            from django.utils.log import DEFAULT_LOGGING
+            from django.utils.module_loading import import_by_path
+            # First find the logging configuration function ...
+            logging_config_func = import_by_path(self.LOGGING_CONFIG)
+            logging_config_func(DEFAULT_LOGGING)
+            # ... then invoke it with the logging settings
+            if self.LOGGING:
+                logging_config_func(self.LOGGING)
+
+
+# Install our `LazySettings` as the Django-global settings class. First,
+# ensure that Django hasn't yet loaded its settings.
+assert not django.conf.settings.configured
+# This is needed because Django's `LazySettings` overrides `__setattr__`.
+object.__setattr__(django.conf.settings, "__class__", LazySettings)
+
+# Force Django configuration.
+os.environ["DJANGO_SETTINGS_MODULE"] = "maas19settings"
+
 # Inject the sys.path from the parent process so that the python path is
 # is similar, except that the directory that this script is running from is
 # already the first path in sys.path.
@@ -132,6 +170,11 @@
         tempdir = tempfile.mkdtemp(prefix='maas-upgrade-')
         subprocess.check_call([
             "tar", "zxf", path_to_tarball, "-C", tempdir])
+
+        settings_json = os.path.join(tempdir, "maas19settings.json")
+        with open(settings_json, "w", encoding="utf-8") as fd:
+            fd.write(json.dumps({"DATABASES": settings.DATABASES}))
+
         script_path = os.path.join(tempdir, "migrate.py")
         with open(script_path, "wb") as fp:
             fp.write(MAAS_UPGRADE_SCRIPT.encode("utf-8"))

=== modified file 'src/maasserver/management/commands/tests/test_dbupgrade.py'
--- src/maasserver/management/commands/tests/test_dbupgrade.py	2016-03-28 13:54:47 +0000
+++ src/maasserver/management/commands/tests/test_dbupgrade.py	2016-12-02 09:29:56 +0000
@@ -64,7 +64,10 @@
         env = os.environ.copy()
         env["MAAS_PREVENT_MIGRATIONS"] = "0"
         mra = os.path.join(root, "bin", "maas-region")
-        cmd = [mra, "dbupgrade", "--settings", "maas.settings"]
+        cmd = [
+            mra, "dbupgrade", "--settings",
+            "maasserver.djangosettings.settings",
+        ]
         if always_south:
             cmd.append("--always-south")
         self.execute(cmd, env=env)

=== modified file 'src/maasserver/migrations/south/django16_south_maas19.tar.gz'
Binary files src/maasserver/migrations/south/django16_south_maas19.tar.gz	2016-11-17 11:46:08 +0000 and src/maasserver/migrations/south/django16_south_maas19.tar.gz	2016-12-02 09:29:56 +0000 differ
=== modified file 'utilities/check-imports'
--- utilities/check-imports	2016-10-19 17:14:02 +0000
+++ utilities/check-imports	2016-12-02 09:29:56 +0000
@@ -263,19 +263,6 @@
 )
 
 
-RegionControllerConfig = files("src/maas/**/*.py")
-RegionControllerConfigRule = Rule(
-    Allow("django|django.**"),
-    Allow("formencode.validators.*"),
-    Allow("maasserver.config.*"),
-    Allow("maasserver.testing.testcase.*"),
-    Allow("maas|maas.**"),
-    Allow("provisioningserver|provisioningserver.**"),
-    Allow("psycopg2|psycopg2.**"),
-    Allow(StandardLibraries),
-)
-
-
 Tests = files(
     "src/**/test_*.py",
     "src/**/testing/**/*.py",
@@ -424,20 +411,6 @@
             Allow(TestingLibraries),
         ),
     ),
-
-    #
-    # REGION CONTROLLER CONFIGURATION
-    #
-    (
-        RegionControllerConfig - Tests,
-        RegionControllerConfigRule,
-    ),
-    (
-        RegionControllerConfig & Tests,
-        RegionControllerConfigRule | Rule(
-            Allow(TestingLibraries),
-        ),
-    ),
     #
     # TESTING HELPERS
     #

=== modified file 'utilities/remote-reinstall'
--- utilities/remote-reinstall	2016-10-27 00:01:01 +0000
+++ utilities/remote-reinstall	2016-12-02 09:29:56 +0000
@@ -159,10 +159,6 @@
     root@${hostname}:/etc/maas
 ssh_run "chown -R maas:maas /etc/maas"
 
-echo " - src/maas --> /usr/share/maas/maas"
-try rsync -${rsync_options} src/maas/ \
-    root@${hostname}:/usr/share/maas/maas
-
 echo ""
 echo "Synchronizing static web content..."
 echo " - Removing /usr/share/maas/web/static/*..."