Merge lp:~ptressel/sahana-eden/devel2 into lp:sahana-eden

[Pat Tressel]

Here's an updated update check. Cleanup plus some functional changes: Any template file can have an "edited" flag. (Non-Python files can put these in comments.) Adds "version" flags for template files, so it can tell when the template is ahead of the in-use file. Allows triggering an update check with an url query.

[Pat Tressel (ptressel)]

I gather there was a private (review) comment objecting to the quantity of (code) comments, and on the grounds of that invisible review comment, no review has been done by the actual reviewer (whoever is going to do that...). The comments in the code are there for the benefit of the reviewer, as they explain some of why things were done as they are. They are intended to be moved to the wiki, as appropriate. Some are ToDos. The very first ToDo is:

# @ToDo: Scrape all the how-to and what's-this comments, plus ToDos, into a
# wiki page, then reference that here.

With the comments inline, the reviewer can see easily what they refer to. Can the reviewer please have a look, and decide at least whether they are willing to waste round trips asking why things were done as they are, if the comments are removed prior to the review?

[Fran Boon (flavour)]

I have merged in most of these elements, thanks.
So far I have left out:
* 000_config VERSION check
- since currently this will break all existing installs without any current reason to do so (we can introduce this if/when that file has incompatible/mandatory changes added, which is less frequent than it used to be)
* update_check controller function (I don't see the usecase for it yet)

I also trimmed the comments.

review: Approve

=== modified file '.bzrignore'
--- .bzrignore	2011-12-06 16:57:57 +0000
+++ .bzrignore	2011-12-09 12:27:24 +0000
@@ -9,6 +9,7 @@
 databases/*
 errors/*
 models/000_config.py
+models/0000_update_check.py
 sessions/*
 static/scripts/tools/httpserver.log
 uploads/survey/translations/*

=== modified file 'deployment-templates/models/000_config.py'
--- deployment-templates/models/000_config.py	2011-11-30 17:15:49 +0000
+++ deployment-templates/models/000_config.py	2011-12-09 12:27:24 +0000
@@ -13,8 +13,11 @@
 
 # Remind admin to edit this file
 FINISHED_EDITING_CONFIG_FILE = False # change to True after you finish editing this file
-if not FINISHED_EDITING_CONFIG_FILE:
-    raise HTTP(501, body="Please edit models/000_config.py first")
+
+# This value will be changed when a new version is supplied. This will be used
+# to alert the admin that there are changes to merge in to the site's
+# customized copy of the file.
+VERSION_CONFIG = 1
 
 # Database settings
 deployment_settings.database.db_type = "sqlite"

=== modified file 'models/000_1st_run.py'
--- models/000_1st_run.py	2011-11-27 17:45:11 +0000
+++ models/000_1st_run.py	2011-12-09 12:27:24 +0000
@@ -2,6 +2,7 @@
 """
     1st RUN:
 
+    - Run update check if needed.
     - Import the S3 Framework Extensions
     - If needed, copy deployment specific templates to the live installation.
       Developers: note that the templates are version-controlled, while their
@@ -10,6 +11,128 @@
                   If you add something new to these files, you should also
                   make the change at deployment-templates and commit it.
 """
+# -----------------------------------------------------------------------------
+
+# Perform update checks: These are tests for proper configuration that are run
+# when the need is indicated by incrementing the CURRENT_UPDATE_CHECK_ID.
+# The previously checked id is written into the update check "canary" file,
+# 0000_update_check.py. If the file doesn't exist, or the ids don't match, the
+# update checks are run. Checks can also be run by doing a query with
+# "update_check" as the controller, e.g.: http://eden.example.com/update_check
+# In this case, after the check completes the controller will be set to "index".
+#
+# @ToDo: Scrape all the how-to and what's-this comments, plus ToDos, into a
+# wiki page, then reference that here.
+# @ToDo: If the deployment includes their own add-on modules, they may need an
+# independent update check id -- consider how that could be provided (e.g.
+# a 0000_site_update_check.
+# @ToDo: On a heavily used site, it is possible that more than one request will
+# find a mismatch in the update check id, and run the checks. If this is a
+# concern, consider adding an interlock and flag for a fatal result, e.g.
+# written into the canary file. On the other hand, if there is a fatal error,
+# the site admins should disable the site and fix the problem, so multiple
+# complaints of 500 errors might yield faster action.
+
+CURRENT_UPDATE_CHECK_ID = 1
+update_check_needed = False
+if request.controller == "update_check":
+    update_check_needed = True
+else:
+    try:
+        if CANARY_UPDATE_CHECK_ID != CURRENT_UPDATE_CHECK_ID:
+            update_check_needed = True
+    except NameError:
+        update_check_needed = True
+if update_check_needed:
+    # Run update checks -- these are the update_check() functions from each
+    # Python file in private/update_check that has such a function.
+    #
+    # These functions will have the Web2py environment available, but will run
+    # before any site configuration has been read or database opened. If either
+    # are needed for a check, the function can read 000_config.py, or open the
+    # database -- we are not concerned about speed at this point, as this is a
+    # "once"-only event after an update (for some value of "once" -- see ToDo
+    # above), and it's likely the sysadmin getting hit.
+    # @ToDo: To facilitate doing a database connectivity check, we might package
+    # the code in 00_db.py that opens the database into a function.
+    #
+    # update_check functions will receive the current globals (mainly those
+    # supplied by Web2py) as their sole argument. They should return a dict with
+    # either or both of these key / value pairs:
+    # {"error_messages": [...], "warning_messages": [...]}
+    # where the values are lists of message strings.
+    #
+    # Errors, reported in the error_messages list, are conditions that would
+    # prevent Eden from running, cause damage to data, or indicate configuration
+    # is incomplete. If any errors are reported, execution will end with a 500
+    # error. The supplied error messages are sent with the 500 error and printed
+    # on the console. Each message will be prefixed with ERROR:.
+    #
+    # Warnings, reported in the warning_messages list, do not stop execution,
+    # and are only printed on the console. Each message will be prefixed with
+    # WARNING:.
+    # @ToDo: The only material reason for reporting warnings in the update check
+    # is to get them all out in one batch. Otherwise, we might as well just let
+    # them be reported in place where the missing feature is wanted. Since
+    # warnings are not fatal and do not force the site to update, unlike errors,
+    # tests for their features must be left in the code where they're used. We
+    # could remove the warning messages there, though.
+    import os
+    from gluon.fileutils import listdir
+    update_check_path_parts = [
+        "applications", request.application, "private", "update_check"]
+    update_check_path = os.path.join(*update_check_path_parts)
+    update_check_import_path = ".".join(update_check_path_parts)
+    errors = []
+    warnings = []
+    # Supply the current (Web2py) environment. Pick out only the items that are
+    # safe for the check functions to combine with their own environments, i.e.
+    # not anything of the form __x__.
+    environment = dict((k, v) for (k, v) in globals().iteritems() if not k.startswith("__"))
+    for filename in listdir(update_check_path, expression = ".*\.py$"):
+        try:
+            exec "from %s.%s import update_check" % \
+                 (update_check_import_path, filename[0:-3])
+        except ImportError:
+            continue
+        messages = update_check(environment)
+        errors.extend(messages.get("error_messages", []))
+        warnings.extend(messages.get("warning_messages", []))
+
+    # Temporary catch-all check for dependency errors. This does not satisfy
+    # the goal of calling out all the setup errors at once -- it will die on
+    # the first fatal error encountered.
+    try:
+        import s3 as s3base
+    except Exception, errmsg:
+        errors.extend(errmsg)
+
+    # Report (non-fatal) warnings.
+    if warnings:
+        prefix = "\n" + T("WARNING: ")
+        msg = prefix + prefix.join(warnings)
+        import sys
+        print >> sys.stderr, msg
+    # Report errors and stop.
+    if errors:
+        prefix = "\n" + T("ERROR: ")
+        msg = prefix + prefix.join(errors)
+        import sys
+        print >> sys.stderr, msg
+        raise HTTP(500, body=msg)
+
+    # If we are still here, create or update the canary file.
+    from gluon import portalocker
+    canary = open(
+        "applications/%s/models/0000_update_check.py" % request.application, "w")
+    portalocker.lock(canary, portalocker.LOCK_EX)
+    statement = "CANARY_UPDATE_CHECK_ID = %s" % CURRENT_UPDATE_CHECK_ID
+    canary.write(statement)
+    canary.close()
+    if request.controller == "update_check":
+        redirect(URL(c="default", f="index"))
+
+# -----------------------------------------------------------------------------
 import os
 from gluon import current
 from gluon.storage import Storage
@@ -19,64 +142,12 @@
 
 # Import the S3 Framework
 import s3 as s3base
-
-# -----------------------------------------------------------------------------
 # Keep all S3 framework-level elements stored in response.s3, so as to avoid
 # polluting global namespace & to make it clear which part of the framework is
 # being interacted with.
 # Avoid using this where a method parameter could be used: http://en.wikipedia.org/wiki/Anti_pattern#Programming_anti-patterns
 response.s3 = Storage()
 s3 = response.s3
-
-# Minimum usable Web2py version: If Eden depends on a feature in Web2py that
-# only becomes available with a specific Web2py version, copy the text from
-# that Web2py's VERSION file here.
-s3.web2py_minimum_version = "Version 1.99.2 (2011-09-26 00:51:34) stable"
-# Check that the installed Web2py is sufficiently recent. The datetime string,
-# which changes with individual commits, is finer-grained than the released version.
-web2py_version_ok = True
-try:
-    from gluon.fileutils import parse_version
-except ImportError:
-    web2py_version_ok = False
-if web2py_version_ok:
-    web2py_minimum_datetime = parse_version(s3.web2py_minimum_version)[3]
-    web2py_installed_datetime = request.global_settings.web2py_version[3]
-    web2py_version_ok = web2py_installed_datetime >= web2py_minimum_datetime
-if not web2py_version_ok:
-    error = "ERROR: The installed version of Web2py is too old for the installed version of Eden.\nPlease upgrade Web2py to at least version: %s" % \
-            s3.web2py_minimum_version
-    import sys
-    print >> sys.stderr, error
-    raise HTTP(501, body=error)
-
-# -----------------------------------------------------------------------------
-template_src = os.path.join("applications", request.application, "deployment-templates")
-template_dst = os.path.join("applications", request.application)
-
-template_files = (
-    "models/000_config.py",
-    # Deprecated by Scheduler
-    #"cron/crontab"
-)
-
-copied_from_template = []
-
-for t in template_files:
-    dst_path = os.path.join(template_dst, t)
-    try:
-        os.stat(dst_path)
-    except OSError:
-        # not found, copy from template
-        import shutil
-        shutil.copy(os.path.join(template_src, t), dst_path)
-        copied_from_template.append(t)
-
-if copied_from_template:
-    raise HTTP(501, body="The following files were copied from templates and should be edited: %s" %
-                         ", ".join(copied_from_template))
-
-# -----------------------------------------------------------------------------
 response.s3.gis = Storage()  # Defined early for use by S3Config.
 deployment_settings = s3base.S3Config()
 current.deployment_settings = deployment_settings

=== modified file 'models/00_settings.py'
--- models/00_settings.py	2011-12-05 14:29:54 +0000
+++ models/00_settings.py	2011-12-09 12:27:24 +0000
@@ -257,10 +257,7 @@
 # Auth
 ######
 
-try:
-    auth.settings.password_min_length = 4
-except:
-    raise HTTP(501, body="Error: Running Web2Py < 3654, so please upgrade")
+auth.settings.password_min_length = 4
 auth.settings.expiration = 28800  # seconds
 
 #auth.settings.username_field = True

=== added file 'private/__init__.py'
=== added directory 'private/update_check'
=== added file 'private/update_check/__init__.py'
=== added file 'private/update_check/eden_update_check.py'
--- private/update_check/eden_update_check.py	1970-01-01 00:00:00 +0000
+++ private/update_check/eden_update_check.py	2011-12-09 12:27:24 +0000
@@ -0,0 +1,180 @@
+# -*- coding: utf-8 -*-
+"""
+    Check whether the configuration is sufficient to run Eden.
+"""
+
+def update_check(environment):
+    # Get Web2py environment into our globals.
+    globals().update(**environment)
+
+    import os
+    app_path_parts = ["applications", request.application]
+    app_path = os.path.join(*app_path_parts)
+
+    # Fatal configuration errors.
+    errors = []
+    # Non-fatal warnings.
+    warnings = []
+
+    # Minimum usable Web2py version: If Eden depends on a feature in Web2py,
+    # there are several ways one can test for that feature's availability.
+    # One is to check for the existence of some class or global variable the
+    # the feature provides uniquely. Lacking that, one can copy the text from
+    # VERSION in a version of Web2py at or after where the feature becomes
+    # available, then compare the timestamps in that version string with the
+    # running Web2py's version.
+
+    # Currently, the minimum usable Web2py is determined by the existence of
+    # the global "current".
+    try:
+        from gluon import current
+    except ImportError:
+        errors.append(
+            "The installed version of Web2py is too old -- it does not define current."
+            "\nPlease upgrade Web2py to a more recent version.")
+    # @ToDo: Could find the version or revision where current becomes available
+    # and say "upgrade to at least ..."
+
+    # Web2py's Scheduler was revamped and became usable at the following
+    # version. Scheduler isn't strictly required, so for the moment, this is
+    # only a warning. In future, the version, message to print, and whether
+    # this is a warning or error can all change -- the test mechanism itself
+    # remains the same.
+    web2py_minimum_version = "Version 1.99.2 (2011-09-26 00:51:34) stable"
+    # Check that the installed Web2py is sufficiently recent. The datetime
+    # string, which changes with individual commits, is finer-grained than the
+    # released version. (Note: If the version check is not needed just now,
+    # don't delete the code -- just comment it out. That avoids having someone
+    # redo the work of digging the info out of Web2py. Thank you.)
+    web2py_version_ok = True
+    try:
+        from gluon.fileutils import parse_version
+    except ImportError:
+        web2py_version_ok = False
+    if web2py_version_ok:
+        web2py_minimum_datetime = parse_version(web2py_minimum_version)[3]
+        web2py_installed_datetime = request.global_settings.web2py_version[3]
+        web2py_version_ok = web2py_installed_datetime >= web2py_minimum_datetime
+    if not web2py_version_ok:
+        warnings.append(
+            "The installed version of Web2py is too old to provide the Scheduler,"
+            "\nso scheduled tasks will not be available. If you need scheduled tasks,"
+            "\nplease upgrade Web2py to at least version: %s" % \
+            web2py_minimum_version)
+
+    # --------------------------------------------------------------------------
+    template_src = os.path.join(app_path, "deployment-templates")
+    template_dst = app_path
+
+    template_files = (
+        os.path.join("models", "000_config.py"),
+        # Deprecated by Scheduler
+        #"cron/crontab"
+    )
+
+    copied_from_template = []
+
+    for t in template_files:
+        src_path = os.path.join(template_src, t)
+        dst_path = os.path.join(template_dst, t)
+        try:
+            os.stat(dst_path)
+        except OSError:
+            # not found, copy from template
+            import shutil
+            shutil.copy(src_path, dst_path)
+            copied_from_template.append(t)
+        else:
+            # Found the file in the destination -- check if it's up to date and
+            # has been edited.
+            #
+            # Each file can have a flag for whether it's been edited. This
+            # should be of the form:
+            # FINISHED_EDITING_\w*\s*=\s*(True|False)
+            # Any suffix is ok after FINISHED_EDITING_ and the "assignment" can
+            # be in a comment if the file is not a Python file.
+            #
+            # Likewise, each file can have a version number. This should be of
+            # the form:
+            # VERSION_\w*\s*=\s*[0-9]+
+            # No ordering is implied for the "number" -- it's matched as a
+            # string -- but it will be easier to maintain if it's incremented
+            # for new versions.
+            #
+            # No spurious errors will be reported for files that don't have
+            # edited or version flags.
+            #
+            # @Note the versions are not directly tied to the update check id
+            # in 000_1st_run. A file's version would be changed when the file
+            # changes, and a file change might need an update check, but an
+            # update check might be required for some other reason.
+            import re
+            edited_pattern = r"FINISHED_EDITING_\w*\s*=\s*(True|False)"
+            version_pattern = r"VERSION_\w*\s*=\s*([0-9]+)"
+            edited_matcher = re.compile(edited_pattern).match
+            version_matcher = re.compile(version_pattern).match
+            has_edited = False
+            has_version = False
+
+            # Get version and edited from the in-use copy of the file.
+            with open(dst_path) as f:
+                for line in f:
+                    if not has_edited:
+                        edited_result = edited_matcher(line)
+                        if edited_result:
+                            has_edited = True
+                            edited = edited_result.group(1)
+                    if not has_version:
+                        version_result = version_matcher(line)
+                        if version_result:
+                            has_version = True
+                            version = version_result.group(1)
+                    if has_edited and has_version:
+                        break
+            if has_edited and (edited != "True"):
+                errors.append("Please edit %s before starting the system." % t)
+
+            # Get version from deployment-templates.
+            template_has_version = False
+            with open(src_path) as f:
+                for line in f:
+                    version_result = version_matcher(line)
+                    if version_result:
+                        template_has_version = True
+                        template_version = version_result.group(1)
+                        break
+            if (has_version != template_has_version) or \
+               has_version and (version != template_version):
+                errors.append(
+                    "Version of %s does not match that in deployment-templates.\n" % t +
+                    "Please merge in changes from the template.")
+                
+    if copied_from_template:
+        errors.append(
+            "The following files were copied from templates and should be edited: %s" %
+            ", ".join(copied_from_template))
+
+    # @ToDo specifically about 000_config:
+    # Right now we only want these flags, but later we may want more of
+    # the config values. For that, execfile() would be appropriate so
+    # it can be evaluated in an appropriate global / local environment.
+    # However, that fails with what appears to be a compile() error:
+    #   TypeError: must be string without null bytes, not str
+    # Are there actually null bytes in 000_config.py, and are they
+    # somehow just fine when Web2py compiles it? Is this because of the
+    # Unicode chars -- do some of them contain zero bytes? I read the
+    # file as bytes and didn't find nulls, so suspect the error is a red
+    # herring. Time to download the Python source code...
+    # 
+    # @Note: What about import to read values from 000_config? It isn't
+    # designed to be imported, but rather to have its code executed in a
+    # pre-existing environment (it has unsatisfied references if one
+    # attempts to import it) so can't import it to get the value of the
+    # "edited" flag. It could be made importable by adding some (fairly
+    # time consuming) environment setup. Or turned into a function, as
+    # per this file.
+    #
+    # @Note: Perhaps web setup could provide config settings in some
+    # more accessible form.
+    
+    return {"error_messages": errors, "warning_messages": warnings}
\ No newline at end of file