Merge lp:~ensoft-opensource/ensoft-sextant/v1_ssh into lp:ensoft-sextant

[Patrick Stevens]

Add support for SSH-ing into the Neo4j instance, rather than connecting directly.

Add support for SSH-ing into the Neo4j instance, rather than connecting directly.

Updates the documentation to add a new SSH section; adds a section to sextant.__main__ to handle SSHing. This parses the input arguments to Sextant and searches for remote_neo4j, then uses that or whichever configured default exists. It picks a random high-numbered open port, creates a port forward between localhost:highport and remote_neo4j, then passes localhost:highport in to Sextant proper. Once Sextant has finished, closes the port forward.
Adds an option display_neo4j to Sextant's internal functions (hidden to end-users) so that Sextant functions can use remoteneo4j:7474 rather than localhost:highport in messages output to the user.
Adds a use_ssh_tunnel config option (defaulting to True) to make Sextant connect directly rather than by SSH.
Moves any configuration-wrangling into sextant.environment rather than sextant.__main__, because this allows wrappers to Sextant to access configuration.

Also a small bugfix where sextant.export.ProgramConverter.to_dot was ignoring its remove_self_calls input, and some cosmetic changes (primarily "== None" and "!= None" to "is None" and "is not None") in sextant.query. Sorry for having these in this branch.

[Patrick Stevens (patrickas)]

The major qualm I have with this is that the config file is currently parsed twice (because environment.load_config is called once in the wrapper and once in sextant.__main__) - this isn't really a problem, but it seems sub-optimal. Can anyone think of any easy fixes?

[James (jamesh-f)]

Please see comments on Documentation.

review: Needs Fixing

[ChrisD (gingerchris)]

Big question (that I should perhaps know the answer to):
Why did we decide to implement this as a wrapper to sextant and not simply make it optional functionality that sextant can provide?

I'm perhaps missing something, but if we integrate this with sextant that removes a lot of faff e.g. wrappers extra odd args...

Note that some of my inline comments were added prior to this realisation.

[Patrick Stevens (patrickas)]

> Big question (that I should perhaps know the answer to):
> Why did we decide to implement this as a wrapper to sextant and not simply
> make it optional functionality that sextant can provide?
>
> I'm perhaps missing something, but if we integrate this with sextant that
> removes a lot of faff e.g. wrappers extra odd args...
>
> Note that some of my inline comments were added prior to this realisation.

This was after discussion with Rob - it also seems logically sensible to me, to separate the fact that we're SSHing for all of Sextant from Sextant itself. (It doesn't get rid of that many extra args to put it inline - we still want to pass stuff in to sextant.query, sextant.add_program and so forth...)

[ChrisD (gingerchris)]

But by virtue of having those args and the wrapper script parsing sextants
config, importing its modules and understating its cli there is already no
meaningful separation.

Further, trying to maintain separation is introducing hacks.

Some args may still exist to internal Apis but there won't be any need to
have extra external (e.g. Cli) args.

On Friday, 29 August 2014, Patrick Stevens <email address hidden> wrote:

> > Big question (that I should perhaps know the answer to):
> > Why did we decide to implement this as a wrapper to sextant and not
> simply
> > make it optional functionality that sextant can provide?
> >
> > I'm perhaps missing something, but if we integrate this with sextant that
> > removes a lot of faff e.g. wrappers extra odd args...
> >
> > Note that some of my inline comments were added prior to this
> realisation.
>
> This was after discussion with Rob - it also seems logically sensible to
> me, to separate the fact that we're SSHing for all of Sextant from Sextant
> itself. (It doesn't get rid of that many extra args to put it inline - we
> still want to pass stuff in to sextant.query, sextant.add_program and so
> forth...)
> --
>
> https://code.launchpad.net/~ensoft-opensource/ensoft-sextant/v1_ssh/+merge/232403
> Your team Ensoft Open Source is subscribed to branch
> lp:~ensoft-opensource/ensoft-sextant/v1_ssh.
>

[Patrick Stevens (patrickas)]

Changes made per Chuck's comments. In particular, elide sextant_wrapper into sextant.__main__.

[Patrick Stevens (patrickas)]

Two comments (I've made Chuck's changes on anything I haven't commented)

[Patrick Stevens (patrickas)]

I tried that - replaced the <for key in parsers> loop with:
    argumentparser.add_argument('--remote-neo4j', metavar="URL",
                                  help="URL of neo4j server", type=str,
                                  default=config.remote_neo4j)
    argumentparser.add_argument('--use-ssh-tunnel', metavar="BOOL", type=str,
                                  help="whether to SSH into the remote server,"
                                       "True/False",
                                  default=str(config.use_ssh_tunnel))

and that causes the call <./bin/sextant query programs --remote-neo4j http://localhost:7474> to claim an error in __main__.py:
error: unrecognized arguments: --remote-neo4j http://localhost:7474

According to [1], the correct way is to use parents (so parser.add_subparser('run', parents=[parser]) as the way to add a subparser), but my first attempt at that failed for some reason I don't remember.

[1]: http://stackoverflow.com/questions/7066826/in-python-how-to-get-subparsers-to-read-in-parent-parsers-argument

[James (jamesh-f)]

review: Approve

=== added file 'bin/sextant'
--- bin/sextant	1970-01-01 00:00:00 +0000
+++ bin/sextant	2014-08-29 14:18:31 +0000
@@ -0,0 +1,12 @@
+#!/bin/bash
+
+DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
+
+export PYTHONPATH=${PYTHONPATH}:${DIR}/../src
+
+# We allow people to specify whether they want to run Sextant under Python 2
+# or 3, by setting the environment variable SEXTANT_PYTHON to the Python they
+# want to use.
+export SEXTANT_PYTHON=${SEXTANT_PYTHON:=python}
+
+$SEXTANT_PYTHON -m sextant $@

=== removed file 'bin/sextant'
--- bin/sextant	2014-08-19 16:04:41 +0000
+++ bin/sextant	1970-01-01 00:00:00 +0000
@@ -1,10 +0,0 @@
-#!/bin/bash
-
-DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
-
-export PYTHONPATH=${PYTHONPATH}:${DIR}/../src
-
-export SEXTANT_PYTHON=${SEXTANT_PYTHON:=python}
-
-$SEXTANT_PYTHON -m sextant $@
-

=== modified file 'doc/wiki/Reference'
--- doc/wiki/Reference	2014-08-26 11:04:04 +0000
+++ doc/wiki/Reference	2014-08-29 14:18:31 +0000
@@ -19,6 +19,10 @@
 {{{
 sextant delete-program <program name to be deleted as stored in database>
 }}}
+
+== Audit ==
+To see a detailed view of a list of programs on the server, type {{{ sextant audit }}} into the command line.  This will return a list of programs stored on the Neo4j server, along with who uploaded them, that person's username on the computer they used to upload, and the date they were uploaded.
+
 == Queries ==
 === Command Line ===
 Command line queries produce a text output, either as a list or in GraphML which can be opened in yED.
@@ -40,9 +44,13 @@
 {{{
 sextant query -h
 }}}
+
 === Web ===
 To run the web server type
 {{{ sextant web }}}
 into the command line; then navigate a web browser to the port specified.
-== Audit ==
-To see a more detailed view of a list of programs on the server type {{{ sextant audit }}} into the command line.  This will return a list of programs on the Neo4j server, specified in either the config file of command line, along with their unique program number, who uploaded them, that persons id and the date they were uploaded.
\ No newline at end of file
+
+== SSH ==
+Sextant allows you to create an SSH tunnel between your client and the Neo4J server. Sextant picks a random high-numbered port which is not in use, sets up an SSH tunnel between {{{ localhost:{that port} }}} and the specified {{{ remote_neo4j }}} in the configuration file (or as the {{{ --remote-neo4j }}} command-line option), and then uses {{{ localhost:{port} }}} for communications.  This happens by default; if you do not wish for this to happen, set the config option {{{ use_ssh_tunnel }}} to {{{ False }}} in the configuration file.
+
+Warning: if you have {{{ remote_neo4j }}} set to some variant of {{{ localhost }}}, then SSH may fail (because a machine can't easily SSH to itself as currently implemented). We have some error-catching here - if we detect that we are attempting to SSH to ourself, then we skip using SSH at all - but it can probably be broken by a determined user. If you do encounter an error for this reason, the workaround at present is to set {{{ use_ssh_tunnel }}} to {{{ False }}} in the configuration file.

=== modified file 'etc/sextant.conf'
--- etc/sextant.conf	2014-08-22 14:49:33 +0000
+++ etc/sextant.conf	2014-08-29 14:18:31 +0000
@@ -1,7 +1,15 @@
 [Neo4j]
 # URL of the Neo4J server
-remote_neo4j = http://localhost:7474
-# port on which to serve Sextant Web
-port = 2905
-# number of calls incoming before we consider a function to be 'common'
-common_function_calls = 10
+remote_neo4j = 
+# Port on which to serve Sextant Web
+#port = 2905
+# Number of calls incoming before we consider a function to be 'common'.
+# Defaults to 10.
+#common_function_calls = 10
+# Whether to connect using SSH to the remote server. Warning: if remote_neo4j
+# is some kind of localhost, we won't be able to SSH. Default is True (that is,
+# automatically SSH).
+#use_ssh_tunnel = True
+# Which username to use on the remote SSH server. Default, or empty, is your
+# currently-logged-in username.
+#ssh_username =

=== modified file 'setup.py'
--- setup.py	2014-08-18 08:04:35 +0000
+++ setup.py	2014-08-29 14:18:31 +0000
@@ -18,7 +18,7 @@
     packages=['sextant', 'sextant.web', 'resources', 'etc'],
     package_dir={'sextant': 'src/sextant', 'resources': 'resources', 'etc': 'etc'},
     scripts=['bin/sextant'],
-    install_requires=['neo4jrestclient', 'twisted'],
+    install_requires=['twisted'],
     package_data={'resources': ['sextant/web/*'], 'etc': ['*.conf']},
 )
 

=== modified file 'src/sextant/__main__.py'
--- src/sextant/__main__.py	2014-08-26 11:04:04 +0000
+++ src/sextant/__main__.py	2014-08-29 14:18:31 +0000
@@ -7,223 +7,123 @@
 
 from __future__ import absolute_import, print_function
 
-import argparse
-try:
-    import ConfigParser
-except ImportError:
-    import configparser as ConfigParser
+import sys
+import random
+import socket
 import logging
 import logging.config
-import os
+import argparse
 import requests
-import sys
-
-from . import update_db
+import subprocess
+
+try:
+    from urllib import parse
+except ImportError:  # fall back to Python 2
+    import urlparse as parse
+
 from . import query
 from . import db_api
+from . import update_db
 from . import environment
 
-
-# @@@ Logging level should be configurable (with some structure to setting up
-# logging).
-logging.config.dictConfig({
-    "version": 1,
-    "handlers": {
-        "console": {
-            "class": "logging.StreamHandler",
-            "level": logging.INFO,
-            "stream": "ext://sys.stderr",
-        },
-    },
-    "root": {
-        "level": logging.DEBUG,
-        "handlers": ["console"],
-    },
-})
-log = logging.getLogger()
-
-
-def get_config_file():
-    # get the config file option for neo4j server location and web port number
-    _ROOT = os.path.abspath(os.path.dirname(__file__))
-
-    def get_data(path, file_name):
-        return os.path.join(_ROOT, path, file_name)
-
-    home_config = environment.HOME_CONFIG_FILE
-    env_config = os.environ.get(environment.ENV_CONFIG_FILE, "")
-    example_config = environment.DEFAULT_CONFIG_FILE
-
-    try:
-        conffile = next(p
-                        for p in (home_config, env_config, example_config)
-                        if os.path.exists(p))
-    except StopIteration:
-        #No config files accessable
-        if environment.ENV_CONFIG_FILE in os.environ:
-            #SEXTANT_CONFIG environment variable is set
-            log.error("{} file %r doesn't exist.".format(environment.ENV_CONFIG_FILE), env_config)
-        log.error("Sextant requires a configuration file.")
-        sys.exit(1)
-
-    log.info("Sextant is using config file {}".format(conffile))
-    return conffile
-
-conffile = get_config_file()
-
-conf = ConfigParser.ConfigParser()
-conf.read(conffile)
-
-remote_neo4j = 'http://localhost:7474'
-web_port = 2905
-common_def = 10  # definition of a 'common' node
-try:
-    options = conf.options('Neo4j')
-except ConfigParser.NoSectionError:
-    pass
-else:
-    try:
-        remote_neo4j = conf.get('Neo4j', 'remote_neo4j')
-    except ConfigParser.NoOptionError:
-        pass
-
-    try:
-        web_port = conf.get('Neo4j', 'port')
-    except ConfigParser.NoOptionError:
-        pass
-
-    try:
-        common_def = conf.get('Neo4j', 'common_function_calls')
-    except ConfigParser.NoOptionError:
-        common_def = 10
-
-argumentparser = argparse.ArgumentParser(description="Invoke part of the SEXTANT program")
-subparsers = argumentparser.add_subparsers(title="subcommands")
-
-#set what will be defined as a "common function"
-db_api.set_common_cutoff(common_def)
-
-parsers = dict()
-
-# add each subparser in turn to the parsers dictionary
-
-parsers['add'] = subparsers.add_parser('add-program',
-                                       help="add a program to the database")
-parsers['add'].add_argument('input_file', metavar="FILE_NAME",
-                            help="name of file to be put into database",
-                            type=str)
-parsers['add'].add_argument('--name-in-db', metavar="PROGRAM_NAME",
-                            help="string to store this program under", type=str,
-                            nargs=1)
-parsers['add'].add_argument('--not-object-file',
-                            help='default False, if the input file is an '
-                                 'object to be disassembled',
-                            action='store_true')
-
-parsers['delete'] = subparsers.add_parser('delete-program',
-                                          help="delete a program from the database")
-parsers['delete'].add_argument('program_name', metavar="PROG_NAME",
-                               help="name of program as stored in the database",
-                               type=str)
-
-parsers['query'] = subparsers.add_parser('query',
-                                         help="make a query of the database")
-parsers['query'].add_argument('query', metavar="QUERY",
-                              help="functions-calling, functions-called-by, "
-                                   "all-call-paths, whole-program, "
-                                   "shortest-call-path, programs or "
-                                   "functions; if the latter, "
-                                   "supply argument --program",
-                              type=str)
-parsers['query'].add_argument('--program', metavar="PROG_NAME",
-                              help="name of program as stored in the database; "
-                                   "required for all queries except 'programs'",
-                              type=str, nargs=1)
-parsers['query'].add_argument('--funcs', metavar='FUNCS',
-                              help='functions to pass to the query',
-                              type=str, nargs='+')
-parsers['query'].add_argument('--suppress-common', metavar='BOOL',
-                              help='suppress commonly called functions (True or False)',
-                              type=str, nargs=1)
-
-parsers['web'] = subparsers.add_parser('web', help="start the web server")
-parsers['web'].add_argument('--port', metavar='PORT', type=int,
-                            help='port on which to serve Sextant Web',
-                            default=web_port)
-
-parsers['audit'] = subparsers.add_parser('audit', help='view usage of Sextant')
-
-for parser_key in parsers:
-    parsers[parser_key].add_argument('--remote-neo4j', metavar="URL",
-                                     help="URL of neo4j server", type=str,
-                                     default=remote_neo4j)
+config = environment.load_config()
+
+
+def _displayable_url(args):
+    """
+    Return the URL specified by the user for Sextant to look at.
+
+    This is needed because we may be using SSH (that is, Sextant sees a
+    localhost port for the Neo4j server, which ssh is forwarding somewhere
+    else); in this case, we want to display the URL that the user thinks
+    Sextant is using, rather than the URL it's actually using.
+
+    :param args: the argparse object Sextant was invoked with
+    :return: the URL the Sextant invoker expects Sextant to be using
+
+    """
+    try:
+        if args.display_neo4j:
+            return args.display_neo4j
+    except AttributeError:
+        return args.remote_neo4j
+
+    return args.remote_neo4j
+
+
+# Beginning of functions which handle the actual invocation of Sextant
+
+def _start_web(args):
+    # Don't import at top level - makes twisted dependency semi-optional,
+    # allowing non-web functionality to work with Python 3.
+    if sys.version_info[0] == 2:
+        from .web import server
+    else:  # twisted won't be available - Python 2 required
+        logging.error('Web server must be run on Python 2.')
+        return
+    logging.info("Serving site on port {}".format(args.port))
+
+    # server is .web.server, imported a couple of lines ago
+    server.serve_site(input_database_url=args.remote_neo4j, port=args.port)
+
 
 def _audit(args):
     try:
         audited = query.audit(args.remote_neo4j)
     except requests.exceptions.ConnectionError as e:
-        logging.error('Connection error to server {}: {}'.format(args.remote_neo4j, e))
+        msg = 'Connection error to server {url}: {exception}'
+        logging.error(msg.format(url=_displayable_url(args)), exception=e)
 
     if not audited:
-        print('No programs on database at {}.'.format(args.remote_neo4j))
+        location = _displayable_url(args)
+        logging.warning('No programs on database at {}.'.format(location))
     else:
         for program in audited:
-            st = 'Program {} with {} functions uploaded by {} (id {}) on {}.'
-            print(st.format(program.program_name, program.number,
-                            program.uploader, program.uploader_id,
-                            program.date))
-    
-parsers['audit'].set_defaults(func=_audit)
-
-def _start_web(args):
-    # Don't import at top level -- this makes twisted dependency semi-optional,
-    # allowing non-web functionality to work with Python 3.
-    try:
-        from .web import server
-    except ImportError as e:  # twisted wasn't available - Python 2
-        log.error('Web server must be run on Python 2.')
-        log.error(e)
-        return 
-    log.info("Serving site on port %s", args.port)
-    server.serve_site(input_database_url=args.remote_neo4j, port=args.port)
-
-parsers['web'].set_defaults(func=_start_web)
-
-def add_file(namespace):
-    try:
-        alternative_name = namespace.name_in_db[0]
+            st = ('Program {progname} with {numfuncs} functions '
+                  'uploaded by {uploader} (id {uploaderid}) on {date}.')
+            print(st.format(progname=program.program_name,
+                            numfuncs=program.number,
+                            uploader=program.uploader,
+                            uploaderid=program.uploader_id,
+                            date=program.date))
+
+
+def _add_program(args):
+    try:
+        alternative_name = args.name_in_db[0]
     except TypeError:
         alternative_name = None
 
-    not_object_file = namespace.not_object_file
+    not_object_file = args.not_object_file
     # the default is "yes, this is an object file" if not-object-file was
-    # unsupplied=
+    # unsupplied
+
     try:
-        update_db.upload_program(namespace.input_file[0],
-                                 namespace.remote_neo4j,
-                                 alternative_name,
-                                 not_object_file)
+        update_db.upload_program(file_path=args.input_file,
+                                 db_url=args.remote_neo4j,
+                                 alternative_name=alternative_name,
+                                 not_object_file=not_object_file,
+                                 display_url=_displayable_url(args))
     except requests.exceptions.ConnectionError as e:
-        logging.error('Connection error to server {}: {}'.format(namespace.remote_neo4j,
-                                                                 e))
+        msg = 'Connection error to server {}: {}'
+        logging.error(msg.format(_displayable_url(args), e))
 
     except IOError as e:
         # in case of Python 2, where FileNotFoundError is undefined
-        logging.error('Input file {} was not found.'.format(namespace.input_file[0]))
+        # note: ConnectionError subclasses IOError, so must come first
+        logging.error('Input file {} was not found.'.format(args.input_file[0]))
         logging.error(e)
         logging.debug(e, exc_info=True)
     except ValueError as e:
         logging.error(e)
 
-def delete_file(namespace):
+
+def _delete_program(namespace):
     update_db.delete_program(namespace.program_name,
                              namespace.remote_neo4j)
 
-parsers['add'].set_defaults(func=add_file)
-parsers['delete'].set_defaults(func=delete_file)
-
-
-def make_query(namespace):
+
+def _make_query(namespace):
     arg1 = None
     arg2 = None
     try:
@@ -245,17 +145,276 @@
         suppress_common = False
 
     query.query(remote_neo4j=namespace.remote_neo4j,
+                display_neo4j=_displayable_url(namespace),
                 input_query=namespace.query,
-                program_name=program_name, argument_1=arg1, argument_2=arg2,
+                program_name=program_name,
+                argument_1=arg1, argument_2=arg2,
                 suppress_common=suppress_common)
 
-parsers['query'].set_defaults(func=make_query)
-
-# parse the arguments
-
-parsed = argumentparser.parse_args()
-
-# call the appropriate function
-
-parsed.func(parsed)
+# End of functions which invoke Sextant
+
+def parse_arguments():
+    """
+    Parses the command-line arguments to Sextant.
+
+    The resulting object, result, has a .func property, which is a method to be
+    called with result as its only parameter. This .func method runs whichever
+    of Sextant's functionality is appropriate.
+    :return: namespace summarising the arguments
+
+    """
+
+    argumentparser = argparse.ArgumentParser(description="Invoke part of the SEXTANT program")
+    subparsers = argumentparser.add_subparsers(title="subcommands")
+
+    #set what will be defined as a "common function"
+    db_api.set_common_cutoff(config.common_cutoff)
+
+    parsers = dict()
+
+    # add each subparser in turn to the parsers dictionary
+
+    parsers['add'] = subparsers.add_parser('add-program',
+                                           help="add a program to the database")
+    parsers['add'].add_argument('input_file', metavar="FILE_NAME",
+                                help="name of file to be put into database",
+                                type=str)
+    parsers['add'].add_argument('--name-in-db', metavar="PROGRAM_NAME",
+                                help="string to store this program under", type=str,
+                                nargs=1)
+    parsers['add'].add_argument('--not-object-file',
+                                help='default False, if the input file is an '
+                                     'object to be disassembled',
+                                action='store_true')
+
+    parsers['delete'] = subparsers.add_parser('delete-program',
+                                              help="delete a program from the database")
+    parsers['delete'].add_argument('program_name', metavar="PROG_NAME",
+                                   help="name of program as stored in the database",
+                                   type=str)
+
+    parsers['query'] = subparsers.add_parser('query',
+                                             help="make a query of the database")
+    parsers['query'].add_argument('query', metavar="QUERY",
+                                  help="functions-calling, functions-called-by, "
+                                       "all-call-paths, whole-program, "
+                                       "shortest-call-path, programs or "
+                                       "functions; if the latter, "
+                                       "supply argument --program",
+                                  type=str)
+    parsers['query'].add_argument('--program', metavar="PROG_NAME",
+                                  help="name of program as stored in the database; "
+                                       "required for all queries except 'programs'",
+                                  type=str, nargs=1)
+    parsers['query'].add_argument('--funcs', metavar='FUNCS',
+                                  help='functions to pass to the query',
+                                  type=str, nargs='+')
+    parsers['query'].add_argument('--suppress-common', metavar='BOOL',
+                                  help='suppress commonly called functions (True or False)',
+                                  type=str, nargs=1)
+
+    parsers['web'] = subparsers.add_parser('web', help="start the web server")
+    parsers['web'].add_argument('--port', metavar='PORT', type=int,
+                                help='port on which to serve Sextant Web',
+                                default=config.web_port)
+
+    parsers['audit'] = subparsers.add_parser('audit', help='view usage of Sextant')
+
+    for key in parsers:
+        parsers[key].add_argument('--remote-neo4j', metavar="URL",
+                                  help="URL of neo4j server", type=str,
+                                  default=config.remote_neo4j)
+        parsers[key].add_argument('--use-ssh-tunnel', metavar="BOOL", type=str,
+                                  help="whether to SSH into the remote server,"
+                                       "True/False",
+                                  default=str(config.use_ssh_tunnel))
+
+    parsers['audit'].set_defaults(func=_audit)
+    parsers['web'].set_defaults(func=_start_web)
+    parsers['add'].set_defaults(func=_add_program)
+    parsers['delete'].set_defaults(func=_delete_program)
+    parsers['query'].set_defaults(func=_make_query)
+
+    # parse the arguments
+
+    return argumentparser.parse_args()
+
+
+def _start_tunnel(local_port, remote_host, remote_port, ssh_user=''):
+    """
+    Creates an SSH port-forward.
+
+    This will result in localhost:local_port appearing to be
+    remote_host:remote_port.
+
+    :param local_port: integer port number to open at localhost
+    :param remote_host: string address of remote host (no port number)
+    :param remote_port: port to 'open' on the remote host
+    :param ssh_user: user to log in on the remote_host as
+
+    """
+
+    if not (isinstance(local_port, int) and local_port > 0):
+        raise ValueError(
+            'Local port {} must be a positive integer.'.format(local_port))
+    if not (isinstance(remote_port, int) and remote_port > 0):
+        raise ValueError(
+            'Remote port {} must be a positive integer.'.format(remote_port))
+
+    logging.debug('Starting SSH tunnel...')
+
+    # this cmd string will be .format()ed in a few lines' time
+    cmd = ['ssh']
+
+    if ssh_user:
+        # ssh -l {user} ... sets the remote login username
+        cmd += ['-l', ssh_user]
+
+    # -L localport:localhost:remoteport forwards the port
+    # -M makes SSH able to accept slave connections
+    # -S sets the location of a control socket (in this case, sextant-controller
+    #    with a unique identifier appended, just in case we run sextant twice
+    #    simultaneously), so we know how to close the port again
+    # -f goes into background; -N does not execute a remote command;
+    # -T says to remote host that we don't want a text shell.
+    cmd += ['-M',
+            '-S', 'sextantcontroller{tunnel_id}'.format(tunnel_id=local_port),
+            '-fNT',
+            '-L', '{0}:localhost:{1}'.format(local_port, remote_port),
+            remote_host]
+
+    logging.debug('Running {}'.format(' '.join(cmd)))
+
+    exit_code = subprocess.call(cmd)
+    if exit_code:
+        raise OSError('SSH setup failed with error {}'.format(exit_code))
+
+    logging.debug('SSH tunnel created.')
+
+
+def _stop_tunnel(local_port, remote_host):
+    """
+    Tear down an SSH port-forward which was previously set up with start_tunnel.
+
+    We use local_port as an identifier.
+    :param local_port: the port on localhost we are using as the entrypoint
+    :param remote_host: remote host we tunnelled into
+
+    """
+
+    logging.debug('Shutting down SSH tunnel...')
+
+    # ssh -O sends a command to the slave specified in -S
+    cmd = ['ssh',
+           '-S', 'sextantcontroller{}'.format(local_port),
+           '-O', 'exit',
+           remote_host]
+
+    return_code = subprocess.call(cmd)
+    if return_code == 0:
+        logging.debug('Shut down successfully.')
+    else:
+        logging.warning(
+            'SSH tunnel shutdown returned error code {}'.format(return_code))
+        logging.warning(
+            'You may need to close sextantcontroller{} manually.'.format(local_port))
+
+
+def _is_port_used(port):
+    """
+    Checks with the OS to see whether a port is open.
+
+    Beware: port is passed directly to the shell. Make sure it is an integer.
+    We raise ValueError if it is not.
+    :param port: integer port to check for openness
+    :return: bool(port is in use)
+
+    """
+
+    # we follow http://stackoverflow.com/questions/2838244/get-open-tcp-port-in-python
+    if not (isinstance(port, int) and port > 0):
+        raise ValueError('port {} must be a positive integer.'.format(port))
+
+    sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+    try:
+        sock.bind(('127.0.0.1', port))
+    except socket.error as e:
+        if e.errno == 98:  # Address already in use
+            return True
+        raise
+
+    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
+
+    return False  # that is, the port is not used
+
+
+def _get_unused_port():
+    """
+    Returns a port number between 10000 and 50000 which is not currently open.
+
+    """
+
+    keep_going = True
+    while keep_going:
+        portnum = random.randint(10000, 50000)
+        keep_going = _is_port_used(portnum)
+    return portnum
+
+
+def _get_host_and_port(url):
+    """Given a URL as http://host:port, returns (host, port)."""
+    parsed = parse.urlparse(url)
+    return (parsed.hostname, parsed.port)
+
+
+def _is_localhost(host, port):
+    """Checks whether a host is an alias to localhost."""
+    return socket.getaddrinfo(host, port)[0][4][0] in ('127.0.0.1', '::1')
+
+
+def main():
+    args = parse_arguments()
+
+    if args.use_ssh_tunnel.lower() == 'true':
+        localport = _get_unused_port()
+
+        remotehost, remoteport = _get_host_and_port(args.remote_neo4j)
+
+        if _is_localhost(remotehost, remoteport):
+            # we are attempting to connect to localhost anyway, so we won't
+            # bother to SSH to it.
+            # There may be some ways the user can trick us into trying to SSH
+            # to localhost anyway, but this will do as a first pass.
+            # SSHing to localhost is undesirable because on my test computer,
+            # we get 'connection refused' if we try.
+            args.func(args)
+
+        else:  # we need to SSH
+            try:
+                _start_tunnel(localport, remotehost, remoteport,
+                              ssh_user=config.ssh_user)
+            except OSError as e:
+                logging.error(str(e))
+                return
+
+            try:
+                args.display_neo4j = args.remote_neo4j
+                args.remote_neo4j = 'http://localhost:{}'.format(localport)
+                args.func(args)
+            except KeyboardInterrupt:
+                # this probably happened because we were running Sextant Web
+                # and Ctrl-C'ed out of it
+                logging.info('Keyboard interrupt detected. Halting.')
+                pass
+
+            finally:
+                _stop_tunnel(localport, remotehost)
+
+    else:  # no need to set up the ssh, just run sextant
+        args.func(args)
+
+
+if __name__ == '__main__':
+    main()
+
 

=== modified file 'src/sextant/environment.py'
--- src/sextant/environment.py	2014-08-21 13:03:24 +0000
+++ src/sextant/environment.py	2014-08-29 14:18:31 +0000
@@ -10,9 +10,17 @@
 """
 
 __all__ = ("RESOURCES_DIR", "DEFAULT_CONFIG_FILE", "ENV_CONFIG_FILE", 
-           "HOME_CONFIG_FILE")
+           "HOME_CONFIG_FILE", "load_config")
 
 import os
+import logging
+import logging.config
+import collections
+
+try:
+    import ConfigParser
+except ImportError:
+    import configparser as ConfigParser
 
 
 RESOURCES_DIR = ""         # where the resources folder is located
@@ -41,3 +49,92 @@
 RESOURCES_DIR = _find_folder('resources')
 _conf_path = _find_folder('etc')
 DEFAULT_CONFIG_FILE = os.path.join(_conf_path, 'sextant.conf')
+
+
+Config = collections.namedtuple('Config', ['web_port', 'remote_neo4j',
+                                           'common_cutoff', 'use_ssh_tunnel',
+                                           'ssh_user'])
+
+
+def load_config(check_here=''):
+    """
+    Loads in the config file (checking in the various locations).
+
+    If check_here is supplied as a location for the config file, it is used
+    preferentially. Then, in order, we use HOME_CONFIG_FILE, the contents of
+    the SEXTANT_CONFIG environment variable, and finally the bundled
+    etc/sextant.conf if no other config file was found.
+
+    :return: configuration object with properties for each config option
+    """
+
+    # @@@ Logging level should be configurable (with some structure to setting up
+    # logging).
+    logging.config.dictConfig({
+        "version": 1,
+        "handlers": {
+            "console": {
+                "class": "logging.StreamHandler",
+                "level": logging.WARNING,
+                "stream": "ext://sys.stderr",
+            },
+        },
+        "root": {
+            "level": logging.DEBUG,
+            "handlers": ["console"],
+        },
+    })
+    log = logging.getLogger()
+
+    def get_config_file(check=''):
+        env_config = os.environ.get(ENV_CONFIG_FILE, "")
+        try:
+            conffile = next(p
+                            for p in (check, HOME_CONFIG_FILE,
+                                      env_config, DEFAULT_CONFIG_FILE)
+                            if os.path.exists(p))
+        except StopIteration:
+            # No config files accessible
+            if ENV_CONFIG_FILE in os.environ:
+                # SEXTANT_CONFIG environment variable is set
+                log.error("{} file %r doesn't exist.".format(ENV_CONFIG_FILE), env_config)
+            raise Exception("Sextant requires a configuration file.")
+
+        log.info("Sextant is using config file {}".format(conffile))
+        return conffile
+
+    conf_file = get_config_file(check_here)
+
+    conf = ConfigParser.ConfigParser({'remote_neo4j': 'http://localhost:7474',
+                                      'port': '2905',
+                                      'common_function_calls': '10',
+                                      'use_ssh_tunnel': 'True',
+                                      'ssh_username': ''})
+    # It turns out that getint, getboolean etc want the defaults to be strings.
+    # When getboolean is called and the default is not a string, and the
+    # default is used, it complains that it can't work with non-strings.
+    # Ultimately, however, we output an object with int common_function_calls
+    # and port.
+
+    conf.read(conf_file)
+
+    remote_neo4j = ''
+    web_port = 0
+    common_def = 0  # definition of a 'common' node
+    use_ssh_tunnel = True
+    ssh_user = ''
+    try:
+        conf.options('Neo4j')
+    except ConfigParser.NoSectionError:
+        pass
+    else:
+        remote_neo4j = conf.get('Neo4j', 'remote_neo4j')
+        web_port = conf.getint('Neo4j', 'port')
+        common_def = conf.getint('Neo4j', 'common_function_calls')
+        use_ssh_tunnel = conf.getboolean('Neo4j', 'use_ssh_tunnel')
+        ssh_user = conf.get('Neo4j', 'ssh_username')
+
+    return Config(remote_neo4j=remote_neo4j, web_port=web_port,
+                  common_cutoff=common_def, use_ssh_tunnel=use_ssh_tunnel,
+                  ssh_user=ssh_user)
+

=== modified file 'src/sextant/export.py'
--- src/sextant/export.py	2014-08-26 11:04:04 +0000
+++ src/sextant/export.py	2014-08-29 14:18:31 +0000
@@ -33,7 +33,7 @@
         return name
 
     @staticmethod
-    def to_dot(program, suppress_common_nodes=False):
+    def to_dot(program, suppress_common_nodes=False, remove_self_calls=False):
         """
         Convert the program to DOT output format.
         program: Nodes and edges to be outputted

=== modified file 'src/sextant/query.py'
--- src/sextant/query.py	2014-08-26 11:04:04 +0000
+++ src/sextant/query.py	2014-08-29 14:18:31 +0000
@@ -14,12 +14,37 @@
 from .export import ProgramConverter
 
 
-def query(remote_neo4j, input_query, program_name=None, argument_1=None, argument_2=None, suppress_common=False):
+def query(remote_neo4j, input_query, display_neo4j='', program_name=None,
+          argument_1=None, argument_2=None, suppress_common=False):
+    """
+    Run a query against the database at remote_neo4j.
+
+    input_query may be in ('functions-calling', 'functions-called-by',
+    'all-call-paths', 'whole-program', 'shortest-call-path', 'functions',
+    'programs').
+
+    If we wish to display a different URL to the user than remote_neo4j,
+    display_neo4j should be used for that different URL.
+
+    :param remote_neo4j: location to sent HTTP requests to
+    :param input_query: string as detailed above
+    :param display_neo4j: Neo4j URL to display in messages to the user
+    :param program_name: string name of program
+    :param argument_1: string argument 1 to the query
+    :param argument_2: string argument 2 to the query
+    :param suppress_common: whether or not to suppress links to common functions
+
+    """
+
+    if display_neo4j:
+        display_url = display_neo4j
+    else:
+        display_url = remote_neo4j
 
     try:
         db = db_api.SextantConnection(remote_neo4j)
     except requests.exceptions.ConnectionError as err:
-        logging.error("Could not connect to Neo4J server {}. Are you sure it is running?".format(remote_neo4j))
+        logging.error("Could not connect to Neo4J server {}. Are you sure it is running?".format(display_url))
         logging.error(str(err))
         return 2
     #Not supported in python 2
@@ -38,50 +63,51 @@
     remove_self_calls = False
 
     if input_query == 'functions-calling':
-        if argument_1 == None:
+        if argument_1 is None:
             print('Supply one function name to functions-calling.')
             return 1
         prog = db.get_all_functions_calling(program_name, argument_1)
     elif input_query == 'functions-called-by':
-        if argument_1 == None:
+        if argument_1 is None:
             print('Supply one function name to functions-called-by.')
             return 1
         prog = db.get_all_functions_called(program_name, argument_1)
     elif input_query == 'all-call-paths':
-        if (argument_1 == None and argument_2 == None):
+        if argument_1 is None and argument_2 is None:
             print('Supply two function names to calls-between.')
             return 1
         prog = db.get_call_paths(program_name, argument_1, argument_2)
     elif input_query == 'whole-program':
         prog = db.get_whole_program(program_name)
     elif input_query == 'shortest-call-path':
-        if argument_1 == None and argument_2 == None:
+        if argument_1 is None and argument_2 is None:
             print('Supply two function names to shortest-path.')
             return 1
         prog = db.get_shortest_path_between_functions(program_name, argument_1, argument_2)
     elif input_query == 'functions':
-        if program_name != None:
+        if program_name is not None:
             func_names = db.get_function_names(program_name)
             if func_names:
                 names_list = list(func_names)
             else:
-                print('No functions were found in program %s on server %s.' % (program_name, remote_neo4j))
+                print('No functions were found in program %s on server %s.' % (program_name, display_url))
         else:
             list_of_programs = db.get_program_names()
             if not list_of_programs:
-                print('Server %s database empty.' % (remote_neo4j))
+                print('Server %s database empty.' % (display_url))
                 return 0
-            func_list = []
-            for prog_name in list_of_programs:
-                func_list += db.get_function_names(prog_name)
+
+            func_list = [db.get_function_names(prog_name)
+                         for prog_name in list_of_programs]
+
             if not func_list:
-                print('Server %s contains no functions.' % (remote_neo4j))
+                print('Server %s contains no functions.' % (display_url))
             else:
                 names_list = func_list
     elif input_query == 'programs':
         list_found = list(db.get_program_names())
         if not list_found:
-            print('No programs were found on server {}.'.format(remote_neo4j))
+            print('No programs were found on server {}.'.format(display_url))
         else:
             names_list = list_found
     else:
@@ -100,6 +126,3 @@
     db = db_api.SextantConnection(remote_neo4j)
 
     return db.programs_with_metadata()
-
-if __name__ == '__main__':
-    main()

=== modified file 'src/sextant/update_db.py'
--- src/sextant/update_db.py	2014-08-21 16:08:10 +0000
+++ src/sextant/update_db.py	2014-08-29 14:18:31 +0000
@@ -13,21 +13,27 @@
 import logging
 
 
-def upload_program(file_path, db_url, 
+def upload_program(file_path, db_url, display_url='',
                    alternative_name=None,
                    not_object_file=False):
     """
     Uploads a program to the remote database.
+
     Raises requests.exceptions.ConnectionError if the server didn't exist.
     Raises IOError if file_path doesn't correspond to a file.
     Raises ValueError if the desired alternative_name (or the default, if no
     alternative_name was specified) already exists in the database.
     :param file_path: the path to the local file we wish to upload
     :param db_url: the URL of the database (eg. http://localhost:7474)
+    :param display_url: alternative URL to display instead of db_url
     :param alternative_name: a name to give the program to override the default
     :param object_file: bool(the file is an objdump text output file, rather than a compiled binary)
+
     """
 
+    if not display_url:
+        display_url = db_url
+
     connection = SextantConnection(db_url)
 
     program_names = connection.get_program_names()
@@ -61,7 +67,7 @@
             if not program_representation.add_function_call(obj.name, called[-1]): # called is a tuple (address, name)
                 logging.error('Validation error: {} calling {}'.format(obj.name, called[-1]))
 
-    logging.info('Sending {} named objects to server {}...'.format(len(parsed_objects), db_url))
+    logging.info('Sending {} named objects to server {}...'.format(len(parsed_objects), display_url))
     program_representation.commit()
     logging.info('Sending complete! Exiting.')