== The new Twisted node daemon, lib/devscripts/ec2test/remotenode.py ==

Diff: http://pastebin.ubuntu.com/315123/

This daemon is meant to make the following possible:

1. Start up N instances (on EC2, but the daemon makes no assumptions
   on that front).

2. One of them is told it is the supervisor. It immediately builds the
   LP tree and calculates the complete list of tests to run. It puts
   these tests into N similarly sized parcels for workers to pick up
   later. It also starts the subunit results aggregator service.

3. Every instance is told where the supervisor is. These instances
   become workers, and ask the supervisor for work.

4. Each worker runs the tests it has been given, and streams the
   results back to the supervisor's result aggregator.

5. When a worker has finished its run, with or without error, it tells
   the supervisor that it has attempted to run every test in its
   parcel.

6. The supervisor keeps track of which tests it actually has results
   for; there seem to be some problems where workers don't actually
   run all of the tests they're asked to.

7. The worker asks for more work. If there isn't any, it shuts
   down.

8. Once the supervisor has responses for all parcels, it sends a
   comprehensive report email, then shuts down.

9. Another process (remotenodekiller.py) takes care of actually
   shutting each instance down, when either the node daemon has
   exited, or if the logs have not had any activity for 10 minutes.


Currently there is no support for submitting to PQM after a test run,
and the email address is hard-wired to <email address hidden>. I
think I might fix the latter before I land this branch :)

Anyway, the daemon has three services: controller, subunit results
aggregator, and inspection. The first two are needed to run the
testing service, the last helps during development, and for debugging.


=== Controller ===

The main class here is TestNode. This currently mixes in many
responsibilities that should be split out. It contains the logic for
being a supervisor (including starting the aggregator service and
sending report emails), being a worker, and shutting down. It is also,
in Perspective Broker parlance, a Referencable object, which means its
remote_* methods can be called remotely.

Ideally I would like to separate out concepts like the branch, the
supervisor and the worker, and then have a Referencable object to be
the remote control for them.


=== Subunit Aggregator ===

This is simply a socket that receives line-based output from a
remotely running bin/test process that's been asked to produce subunit
output (the --subunit flag). For each connection, it creates a
subunit.TestProtocolServer instance, to which it blindly passes the
input. This TestProtocolServer is given a results object that the
supervisor has created. This way all test results from all workers are
recorded into a single test result object.

The test result object also calls back to the supervisor for every
test that is run so that the supervisor can keep track of the tests
for which a result is available. It wouldn't be good if it submitted
to PQM if 1000 tests had never actually run.


=== Inspection service ===

This opens an SSH service on port 8791 (username: admin, password:
admin) that runs a REPL within the running python process. That way
it's possible to poke around and debug the daemon if it does something
unexpected.


== Getting to the point where we can run remotenode.py ==

Diff: http://pastebin.ubuntu.com/315124/

lib/devscripts/ec2test/account.py

  New commands parallel-test (implemented in cmd_parallel_test). This
  does not (yet) support the full range of options that the test
  command does. All the code is also inline in the run() method, but I
  think this should be okay for now; it can be refactored later. In my
  defence, the code is already in functions and that this seems to be
  a not uncommon practice for dealing with Twisted callbacks.

lib/devscripts/ec2test/instance.py

  New method _command_with_locale() to prevent stupid locale warnings
  from bzr, although it does add "export LC_ALL=C; " to every command
  run, so it's a trade off I guess.

  New method run_as_daemon() to start a daemon process by starting a
  background process within a bash subshell. This seems to work
  okay. It's here to support starting remotenodekiller.py.

lib/devscripts/ec2test/testrunner.py

  Always create a bazaar.conf on the remote node.

  New method, run_node() to start up the remotenode.py daemon.


== Utilities ==

Diff: http://pastebin.ubuntu.com/315127/

lib/devscripts/ec2test/remotenodekiller.py

  This script watches a specified list of pids, pid files and the pids
  contained therein, and a list of log files (subject to glob
  expansion) for activity. If a process stops running, or a pid file
  disappears, or a log file gets stale, a given command is run, which
  could, for example, shutdown the system.

utilities/pb-shell

  This gives the user a half decent REPL for interacting with the
  remotenode.py process (or any Perspective Broker service
  really). The REPL is started in a thread and it calls back into the
  Twisted reactor to communicate via PB.


== Cleanup ==

Diff: http://pastebin.ubuntu.com/315128/

lib/devscripts/ec2test/ec2test-remote.py => lib/devscripts/ec2test/remote.py

  This is the current/old script for running tests on EC2. Originally
  I put all the new Twisted code in this file, but it got out of hand,
  and just didn't feel right, so I ripped it back out again. However,
  Jono and I made a lot of cleanups to this file and it would be a
  shame to lose them.

  The main thing was to pass in the prefix directory to run under,
  /var by default. The idea is to make it easier to test locally.

  Also, the test() method has been broken into two pieces, run() and
  run_tests(). It's still a less than perfect split, but it's a little
  easier to understand now, IMO.

  Of course, I want the new parallelelelelel stuff to supersede this,
  but it's not ready for that yet.