Merge ~cjwatson/launchpad:doc-running into launchpad:master

[Colin Watson]

Add "Running Launchpad" tutorial from dev.launchpad.net

This is a conglomeration of three pages:

  https://dev.launchpad.net/Running
  https://dev.launchpad.net/Running/LXD
  https://dev.launchpad.net/Running/RemoteAccess

[Jürgen Gmach (jugmac00)]

Only made it halfway through until it got too late.

[Jürgen Gmach (jugmac00)]

LGTM

I try to see a pattern for the four different kinds of documentation Daniele told us about.

I think the "Details section" would belong to the "Explanation" mode.

That is nothing that needs to be done now.

[Jürgen Gmach (jugmac00)]

review: Approve

[Colin Watson (cjwatson)]

I think you're right - I pushed the "Details" section out to a separate document and added a link. How's this?

[Jürgen Gmach (jugmac00)]

Looks good!

review: Approve

diff --git a/doc/index.rst b/doc/index.rst
index ed599e7..1677578 100644
--- a/doc/index.rst
+++ b/doc/index.rst
@@ -22,6 +22,14 @@ Overview
    values
    faq
 
+Tutorials
+=========
+
+.. toctree::
+   :maxdepth: 1
+
+   running
+
 Guides
 ======
 
diff --git a/doc/running-details.rst b/doc/running-details.rst
new file mode 100644
index 0000000..aad600b
--- /dev/null
+++ b/doc/running-details.rst
@@ -0,0 +1,57 @@
+==============================
+Launchpad installation details
+==============================
+
+What the installation process does
+----------------------------------
+
+The ``rocketfuel-setup`` script first determines what release of Ubuntu
+you're running, then installs various lines into files under ``/etc``, to
+enable you to run Launchpad services locally.  For example, it adds entries
+for "launchpad.test", "bazaar.launchpad.test", "lists.launchpad.test", and
+others to your ``/etc/hosts`` file, so that after you build Launchpad you
+can browse to ``launchpad.test`` and see a locally-running instance.  It
+also installs some packages, dependencies that Launchpad needs in order to
+run.  This is why the ``sudo`` access is necessary; consult the script for
+details of what it's doing.
+
+Once it's got the system preparation out of the way, the script clones
+Launchpad's Git repository (that's the ``launchpad`` directory above).  That
+will take a while.
+
+After it gets that, it fetches the other dependencies, the third-party
+libraries, by invoking a separate script,
+``launchpad/utilities/rocketfuel-get``.  That will take a while too, as
+there are over 200 such libraries.
+
+Once it has all the dependencies, it links them into the trunk working tree,
+using the script ``launchpad/utilities/link-external-sourcecode``.
+
+Do-it-yourself installation
+---------------------------
+
+**We only support using rocketfuel-setup to set up Launchpad.**  It adjusts
+a lot of things to get the development process running smoothly, as
+summarized above.  However, sometimes you might want to just get a build of
+Launchpad to run its tests, or to run a script packaged with Launchpad, or
+to do your own manual changes of the files that ``rocketfuel-setup`` would
+normally touch.  These are the basics of what needs to be done for that
+route - **unsupported hints**.
+
+You'll need packages from a PPA: ``ppa:launchpad/ubuntu/ppa``.
+
+.. code-block:: shell-session
+
+    $ sudo apt-add-repository ppa:launchpad/ubuntu/ppa
+
+Install the ``launchpad-developer-dependencies`` package.
+
+Get the code:
+
+.. code-block:: shell-session
+
+    $ git clone https://git.launchpad.net/launchpad
+    $ cd launchpad
+    $ utilities/update-sourcecode
+    $ git clone --depth=1 https://git.launchpad.net/lp-source-dependencies download-cache
+    $ make
diff --git a/doc/running.rst b/doc/running.rst
new file mode 100644
index 0000000..304138d
--- /dev/null
+++ b/doc/running.rst
@@ -0,0 +1,351 @@
+=================
+Running Launchpad
+=================
+
+.. note::
+
+    `Ask for help <https://dev.launchpad.net/Help>`_ right away if you run
+    into problems.
+
+This page explains how to set up and run Launchpad (for development) on your
+own machine, using a `LXD
+<https://linuxcontainers.org/lxd/introduction/>`_-managed container to
+isolate it from the rest of your system.
+
+Supported operating systems
+===========================
+
+For the host system, any reasonably modern Ubuntu release should work.
+Other Linux distributions that have LXD should work too, though we don't
+test on them.
+
+For the containers, these instructions should work with Ubuntu 16.04 LTS or
+later.  We currently test on 16.04 and 18.04, with the aim of upgrading
+production to 18.04 soon.  20.04 is known not to work yet.
+
+Launchpad requires at least Python 3.5 (i.e. newer than Ubuntu 14.04 LTS).
+
+We'd like Launchpad to run on other operating systems, especially `Debian
+GNU/Linux <https://www.debian.org/>`_, so that more people can contribute to
+Launchpad development.  If you're interested in working on Launchpad
+portability, please `let us know <https://dev.launchpad.net/Help>`_.  Note
+that our focus is on getting Launchpad to build easily so more people can
+participate in Launchpad development.  Running a stable production instance
+would be *much* harder than running a single-developer test instance, and we
+don't recommend it.  Unlike many open source projects, we're not seeking to
+maximize the number of installations; our goal is to improve the `instance
+we're already running <https://launchpad.net/>`_.
+
+Why use LXD containers?
+=======================
+
+Launchpad development setup makes significant changes to your machine; it's
+nice to be unaffected by those when you're not doing such development.
+Using containers means that the version of Ubuntu on your host system
+doesn't need to match Launchpad's requirements.  Multiple containers can be
+used to work around Launchpad's limitations regarding concurrent test runs
+on a single machine.
+
+LXD also has some nice snapshotting and ZFS capabilities, and is what other
+Launchpad developers use, so other people will be able to help you debug
+your setup if needed.
+
+The installation script is written with the assumption that it will be
+running inside such a container.  If you run it on your host system, it may
+well break your host system.
+
+Create a LXD container
+======================
+
+This assumes you already have LXD set up.  If not, follow the `instructions
+<https://linuxcontainers.org/lxd/getting-started-cli/>`_ for getting it
+installed and configured on your network.
+
+1. If you haven't done so already, run this script to set up LXD to let you
+   use your home directory inside the container.  Some developers prefer to
+   only mount a subdirectory of their home directory in the container: to do
+   that, replace ``$HOME`` with ``$HOME/src`` or similar.
+
+.. code-block:: sh
+
+    #! /bin/sh
+    
+    id=400000  # some large uid outside of typical range, and outside of already mapped ranges in /etc/sub{u,g}id
+    uid=$(id -u)
+    gid=$(id -g)
+    user=$(id -un)
+    group=$(id -gn)
+    
+    # give lxc permission to map your user/group id through
+    sudo usermod --add-subuids ${uid}-${uid} --add-subgids ${gid}-${gid} root
+    
+    # create a profile to control this
+    lxc profile create $user >/dev/null 2>&1
+    
+    # configure profile
+    cat << EOF | lxc profile edit $user
+    name: $user
+    description: allow home dir mounting for $user
+    config:
+      raw.idmap: |
+        uid $uid $id
+        gid $gid $id
+      user.user-data: |
+        #cloud-config
+        runcmd:
+          - "groupadd $group --gid $id"
+          - "useradd $user --uid $id --gid $group --groups adm,sudo --shell /bin/bash"
+          - "echo '$user ALL=(ALL) NOPASSWD:ALL' >/etc/sudoers.d/90-cloud-init-users"
+          - "chmod 0440 /etc/sudoers.d/90-cloud-init-users"
+    devices:
+      home:
+        type: disk
+        source: $HOME
+        path: $HOME
+    EOF
+
+2. Create a container.  This command creates a Ubuntu 16.04 unprivileged
+   container using the profile created in the previous step.
+
+.. code-block:: sh
+
+    lxc launch ubuntu:16.04 lpdev -p default -p $USER
+
+3. Find the container IP, either from ``lxc list`` or ``lxc info lpdev``.
+
+4. In order to be able to ssh into the container, you need to add your
+   public key to your local ``.ssh/authorized_keys`` configuration.  Also
+   make sure that both ``.ssh`` (700) and ``authorized_keys`` (600) have the
+   correct permissions.
+
+5. Connect as follows.  (The -A permits you to access Launchpad code hosting
+   from within the container without needing to reenter passphrases.)
+
+.. code-block:: sh
+
+    ssh -A $user@IP_ADDRESS_FROM_LXC_LS
+
+Getting Launchpad
+=================
+
+Do all this *inside* the container you set up previously.  Be aware that
+changes in your home directory inside the container will also be seen
+outside the container and vice versa.
+
+If your Launchpad username differs from your local one, then put this in
+``~/.ssh/config`` in the container before doing anything else, replacing
+``LPUSERNAME`` with your Launchpad username::
+
+    Host bazaar.launchpad.net
+            User LPUSERNAME
+    Host git.launchpad.net
+            User LPUSERNAME
+
+Then:
+
+.. code-block:: shell-session
+
+   $ mkdir ~/launchpad
+   $ cd ~/launchpad
+   $ curl https://git.launchpad.net/launchpad/plain/utilities/rocketfuel-setup >rocketfuel-setup
+
+Read through the rocketfuel-setup script at this point and make sure you're
+OK with what it's going to do.  (See :doc:`running-details` if you want to
+know more.)
+
+.. code-block:: shell-session
+
+   $ chmod a+x rocketfuel-setup
+   $ ./rocketfuel-setup
+
+This will take a while -- maybe a few hours to get everything, depending on
+your Internet connection.
+
+Note that you will be prompted for your ``sudo`` password, and for a
+Launchpad login ID (that is, your username on ``launchpad.net``).  The sudo
+access is necessary to get Launchpad running on your box; the Launchpad
+login is not strictly necessary, and you can just hit Return there if you
+want.  See below for an explanation.
+
+Note that this will make changes to your Apache configuration if you already
+have an Apache server in your container.  It will also add entries to
+``/etc/hosts``, and it will setup a PostgreSQL server in your container.
+
+If you are running ``rocketfuel-setup`` to bring up a new container but your
+home directory already has a usable Launchpad tree, you can pass
+``--no-workspace`` to only perform the system-wide setup.
+
+Note that if ``rocketfuel-setup`` bails out with instructions to fix
+something, you just need to run it again and it should pick up where it left
+off.
+
+.. code-block:: shell-session
+
+   $ sudo apt-get dist-upgrade
+
+This is just to make doubly-sure everything from the Launchpad PPA gets
+installed.
+
+.. code-block:: shell-session
+
+   $ ls
+   launchpad/    lp-sourcedeps/
+   $ cd launchpad
+
+You are now in a newly-cloned Git repository, with one branch ('master'),
+into whose working tree the other source dependencies have been symlinked.
+The source dependencies actually live in ``../lp-sourcedeps``.
+
+Installing the pre-commit hook
+==============================
+
+If you intend to make any changes to Launchpad, you should also set up
+`pre-commit <https://pre-commit.com/>`_ now:
+
+1. Install ``pre-commit`` itself.  If your host system is Ubuntu 20.10 or
+   newer, then ``sudo apt install pre-commit`` is enough; otherwise, you can
+   install it in your user account (`pipx <https://pypi.org/project/pipx/>`_
+   works well to keep it isolated; whatever you do, don't run ``pip``
+   system-wide as root!).  We require this to be installed separately rather
+   than including it in Launchpad's virtual environment because developers
+   commonly run ``git commit`` outside the container used for running
+   Launchpad.
+
+2. Install the ``pre-commit`` git hook by running ``pre-commit install`` in
+   your newly-cloned ``launchpad`` repository.
+
+Building
+========
+
+Before you can run Launchpad for the first time, you need to set up PostgreSQL.
+
+.. note::
+
+    **DO NOT run the database setup script below if you use PostgreSQL for
+    anything other than Launchpad!**  Running the script will destroy any
+    PostgreSQL databases on your system.  See
+    https://dev.launchpad.net/DatabaseSetup for details.
+
+.. code-block:: shell-session
+
+    $ ./utilities/launchpad-database-setup $USER
+
+**(Please have read the previous comment before you run the above command!)**
+
+Finally, build the database schema (this may take several minutes):
+
+.. code-block:: shell-session
+
+    $ make schema
+
+Running
+=======
+
+Now you should be able to start up Launchpad:
+
+.. code-block:: shell-session
+
+    $ make run
+
+This only runs the basic web application.  `Codehosting
+<Code/HowToUseCodehostingLocally>`_ and `Soyuz
+<Soyuz/HowToUseSoyuzLocally>`_ require additional steps.
+
+For subsequent builds, you can just do ``make run`` right away.  You don't
+need to do ``make schema`` every time, and you should avoid it because it's
+expensive and because it will clean out any data you might have put into
+your test instance (through the web UI or by running other scripts).
+
+Setting up remote access
+========================
+
+Unless the only thing you're doing is running parts of the test suite, you
+probably want to make your new Launchpad instance accessible from other
+machines on the same local network, or in particular from the host system.
+
+Amending the Apache configuration
+---------------------------------
+
+Launchpad's default development Apache config
+(``/etc/apache2/sites-available/local-launchpad.conf``) only listens on
+127.0.0.88.  This can be overridden with the ``LISTEN_ADDRESS`` environment
+variable when running ``make install``.  You probably want to make it listen
+on everything:
+
+.. code-block:: shell-session
+
+    $ sudo make LISTEN_ADDRESS='*' install
+
+Amending the hosts file
+-----------------------
+
+Launchpad makes extensive use of virtual hosts, so you'll need to add
+entries to ``/etc/hosts`` on any machine from which you want to access the
+Launchpad instance.  You'll see the relevant hostnames in ``/etc/hosts`` on
+the machine running the instance - they need to be added to the remote
+machine, mapped to the server machine or container's external IP address.
+
+If some of those other machines run Windows, it may be helpful to know that
+the Windows equivalent of ``/etc/hosts`` is located at
+``C:\WINDOWS\system32\drivers\etc\hosts``.  Note that Windows' version has a
+line length limit, so you might have to split it across multiple lines or
+only include the hostnames that you need.
+
+You should now be able to access ``https://launchpad.test/`` in a web
+browser on a suitably configured remote computer.  Accept the local
+self-signed certificate.  You can log in as ``[e-mail]`` without
+a password.  (This is only for development convenience, and assumes that you
+trust machines that can route to your LXD containers; of course a production
+deployment would need real authentication.)
+
+Accessing launchpad.test from a single host over SSH
+----------------------------------------------------
+
+As an alternative to the above, SSH provides a SOCKS proxy.  By running that
+proxy on the target machine, you can view its Launchpad web site as if you
+were on that machine, without having to open non-SSH ports to a wider
+network.  To do so:
+
+.. code-block:: shell-session
+
+    $ ssh -D8110 target-machine
+
+Then set your browser's SOCKS proxy settings to use ``target-machine:8110``.
+
+Stopping
+========
+
+You can stop Launchpad by hitting **Control-C** in the terminal where you
+started it:
+
+.. code-block:: shell-session
+
+    ^C
+    [...shutting down Launchpad...]
+    $ 
+
+Or you can be at a prompt in the same directory and run this:
+
+.. code-block:: shell-session
+
+    $ make stop
+
+Troubleshooting
+===============
+
+"The LXC container is not getting an IPv4 address assigned and the network
+connectivity inside the container doesn't work."
+
+On Ubuntu 21.10, ``ufw`` uses ``nftables`` by default, so if you are using
+Ubuntu 21.10 on the host and ``ufw`` is enabled with the default policy of
+blocking incoming and routed traffic, the rules added by LXD will not take
+effect, and hence LXD's traffic will be dropped.
+
+The fix is to add ``ufw allow`` rules to allow incoming and routed traffic
+on the bridge interface, like this (replacing ``lxdbr0`` with the name of
+the bridge interface on your computer):
+
+.. code-block:: sh
+
+    sudo ufw allow in on lxdbr0
+    sudo ufw route allow in on lxdbr0