Merge ~jugmac00/launchpad:add-explanation-for-codeimport into launchpad:master

[Jürgen Gmach]

Add explanation about how code imports work

[Simone Pelosi (pelpsi)]

LGTM!

review: Approve

[Guruprasad (lgp171188)]

LGTM! Approved with comments.

review: Approve

[Ines Almeida (ines-almeida)]

Suggest maybe adding an intro sentence for first time readers. Something along the lines of "Launchpad has the ability to constantly sync code from external repositories by setting up a code import" (and maybe also adding something about it being only one directional?)

[Jürgen Gmach (jugmac00)]

@ines I understand "explanation" in the diátaxis context as something that deepens the understanding, and not as an introduction. So you usually use them when you know already something about a topic. That was the reason why I did not add an introduction in the first place. But I think adding one does not hurt. Thanks for the suggestion.

diff --git a/doc/explanation/codeimport.rst b/doc/explanation/codeimport.rst
new file mode 100644
index 0000000..2f775da
--- /dev/null
+++ b/doc/explanation/codeimport.rst
@@ -0,0 +1,68 @@
+====================
+Code import in depth
+====================
+
+Launchpad has the ability to constantly sync code from external repositories by
+setting up a code import. This is a one-way sync, from the external source to
+Launchpad.
+
+Design decisions
+================
+
+Code import / code mirroring is not triggered by the remote repository,
+but it is rather triggered from the Launchpad side several times a day.
+This should be fine for most use cases.
+
+The benefit of doing it this way is there is no need to install a webhook on
+the remote repository to trigger an import or update of the repository.
+
+Frequency
+=========
+
+While the current implementation has some support for setting it on a
+per-import basis, that's actually not exposed right now.
+
+The frequency depends on the type of the revision control system, and is set in
+``effective_update_interval``,
+see https://git.launchpad.net/launchpad/tree/lib/lp/code/model/codeimport.py.
+
+As the default intervals in the config schema aren't overridden anywhere right
+now, you can look them up in the ``codeimport`` section in the configuration
+schema:
+https://git.launchpad.net/launchpad/tree/lib/lp/services/config/schema-lazr.conf.
+
+Trigger code import
+===================
+
+On a high level, each import has a database field, which indicates when the
+next import should be performed. We could think of this as task queue.
+
+A scheduler queries the database, and in case there are jobs, a worker will
+pick one up.
+
+Let's have a look in detail.
+
+After the import job runs, and if the import is still in the ``REVIEWED``
+state, i.e. it hasn't failed too many times, or it hasn't been suspended or
+similar, it then schedules a new job, that is, it sets a new date in the 
+``date_due`` column.
+
+Depending on the work load, there is no guarantee that a job will be picked up
+in time, as the process works like this:
+
+- cronjobs in ``lp-codeimport`` run workers
+- when the code import worker is looking for work to do, it calls
+  ``getJobForMachine``,
+  see https://git.launchpad.net/launchpad/tree/lib/lp/code/xmlrpc/codeimportscheduler.py
+- that calls ``getJobForMachine``
+  in https://git.launchpad.net/launchpad/tree/lib/lp/code/model/codeimportjob.py
+  which makes a DB query to fetch the first job from a table that has
+  ``due_date`` not in the future, and that is marked as ``pending``
+- it then triggers the import
+
+Conclusion
+==========
+
+The short answer is that we currently aim for every 12 hours for code imports
+from CVS (very few nowadays), and every 6 hours from all other revision control
+systems.
diff --git a/doc/explanation/index.rst b/doc/explanation/index.rst
index 21299fd..1ca8cb5 100644
--- a/doc/explanation/index.rst
+++ b/doc/explanation/index.rst
@@ -14,3 +14,5 @@ Explanation
    pip
    favicon
    charms
+   codeimport
+