Merge lp:~dernils/duplicity/Dockerfile into lp:~duplicity-team/duplicity/0.8-series

[nils]

I added another backend to the docker test infrastructure, updated the setup for testing and updated the documentation.

[Kenneth Loafman (kenneth-loafman)]

A couple of changes we need to make:

1) Your docker image does not match the released Dockerfile. No user, and
in the wrong directory.
2) We need to have an override for the 10.10/16 subnet. It conflicts with
one of mine, so it'll probably hit someone else. A simple define at the
top of the setup would do for now.
3) Looking good! I'll see about getting a duplicity user on Docker Hub.

On Sun, May 14, 2017 at 10:35 AM, <email address hidden> wrote:

> The proposal to merge lp:~dernils/duplicity/Dockerfile into lp:duplicity
> has been updated.
>
> Status: Needs review => Merged
>
> For more details, see:
> https://code.launchpad.net/~dernils/duplicity/Dockerfile/+merge/324027
> --
> Your team duplicity-team is requested to review the proposed merge of
> lp:~dernils/duplicity/Dockerfile into lp:duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

[nils (dernils)]

inline

Kenneth Loafman <email address hidden> schrieb am Mo., 15. Mai 2017 um
01:57 Uhr:

> A couple of changes we need to make:
>
> 1) Your docker image does not match the released Dockerfile. No user, and
> in the wrong directory.
>
NT: I just did a fresh build and upload. Should be consistent now .

> 2) We need to have an override for the 10.10/16 subnet. It conflicts with
> one of mine, so it'll probably hit someone else. A simple define at the
> top of the setup would do for now.
>
NT: good point. I just introduced a the net and IP as a variable. Will
figure out a way to pass it on into the containers (most likely via an
envrionmental variable) as the next step.

> 3) Looking good! I'll see about getting a duplicity user on Docker Hub.
>
NT: That is a very good idea.

>
> On Sun, May 14, 2017 at 10:35 AM, <email address hidden> wrote:
>
> > The proposal to merge lp:~dernils/duplicity/Dockerfile into lp:duplicity
> > has been updated.
> >
> > Status: Needs review => Merged
> >
> > For more details, see:
> > https://code.launchpad.net/~dernils/duplicity/Dockerfile/+merge/324027
> > --
> > Your team duplicity-team is requested to review the proposed merge of
> > lp:~dernils/duplicity/Dockerfile into lp:duplicity.
> >
> > _______________________________________________
> > Mailing list: https://launchpad.net/~duplicity-team
> > Post to : <email address hidden>
> > Unsubscribe : https://launchpad.net/~duplicity-team
> > More help : https://help.launchpad.net/ListHelp
> >
>
> --
> https://code.launchpad.net/~dernils/duplicity/Dockerfile/+merge/324027
> You are the owner of lp:~dernils/duplicity/Dockerfile.
>

=== modified file 'README-TESTING'
--- README-TESTING	2017-04-23 15:04:49 +0000
+++ README-TESTING	2017-05-14 07:45:09 +0000
@@ -39,7 +39,6 @@
 code against different supported environments, including the versions of
 dependencies used by the Launchpad build system.
 
-
 A tox run can be started simply by typing 
 
 ‘tox‘
@@ -95,6 +94,45 @@
 
 The report will be generated and stored in the folder htmlcov. 
 
-
-
+## The wider picture - supporting containers for testing
+Testing duplicity invokes backends. The backends are the places where the backup data is actually stored (e.g. an ftp server). In order to have the highest degree of control over the testing process, backends that can be set up locally are also operated in separated docker containers. The whole test infrastructure is shown in the following picture. 
+
+┌─────────────────────┐         ┌──────────────────────────────────────────┐   
+│docker image         │         │docker image                              │   
+│dernils/duplicitytest│         │dernils/duplicity_testinfrastructure_ssh  │   
+├─────────────────────┴────┐    ├──────────────────────────────────────────┴──┐
+│                          │    │                                             │
+│  ┌──────────────────┐    │    │   ┌──────────────────┐                      │
+│  │       tox        │    │ ┌──┼──▶│       sshd       │                      │
+│  └──────────────────┘    │ │  │   └──────────────────┘                      │
+│            │             │ │  │                                             │
+│            ▼             │ │  └─────────────────────────────────────────────┘
+│  ┌──────────────────┐    │ │  ┌──────────────────────────────────────────┐   
+│  │    unittests     │    │ │  │docker image                              │   
+│  └──────────────────┘    │ │  │dernils/duplicity_testinfrastructure_ftp  │   
+│            │             │ │  ├──────────────────────────────────────────┴──┐
+│            ▼             │ │  │                                             │
+│  ┌──────────────────┐    │ │  │   ┌──────────────────┐                      │
+│  │    duplicity     │◀───┼─┴──┼──▶│    pure-ftpd     │                      │
+│  └──────────────────┘    │    │   └──────────────────┘                      │
+│            │             │    │                                             │
+└────────────┼─────────────┘    └─────────────────────────────────────────────┘
+             │                                                                 
+             │                                                                 
+             │                                                                 
+             └────────────┐                                                    
+                          │                                                    
+       Internet .─────────┼─────────.                                          
+         _.────'          │          `─────.                                   
+     _.─'                 │                 `──.                               
+   ,'                     ▼                     `.                             
+  ;             ┌──────────────────┐              :                            
+  :             │     Dropbox      │              ;                            
+   ╲            └──────────────────┘             ╱                             
+    `.                                         ,'                              
+      `──.                                 _.─'                                
+          `─────.                   _.────'                                    
+                 `─────────────────'                                           
+
+The docker images that contain the test infrastructure are defined in the folder /testing/infrastructure. There is a build script to compile the Dockerfile into actual images (build-duplicitiy-test.sh). However, as all images are also published on the docker hub, it is not necessary to build the images before starting testing. Testing can directly be started by using the script setup.sh. If the required docker images are not yet existing, locally, they will be downloaded by Docker. 
 

=== modified file 'testing/infrastructure/build-duplicity_test.sh'
--- testing/infrastructure/build-duplicity_test.sh	2017-05-06 20:09:20 +0000
+++ testing/infrastructure/build-duplicity_test.sh	2017-05-14 07:45:09 +0000
@@ -2,5 +2,20 @@
 
 cd duplicity_test
 cp -p ../../../requirements.txt .
+cp -p ../id_rsa .
+cp -p ../id_rsa.pub .
+
 docker build --tag dernils/duplicitytest:latest .
 rm requirements.txt
+rm id_rsa
+rm id_rsa.pub 
+
+cd ..
+cd ftp_server
+docker build --tag dernils/duplicity_testinfrastructure_ftp:latest .
+
+cd ..
+cd ssh_server
+cp -p ../id_rsa.pub .
+docker build --tag dernils/duplicity_testinfrastructure_ssh:latest .
+rm id_rsa.pub 

=== modified file 'testing/infrastructure/duplicity_test/Dockerfile'
--- testing/infrastructure/duplicity_test/Dockerfile	2017-05-06 19:41:30 +0000
+++ testing/infrastructure/duplicity_test/Dockerfile	2017-05-14 07:45:09 +0000
@@ -66,6 +66,14 @@
 # Setting a working directory to home
 WORKDIR /home/test
 
+# Copy a SSH key to the users folder that is used for some test cases
+USER root
+COPY ./id_rsa /home/test/.ssh/
+COPY ./id_rsa.pub /home/test/.ssh/
+RUN chown -R test:test /home/test/.ssh
+RUN chmod 400 /home/test/.ssh/id_rsa
+USER test 
+
 # Branch the duplicity repo for testing
 RUN bzr branch lp:duplicity duplicity
 

=== added file 'testing/infrastructure/id_rsa'
--- testing/infrastructure/id_rsa	1970-01-01 00:00:00 +0000
+++ testing/infrastructure/id_rsa	2017-05-14 07:45:09 +0000
@@ -0,0 +1,27 @@
+-----BEGIN RSA PRIVATE KEY-----
+MIIEowIBAAKCAQEA4ZBvr/4i0X0HMPLcTz5F11qCn84GnymquLlR2s/5U8HDpxSK
+2ZLiZ482yNRkxQkV6JMRIXk9EFQ5XeyAZIPildheqVy17DPhUjyiMf/0OB9OFVUj
+dPAytTU1uicYsmJyBbel/g7wBhOEwFqUNzNZ5qDdWgVIGZVXfr6PS+1lr5oAG+Z8
+cdOpuxJg56sCxYvh+wyBIJ70J/6S79xFRNbRUjANsSCReq8Zq5kpXbH4itZN44Zh
+TCkqgdeYf4p7A1lmNtEfzPGi0voXORxCsTcjJERS4yg4iOA55Wi9bRuOo+ltd+lw
+ByEEq/0f8mhvMrqGa6gfsIwXQQfC/m/nxOygbwIDAQABAoIBAFTwlg5ZDfWDNn0J
+R0sT8Srqa3wQElBHWYGcT6WtYX9X3PyC8NKDzHwg4wl3accE59of33iSdhs7DCtB
+TwwK2f2BcAwo3FzpIHVeXdmkP+wOZdIEsIhtlzrDocrKxDMQfU2H3ZsxNtOi7/+b
+XGr5DGabtx8aFA+GPF9/FOkLqZoZF8OdNHTMdBnVCfS0SPcHLDnljC6aplJE7kyh
+RGmASW60X4W8rzdLbrX1W6PDCFMavLLToVqiecm1lU2/0DIGzij96JYv/0QXe34T
+B8PhBRt+jQUr7Wx5mSS7IX0VFbqM+LkyicblLn9BVHSsKVyxva3R4aqjma2+dsfp
+aYBEqqkCgYEA+pXsTZHg6g0c66LKdbBJKnhccUbfzI0bcjazgPcucxZXuim/Gerz
+aEGXuKg5KWjIv3HDpFuH8LttaQWOg2OqLJ2hBmN2DB2r7OiT927Lqnx5RNUcCm7y
+FSg51IWVaQWqXS1NIzs0INrJjIgY3NzXIt4mZ5bkcxV0KO8GBbouPb0CgYEA5nAc
+ZKY+PEoq0eopPU9ZW999GPzfQAjkbc+fJ9YmuYHwToceh4FZZYe2t0ltmuxe/O8L
+cR6bBKRDK/eHn0Q+qd6axWtYCPp3TJ+SGSsSbY+ohw+f9fUYReNn1eirY39+MSmP
+EGY8xREHjhVbvTBLcujeM5PJOfSeL/g85olwq5sCgYEAxa+86j0piLurkyGsQ0g9
+kGm/1AWrzP2NJ3+MATP/yv9XWiBRCKtYN2l3vk/IOKjXpmCjjWhASGQ0nmOp2HGp
+8OSyAg45NLm7xnx9LviTZ/4m1WyiYIDCLnq7lLOF+YJksnlon2C3cSvpURqVMrhu
+aK2E8rgRpdDJAPoAcEg+7qkCgYBo0s7XqhfdbQ+VdaNU1Gedt0g8uasEggcHPM71
+XCE5ebZKwoRMt7ZmzRl8vtYxBx0eic72oil6Lw6vTWZQOMcYlr5o16+wRd9G57+Z
+21ICrLrI1X66b8ee4xc04yYzmKdazcDagOdb7ac2JtkYcoOfvfZ2F+ANkQz0XUO2
+ywaJyQKBgCe09+E3nHQkCQfhOJO4t46bEHeN2gf4OEqoajAPizK58c8rluodg7xx
+ttBFCuMcRLLXNgo/t5SS7DAmWC5zjVsr9T1ooWzBhZiegsatsPY+f30xaAKl1/ET
+q6Rx6FuLjSv8t98sIu3l2sWuTvSzujbtxRGjy0Z4CF/tmbrAbQRF
+-----END RSA PRIVATE KEY-----

=== added file 'testing/infrastructure/id_rsa.pub'
--- testing/infrastructure/id_rsa.pub	1970-01-01 00:00:00 +0000
+++ testing/infrastructure/id_rsa.pub	2017-05-14 07:45:09 +0000
@@ -0,0 +1,1 @@
+ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDhkG+v/iLRfQcw8txPPkXXWoKfzgafKaq4uVHaz/lTwcOnFIrZkuJnjzbI1GTFCRXokxEheT0QVDld7IBkg+KV2F6pXLXsM+FSPKIx//Q4H04VVSN08DK1NTW6JxiyYnIFt6X+DvAGE4TAWpQ3M1nmoN1aBUgZlVd+vo9L7WWvmgAb5nxx06m7EmDnqwLFi+H7DIEgnvQn/pLv3EVE1tFSMA2xIJF6rxmrmSldsfiK1k3jhmFMKSqB15h/insDWWY20R/M8aLS+hc5HEKxNyMkRFLjKDiI4DnlaL1tG46j6W136XAHIQSr/R/yaG8yuoZrqB+wjBdBB8L+b+fE7KBv [e-mail]

=== modified file 'testing/infrastructure/setup.sh'
--- testing/infrastructure/setup.sh	2017-05-06 19:17:29 +0000
+++ testing/infrastructure/setup.sh	2017-05-14 07:45:09 +0000
@@ -34,10 +34,12 @@
 
 # Remove all running instances of the test system and also remove the containers. This ensure
 # that the test infrastructure is frehshly started.
-docker rm $(docker stop $(docker ps -a -q --filter name=ftpd_server --format="{{.ID}}"))
-docker rm $(docker stop $(docker ps -a -q --filter name=duplicity_test --format="{{.ID}}"))
+docker rm -f $(docker stop $(docker ps -a -q --filter name=d70c0e18-37d5-11e7-a919-92ebcb67fe33-ftpd_server --format="{{.ID}}"))
+docker rm -f $(docker stop $(docker ps -a -q --filter name=ee681ee4-37d5-11e7-a919-92ebcb67fe33-duplicity_ssh_server --format="{{.ID}}"))
+docker rm -f $(docker stop $(docker ps -a -q --filter name=f3c09128-37d5-11e7-a919-92ebcb67fe33-duplicity_test --format="{{.ID}}"))
 
 
 # Start the containers. Docker run will automatically download the image if necessary
-docker run -d --net testnetwork --ip 10.10.10.3 --name ftpd_server -p 21:21 -p 30000-30009:30000-30009 -e "PUBLICHOST=localhost" dernils/duplicity_testinfrastructure_ftp
-docker run --name duplicity_test --net testnetwork --ip 10.10.10.2 -it  dernils/duplicitytest:latest
+docker run -d --net testnetwork --ip 10.10.10.3 --name d70c0e18-37d5-11e7-a919-92ebcb67fe33-ftpd_server -p 21:21 -p 30000-30009:30000-30009 -e "PUBLICHOST=localhost" dernils/duplicity_testinfrastructure_ftp
+docker run -d --net testnetwork --ip 10.10.10.4 --name ee681ee4-37d5-11e7-a919-92ebcb67fe33-duplicity_ssh_server  -p 22:22 -e "PUBLICHOST=localhost" dernils/duplicity_testinfrastructure_ssh:latest 
+docker run --name f3c09128-37d5-11e7-a919-92ebcb67fe33-duplicity_test --net testnetwork --ip 10.10.10.2 -it  dernils/duplicitytest:latest

=== added directory 'testing/infrastructure/ssh_server'
=== added file 'testing/infrastructure/ssh_server/Dockerfile'
--- testing/infrastructure/ssh_server/Dockerfile	1970-01-01 00:00:00 +0000
+++ testing/infrastructure/ssh_server/Dockerfile	2017-05-14 07:45:09 +0000
@@ -0,0 +1,63 @@
+# -*- Mode:Python; indent-tabs-mode:nil; tab-width:4 -*-
+#
+# Copyright 2017 Nils Tekampe <[e-mail]>
+# Thanks to Aleksandar Diklic "https://github.com/rastasheep"
+#
+# This file is part of duplicity.
+# It is the Dockerfile of a simple ftp server that is used for backend testing 
+#
+# Duplicity is free software; you can redistribute it and/or modify it
+# under the terms of the GNU General Public License as published by the
+# Free Software Foundation; either version 2 of the License, or (at your
+# option) any later version.
+#
+# Duplicity is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
+# General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with duplicity; if not, write to the Free Software Foundation,
+# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+# 
+FROM ubuntu:16.04
+
+RUN apt-get update
+
+RUN apt-get install -y openssh-server
+RUN mkdir /var/run/sshd
+
+RUN echo 'root:root' |chpasswd
+
+RUN sed -ri 's/^PermitRootLogin\s+.*/PermitRootLogin yes/' /etc/ssh/sshd_config
+RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config
+RUN echo "Match User userWithOnlyKeyAccess" >> /etc/ssh/sshd_config
+RUN echo "     PasswordAuthentication no" >> /etc/ssh/sshd_config
+
+RUN useradd -ms /bin/bash userWithKeyAccess
+RUN useradd -ms /bin/bash userWithPasswordAccess
+RUN useradd -ms /bin/bash userWithOnlyKeyAccess
+
+USER userWithKeyAccess
+WORKDIR /home/userWithKeyAccess
+COPY ./id_rsa.pub /home/userWithKeyAccess/.ssh/authorized_keys
+
+USER userWithOnlyKeyAccess
+WORKDIR /home/userWithOnlyKeyAccess
+COPY ./id_rsa.pub /home/userWithOnlyKeyAccess/.ssh/authorized_keys
+
+USER userWithPasswordAccess
+WORKDIR /home/userWithPasswordAccess
+
+USER root 
+
+RUN echo 'userWithKeyAccess:userWithKeyAccess' |chpasswd
+RUN echo 'userWithPasswordAccess:userWithPasswordAccess' |chpasswd
+
+RUN chown -R userWithKeyAccess:userWithKeyAccess /home/userWithKeyAccess/.ssh
+RUN chown -R userWithOnlyKeyAccess:userWithOnlyKeyAccess /home/userWithOnlyKeyAccess/.ssh/
+
+
+EXPOSE 22
+
+CMD ["/usr/sbin/sshd", "-D"]
\ No newline at end of file