Merge lp:~charlesk/keeper/untar into lp:keeper/devel

[Charles Kerr]

Add untar

Add untar.

* The Untar class itself, which manages a couple of QProcesses for xzcat and tar x. Add tar & xz to debian dependencies.

* Add untar-test unit test for the untar class

* Add keeper-untar executable that gets a restore socket from keeper, then untars from it

* Add keeper-untar-test that tests keeper-untar against the dbusmock service. Implement restore in the dbusmock service.

* Rename keeper-tar-create as keeper-tar for symmetry with keeper-untar

* Not pushed/completed: integration tests

[unity-api-1-bot (unity-api-1-bot)]

FAILED: Continuous integration, rev:148
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/148/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build/1266/console
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-0-fetch/1273/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/148/rebuild

review: Needs Fixing (continuous-integration)

[unity-api-1-bot (unity-api-1-bot)]

FAILED: Continuous integration, rev:148
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/149/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build/1267/console
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-0-fetch/1274/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/149/rebuild

review: Needs Fixing (continuous-integration)

[Xavi Garcia (xavi-garcia-mena)]

Thanks, Charles.
It looks good and I've tested it with the service and the command line client and it works fine.

I tried to trigger a rebuild myself but jenkins is still complaining.

I've added a comment inline...

A found a minor issue, non blocking, but that in cases that the restore file is very big we could be trying to restore even we had an error in the step method. (the error in not checked when calling step)

And even more minor: the coding style for the brackets is different in some cases.
There are some cases like:
if (blah) {
   blah
}

Shouldn't it be:?
if (blah)
{
   blah
}

review: Needs Fixing

[lp:~charlesk/keeper/untar]

149. By Charles Kerr on 2016-12-14

in debian/control, remove errant newline

150. By Charles Kerr on 2016-12-14

in keeper-untar-test, extract out common test code into helper methods

151. By Charles Kerr on 2016-12-14

in keeper-untar, exit with failure if Untar::step() fails

152. By Charles Kerr on 2016-12-15

in Untar class, add finish() method to check that xz/tar finish and check their return values

153. By Charles Kerr on 2016-12-15

in keeper-untar, exit with failure if Untar.step() or Untar.finish() fail

154. By Charles Kerr on 2016-12-15

fix end-of-namespace semicolon warning

155. By Charles Kerr on 2016-12-15

in untar-test, use Untar.finish()

156. By Charles Kerr on 2016-12-15

in keeper-untar-test, add test cases for restoring corrupt archives

157. By Charles Kerr on 2016-12-15

in service dbusmock, wait for helpers to exit

[unity-api-1-bot (unity-api-1-bot)]

FAILED: Continuous integration, rev:157
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/153/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build/1271/console
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-0-fetch/1278
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/1059/console
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/1059/console
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/1059/console
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/1059/console
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/1059/console
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/1059/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/153/rebuild

review: Needs Fixing (continuous-integration)

[lp:~charlesk/keeper/untar]

158. By Charles Kerr on 2016-12-16

fix bad whitespace that caused whitespace test to fail

159. By Charles Kerr on 2016-12-16

fix flake8 warnings in service dbusmock

160. By Charles Kerr on 2016-12-16

fix minor int conversion warning in tests/utils/main.cpp

[unity-api-1-bot (unity-api-1-bot)]

FAILED: Continuous integration, rev:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/156/
Executed test runs:
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build/1277/console
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-0-fetch/1284
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/1065
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/1065/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/1065
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/1065/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/1065
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/1065/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/1065
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/1065/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/1065
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/1065/artifact/output/*zip*/output.zip
    FAILURE: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/1065/console

Click here to trigger a rebuild:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/156/rebuild

review: Needs Fixing (continuous-integration)

[unity-api-1-bot (unity-api-1-bot)]

PASSED: Continuous integration, rev:160
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/155/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build/1276
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-0-fetch/1283
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/1064
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/1064/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/1064
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/1064/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/1064
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/1064/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/1064
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/1064/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/1064
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/1064/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/1064
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/1064/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/155/rebuild

review: Approve (continuous-integration)

[Xavi Garcia (xavi-garcia-mena)]

Looks good to me, thanks Charles

review: Approve

=== modified file 'debian/control'
--- debian/control	2016-08-26 08:48:05 +0000
+++ debian/control	2016-12-16 19:41:21 +0000
@@ -34,7 +34,7 @@
                python3-flake8,
 Standards-Version: 3.9.5
 Homepage: https://launchpad.net/keeper
-# If you aren't a member of ~indicator-applet-developers but need to upload
+ If you aren't a member of ~indicator-applet-developers but need to upload
 # packaging changes, just go ahead.  ~indicator-applet-developers will notice
 # and sync up the code again.
 Vcs-Bzr: https://code.launchpad.net/keeper/trunk
@@ -45,6 +45,8 @@
 Depends: ${shlibs:Depends}, 
          ${misc:Depends},
          systemd | systemd-shim,
+         tar,
+         xz-utils
 Description: Backup Tool
  A backup/restore utility for Ubuntu
 
@@ -53,6 +55,8 @@
 Depends: ${shlibs:Depends},
          ${misc:Depends},
          systemd | systemd-shim,
+         tar,
+         xz-utils
 Description: Backup Tool
  A backup/restore utility for Ubuntu (client application)
 

=== modified file 'debian/keeper.install'
--- debian/keeper.install	2016-08-10 07:43:36 +0000
+++ debian/keeper.install	2016-12-16 19:41:21 +0000
@@ -1,6 +1,7 @@
 /usr/lib/*/keeper/folder-backup.sh
 /usr/lib/*/keeper/keeper-service
-/usr/lib/*/keeper/keeper-tar-create
+/usr/lib/*/keeper/keeper-tar
+/usr/lib/*/keeper/keeper-untar
 /usr/lib/*/ubuntu-app-launch/backup-helper/exec-tool
 /usr/share/keeper/helper-registry.json
 /usr/share/dbus-1/services/com.canonical.keeper.service

=== modified file 'src/helper/folder-backup.sh.in'
--- src/helper/folder-backup.sh.in	2016-08-10 05:01:08 +0000
+++ src/helper/folder-backup.sh.in	2016-12-16 19:41:21 +0000
@@ -20,4 +20,4 @@
 #
 
 echo $PWD
-find ./ -type f -print0 | @CMAKE_INSTALL_FULL_PKGLIBEXECDIR@/keeper-tar-create -a /com/canonical/keeper/helper
+find ./ -type f -print0 | @CMAKE_INSTALL_FULL_PKGLIBEXECDIR@/keeper-tar -a /com/canonical/keeper/helper

=== modified file 'src/tar/CMakeLists.txt'
--- src/tar/CMakeLists.txt	2016-07-22 04:01:28 +0000
+++ src/tar/CMakeLists.txt	2016-12-16 19:41:21 +0000
@@ -1,8 +1,14 @@
 set(LIB_NAME "keepertar")
-set(APP_NAME "keeper-tar-create")
+set(KEEPER_TAR_APP_NAME "keeper-tar")
+set(KEEPER_UNTAR_APP_NAME "keeper-untar")
+
+##
+##  library
+##
 
 set(LIB_SOURCES
   tar-creator.cpp
+  untar.cpp
 )
 add_library(
   ${LIB_NAME}
@@ -10,38 +16,76 @@
   ${LIB_SOURCES}
 )
 
-set_property(
-  SOURCE main.cpp
-  PROPERTIES APPEND_STRING PROPERTY COMPILE_DEFINITIONS APP_NAME=\"${APP_NAME}\"
-)
-
-set(APP_SOURCES
-  main.cpp
-)
-
-add_executable(
-  ${APP_NAME}
-  ${APP_SOURCES}
-)
-
 link_directories(
   ${SERVICE_DEPS_LIBRARY_DIRS}
 )
 
-target_link_libraries(
-  ${APP_NAME}
-  ${LIB_NAME}
-  qdbus-stubs
-  storage-framework
-  backup-helper
-  ${SERVICE_DEPS_LIBRARIES}
-  Qt5::Core
-  Qt5::DBus
-)
+##
+##  tar creator
+##
+
+set_property(
+  SOURCE tar-creator-main.cpp
+  PROPERTIES APPEND_STRING PROPERTY COMPILE_DEFINITIONS APP_NAME=\"${KEEPER_TAR_APP_NAME}\"
+)
+
+set(KEEPER_UNTAR_APP_SOURCES
+  untar-main.cpp
+)
+
+add_executable(
+  ${KEEPER_UNTAR_APP_NAME}
+  ${KEEPER_UNTAR_APP_SOURCES}
+)
+
+target_link_libraries(
+  ${KEEPER_UNTAR_APP_NAME}
+  ${LIB_NAME}
+  qdbus-stubs
+  storage-framework
+  backup-helper
+  ${SERVICE_DEPS_LIBRARIES}
+  Qt5::Core
+  Qt5::DBus
+)
+
+##
+##  untar
+##
+
+set_property(
+  SOURCE untar-main.cpp
+  PROPERTIES APPEND_STRING PROPERTY COMPILE_DEFINITIONS APP_NAME=\"${KEEPER_UNTAR_APP_NAME}\"
+)
+
+set(KEEPER_TAR_APP_SOURCES
+  tar-creator-main.cpp
+)
+
+add_executable(
+  ${KEEPER_TAR_APP_NAME}
+  ${KEEPER_TAR_APP_SOURCES}
+)
+
+target_link_libraries(
+  ${KEEPER_TAR_APP_NAME}
+  ${LIB_NAME}
+  qdbus-stubs
+  storage-framework
+  backup-helper
+  ${SERVICE_DEPS_LIBRARIES}
+  Qt5::Core
+  Qt5::DBus
+)
+
+##
+##
+##
 
 install(
   TARGETS
-    ${APP_NAME}
+    ${KEEPER_TAR_APP_NAME}
+    ${KEEPER_UNTAR_APP_NAME}
   RUNTIME DESTINATION ${CMAKE_INSTALL_FULL_PKGLIBEXECDIR}
 )
 
@@ -49,6 +93,7 @@
   COVERAGE_REPORT_TARGETS
   ${COVERAGE_REPORT_TARGETS}
   ${LIB_NAME}
-  ${APP_NAME}
+  ${KEEPER_TAR_APP_NAME}
+  ${KEEPER_UNTAR_APP_NAME}
   PARENT_SCOPE
 )

=== renamed file 'src/tar/main.cpp' => 'src/tar/tar-creator-main.cpp'
=== added file 'src/tar/untar-main.cpp'
--- src/tar/untar-main.cpp	1970-01-01 00:00:00 +0000
+++ src/tar/untar-main.cpp	2016-12-16 19:41:21 +0000
@@ -0,0 +1,169 @@
+/*
+ * Copyright (C) 2016 Canonical, Ltd.
+ *
+ * This program is free software: you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 3, as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranties of
+ * MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+ * PURPOSE.  See the GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Authors:
+ *     Charles Kerr <[e-mail]>
+ */
+
+#include "tar/untar.h"
+#include "qdbus-stubs/dbus-types.h"
+#include "qdbus-stubs/keeper_helper_interface.h"
+
+#include <QCommandLineParser>
+#include <QCoreApplication>
+#include <QDebug>
+#include <QDBusConnection>
+#include <QDBusConnectionInterface>
+#include <QDBusUnixFileDescriptor>
+#include <QFile>
+#include <QLocalSocket>
+
+#include <sys/select.h>
+#include <unistd.h>
+
+#include <cstdio> // fileno()
+#include <ctime>
+#include <iostream>
+#include <type_traits>
+
+namespace
+{
+
+std::tuple<QString>
+parse_args(QCoreApplication& app)
+{
+    // parse the command line
+    QCommandLineParser parser;
+    parser.addHelpOption();
+    parser.setApplicationDescription(
+        "\n"
+        "The reverse of keeper-tar. Queries Keeper for a socket fd, then pipes\n"
+        "that socket through xzcat and tar to restore the archive data into the current\n"
+        "working directory.\n"
+        "\n"
+        "Helper usage: "  APP_NAME " -a /bus/path"
+    );
+    QCommandLineOption bus_path_option{
+        QStringList() << "a" << "bus-path",
+        QStringLiteral("Keeper service's DBus path"),
+        QStringLiteral("bus-path")
+    };
+    parser.addOption(bus_path_option);
+    parser.process(app);
+    const auto bus_path = parser.value(bus_path_option);
+
+    // gotta have the bus path
+    if (bus_path.isEmpty()) {
+        std::cerr << "Missing required argument: --bus-path" << std::endl;
+        parser.showHelp(EXIT_FAILURE);
+    }
+
+    return std::make_tuple(bus_path);
+}
+
+QDBusUnixFileDescriptor
+get_socket_from_keeper(const QString& bus_path)
+{
+    QDBusUnixFileDescriptor ret;
+
+    qDebug() << "asking keeper for a socket";
+    DBusInterfaceKeeperHelper helperInterface(
+        DBusTypes::KEEPER_SERVICE,
+        bus_path,
+        QDBusConnection::sessionBus()
+    );
+
+    auto fd_reply = helperInterface.StartRestore();
+    fd_reply.waitForFinished();
+    if (fd_reply.isError()) {
+        qCritical("Call to '%s.StartRestore() at '%s' call failed: %s",
+            DBusTypes::KEEPER_SERVICE,
+            qPrintable(bus_path),
+            qPrintable(fd_reply.error().message())
+        );
+    } else {
+        ret = fd_reply.value();
+    }
+
+    return ret;
+}
+
+bool
+untar_from_socket(Untar& untar, int fd)
+{
+    bool success = false;
+    static constexpr int STEP_BUFSIZE = 4096*4; // arbitrary
+    char buf[STEP_BUFSIZE];
+
+    for (;;)
+    {
+        auto const n_read = read(fd, buf, sizeof(buf));
+
+        if (n_read > 0)
+        {
+            if (!untar.step(buf, n_read))
+                break;
+        }
+        else if (n_read == 0) // eof
+        {
+            qDebug() << Q_FUNC_INFO << "eof reached";
+            success = true;
+            break;
+        }
+        else if (errno == EAGAIN)
+        {
+            QThread::msleep(100);
+            continue;
+        }
+        else
+        {
+            qCritical() << Q_FUNC_INFO << "read() returned" << strerror(errno);
+            break;
+        }
+    }
+
+    if (success)
+        success = untar.finish();
+
+    return success;
+}
+
+} // anonymous namespace
+
+int
+main(int argc, char **argv)
+{
+    QCoreApplication app(argc, argv);
+
+    // get the inputs
+    QString bus_path;
+    std::tie(bus_path) = parse_args(app);
+
+    // ask keeper for a socket to read
+    const auto qfd = get_socket_from_keeper(bus_path);
+    if (!qfd.isValid()) {
+        qCritical() << "Can't proceed without a socket from keeper";
+        return EXIT_FAILURE;
+    }
+
+    // do it!
+    auto const cwd = QDir::currentPath().toStdString();
+    Untar untar{cwd};
+    auto const ret = untar_from_socket(untar, qfd.fileDescriptor())
+        ? EXIT_SUCCESS
+        : EXIT_FAILURE;
+    qInfo() << Q_FUNC_INFO << "returning" << ret;
+    return ret;
+}

=== added file 'src/tar/untar.cpp'
--- src/tar/untar.cpp	1970-01-01 00:00:00 +0000
+++ src/tar/untar.cpp	2016-12-16 19:41:21 +0000
@@ -0,0 +1,144 @@
+/*
+ * Copyright (C) 2016 Canonical, Ltd.
+ *
+ * This program is free software: you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 3, as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranties of
+ * MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+ * PURPOSE.  See the GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Authors:
+ *   Charles Kerr <[e-mail]>
+ */
+
+#include "tar/untar.h"
+
+#include <QDebug>
+#include <QProcess>
+#include <QString>
+
+#include <string>
+#include <vector>
+
+class Untar::Impl
+{
+public:
+
+    explicit Impl(std::string const& path)
+        : path_{path}
+    {
+        uncompress_.setStandardOutputProcess(&untar_);
+        uncompress_.start("xz", QStringList{ "--decompress", "--stdout", "--force" });
+
+        untar_.start("tar", QStringList{ "-xv", "-C", path_.c_str()});
+        untar_.setProcessChannelMode(QProcess::ForwardedChannels);
+    }
+
+    ~Impl()
+    {
+        finish();
+    }
+
+    bool step(char const * buf, size_t buflen)
+    {
+        bool success = true;
+
+        auto n_left = buflen;
+        while (n_left > 0)
+        {
+            auto const n_written_this_pass = uncompress_.write(buf, n_left);
+            if (n_written_this_pass == -1) {
+                qCritical() << Q_FUNC_INFO << strerror(errno);
+                success = false;
+                break;
+            } else {
+                n_left -= n_written_this_pass;
+                buf += n_written_this_pass;
+            }
+        }
+
+        return success;
+    }
+
+    bool finish ()
+    {
+        bool ok = true;
+
+        uncompress_.closeWriteChannel();
+        if (!finish(uncompress_, "xz"))
+            ok = false;
+
+        if (!finish(untar_, "untar"))
+            ok = false;
+
+        return ok;
+    }
+
+private:
+
+    bool finish (QProcess& proc, QString const& name)
+    {
+        if (proc.state() != QProcess::NotRunning)
+        {
+            proc.waitForFinished();
+        }
+
+        bool ok;
+
+        if (proc.state() != QProcess::NotRunning)
+        {
+            qCritical() << name << "did not finish";
+            ok = false;
+        }
+        else if (proc.exitStatus() != QProcess::NormalExit)
+        {
+            qCritical() << name << "exited abnormally";
+            ok = false;
+        }
+        else if (proc.exitCode() != 0)
+        {
+            qCritical() << name << "exited with error code" << proc.exitCode();
+            ok = false;
+        }
+        else
+        {
+            qDebug() << name << "finished ok";
+            ok = true;
+        }
+
+        return ok;
+    }
+
+    std::string const path_;
+    QProcess uncompress_;
+    QProcess untar_;
+};
+
+/**
+***
+**/
+
+Untar::Untar(std::string const& path)
+    : impl_{new Impl{path}}
+{
+}
+
+Untar::~Untar() =default;
+
+bool
+Untar::step(char const * buf, size_t buflen)
+{
+    return impl_->step(buf, buflen);
+}
+
+bool
+Untar::finish()
+{
+    return impl_->finish();
+}

=== added file 'src/tar/untar.h'
--- src/tar/untar.h	1970-01-01 00:00:00 +0000
+++ src/tar/untar.h	2016-12-16 19:41:21 +0000
@@ -0,0 +1,38 @@
+/*
+ * Copyright (C) 2016 Canonical, Ltd.
+ *
+ * This program is free software: you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 3, as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranties of
+ * MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+ * PURPOSE.  See the GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Authors:
+ *   Charles Kerr <[e-mail]>
+ */
+
+#pragma once
+
+#include <cstddef> // size_t
+#include <memory> // shared_ptr
+
+
+class Untar
+{
+public:
+    explicit Untar(std::string const& target_path);
+    ~Untar();
+    bool step(char const * buf, size_t n_bytes);
+    bool finish();
+
+private:
+    class Impl;
+    friend class Impl;
+    std::shared_ptr<Impl> impl_;
+};

=== modified file 'tests/CMakeLists.txt'
--- tests/CMakeLists.txt	2016-11-11 14:50:06 +0000
+++ tests/CMakeLists.txt	2016-12-16 19:41:21 +0000
@@ -29,7 +29,8 @@
     fake-backup-helper-failure
 )
 
-set(KEEPER_TAR_CREATE_BIN ${CMAKE_BINARY_DIR}/src/tar/keeper-tar-create)
+set(KEEPER_TAR_CREATE_BIN ${CMAKE_BINARY_DIR}/src/tar/keeper-tar)
+set(KEEPER_UNTAR_BIN ${CMAKE_BINARY_DIR}/src/tar/keeper-untar)
 set(KEEPER_HELPER_TEST_LOCATION ${CMAKE_BINARY_DIR}/tests/fakes/helpers-test.sh)
 set(BACKUP_HELPER_FAILURE_LOCATION ${CMAKE_BINARY_DIR}/tests/fakes/${BACKUP_HELPER_FAILURE})
 set(RESTORE_HELPER_TEST_LOCATION ${CMAKE_BINARY_DIR}/tests/fakes/${RESTORE_HELPER})

=== modified file 'tests/com_canonical_keeper.py'
--- tests/com_canonical_keeper.py	2016-08-03 18:38:12 +0000
+++ tests/com_canonical_keeper.py	2016-12-16 19:41:21 +0000
@@ -12,6 +12,8 @@
 '''com.canonical.keeper mock template
 '''
 
+# (c) 2016 Canonical Ltd.
+#
 # This program is free software; you can redistribute it and/or modify it under
 # the terms of the GNU Lesser General Public License as published by the Free
 # Software Foundation; either version 3 of the License, or (at your option) any
@@ -167,7 +169,8 @@
             helper_cwd = os.getcwd()
 
         # spawn the helper
-        user.log('starting %s for %s, env %s' % (helper_exec, uuid, henv))
+        user.log('starting %s for %s in %s, env %s' %
+                 (helper_exec, uuid, helper_cwd, henv))
         user.process = subprocess.Popen(
             [helper_exec, HELPER_PATH],
             env=henv, stdout=sys.stdout, stderr=sys.stderr,
@@ -180,7 +183,6 @@
 
 def user_periodic_func(user):
 
-    done = False
     got_data_this_pass = False
 
     if not user.process:
@@ -189,41 +191,61 @@
     uuid = user.current_task
     td = user.task_data[uuid]
 
-    # did the helper exit with an error code?
+    # check the socket
+    socket_done = td.n_left == 0 or not td.sock
+    if not socket_done:
+
+        if td.action == ACTION_SAVING:
+            chunk = td.sock.recv(4096*2)
+            if chunk == '':  # eof
+                socket_done = True
+            else:
+                chunk_len = len(chunk)
+                if chunk_len:
+                    got_data_this_pass = True
+                    td.chunks.append(chunk)
+                    td.n_left -= chunk_len
+
+        if td.action == ACTION_RESTORING:
+            begin = td.n_bytes - td.n_left
+            chunklen = min(td.n_left, 4096*2)
+            end = begin + chunklen
+            chunk = td.blob[begin:end]
+            n_sent = td.sock.send(chunk, socket.MSG_DONTWAIT)
+            if n_sent > 0:
+                got_data_this_pass = True
+                td.n_left -= n_sent
+
+        user.log('after socket pass, n_left==%s' % (td.n_left))
+        if td.n_left == 0:
+            user.log('cleaning up socket because no data left')
+            td.sock.shutdown(socket.SHUT_RDWR)
+            td.sock.close()
+            td.sock = None
+            socket_done = True
+
+    # check the process
     returncode = user.process.poll()
-    if returncode:
-        error = 'helper exited with a returncode of %s' % (str(returncode))
-        user.log(error)
-        td.error = error
-        td.action = ACTION_FAILED
-        done = True
-
-    # try to read the socket
-    if td.sock and not td.error:
-        chunk = td.sock.recv(4096*2)
-        chunk_len = len(chunk)
-        if chunk_len:
-            got_data_this_pass = True
-            td.chunks.append(chunk)
-            td.n_left -= chunk_len
-            user.log('got %s more bytes; %s left' % (chunk_len, td.n_left))
-        if td.n_left <= 0:
-            done = True
-
-    # if done, clean up the socket
-    if done and td.sock:
-        user.log('cleaning up sock')
-        td.sock.shutdown(socket.SHUT_RDWR)
-        td.sock.close()
-        td.sock = None
-
-    # if done successfully, save the blob
-    if done and not td.error:
-        user.log('setting blob')
-        blob = b''.join(td.chunks)
-        td.blob = blob
-        user.log('backup %s done; %s bytes' % (uuid, len(blob)))
-        td.action = ACTION_COMPLETE
+    process_done = returncode is not None
+    if process_done:
+        exitmsg = 'helper exited with a returncode of %s' % (str(returncode))
+        user.log(exitmsg)
+        td.error = exitmsg
+
+    # are we done yet?
+    done = process_done and socket_done
+    if done:
+
+        # update td.action
+        if returncode != 0:
+            td.action = ACTION_FAILED
+        else:
+            if td.action == ACTION_SAVING:
+                user.log('setting blob')
+                blob = b''.join(td.chunks)
+                td.blob = blob
+                user.log('backup %s done; %s bytes' % (uuid, len(td.blob)))
+            td.action = ACTION_COMPLETE
 
     # maybe update the task's state
     if done or got_data_this_pass:
@@ -314,16 +336,13 @@
         task_state[KEY_PERCENT_DONE] = p
 
         # speed
-        helper = mockobject.objects[HELPER_PATH]
         if uuid == user.current_task:
             n_secs = 2
             n_bytes = 0
             too_old = time.time() - n_secs
             for key in td.bytes_per_second:
-                helper.log('key is %s' % (str(key)))
                 if key > too_old:
                     n_bytes += td.bytes_per_second[key]
-            helper.log('n_bytes is %s' % (str(n_bytes)))
             bytes_per_second = n_bytes / n_secs
         else:
             bytes_per_second = 0
@@ -361,7 +380,7 @@
 
     helper.log("got start_backup request for %s bytes" % (n_bytes))
 
-    parent, child = socket.socketpair()
+    sock1, sock2 = socket.socketpair()
 
     user = mockobject.objects[USER_PATH]
     uuid = user.current_task
@@ -369,15 +388,31 @@
     td = user.task_data[uuid]
     td.n_bytes = n_bytes
     td.n_left = n_bytes
-    td.sock = parent
+    td.sock = sock1
 
-    return dbus.types.UnixFd(child.fileno())
+    ret = dbus.types.UnixFd(sock2)
+    sock2.close()
+    return ret
 
 
 def helper_start_restore(helper):
-    helper.parent, child = socket.socketpair()
-    return child
-
+
+    user = mockobject.objects[USER_PATH]
+    uuid = user.current_task
+    blob = user.restore_choices[uuid][KEY_BLOB]
+
+    sock1, sock2 = socket.socketpair()
+
+    td = user.task_data[uuid]
+    td.blob = bytes([int(byte) for byte in blob])
+    td.n_bytes = len(td.blob)
+    td.n_left = td.n_bytes
+    td.sock = sock1
+    td.sock.setblocking(0)
+
+    ret = dbus.types.UnixFd(sock2)
+    sock2.close()
+    return ret
 
 #
 #  Controlling the mock

=== modified file 'tests/unit/tar/CMakeLists.txt'
--- tests/unit/tar/CMakeLists.txt	2016-09-15 16:02:54 +0000
+++ tests/unit/tar/CMakeLists.txt	2016-12-16 19:41:21 +0000
@@ -26,8 +26,26 @@
   ${CMAKE_CURRENT_SOURCE_DIR}/ktc-invoke-nofiles.sh.in
   ${KTC_INVOKE_NOFILES}
 )
+set(
+  KU_INVOKE
+  ${CMAKE_CURRENT_BINARY_DIR}/ku-invoke.sh
+)
+configure_file(
+  ${CMAKE_CURRENT_SOURCE_DIR}/ku-invoke.sh.in
+  ${KU_INVOKE}
+)
+set(
+  KU_INVOKE_NOBUS
+  ${CMAKE_CURRENT_BINARY_DIR}/ku-invoke-nobus.sh
+)
+configure_file(
+  ${CMAKE_CURRENT_SOURCE_DIR}/ku-invoke-nobus.sh.in
+  ${KU_INVOKE_NOBUS}
+)
 
 add_definitions(
+  -DKU_INVOKE="${KU_INVOKE}"
+  -DKU_INVOKE_NOBUS="${KU_INVOKE_NOBUS}"
   -DKTC_INVOKE="${KTC_INVOKE}"
   -DKTC_INVOKE_NOBUS="${KTC_INVOKE_NOBUS}"
   -DKTC_INVOKE_NOFILES="${KTC_INVOKE_NOFILES}"
@@ -56,10 +74,38 @@
   Qt5::Test
 )
 
-#add_test(
-#  ${TAR_CREATOR_TEST}
-#  ${TAR_CREATOR_TEST}
-#)
+add_test(
+  ${TAR_CREATOR_TEST}
+  ${TAR_CREATOR_TEST}
+)
+
+
+#
+# untar-test
+#
+
+set(
+  UNTAR_TEST
+  untar-test
+)
+
+add_executable(
+  ${UNTAR_TEST}
+  untar-test.cpp
+)
+
+target_link_libraries(
+  ${UNTAR_TEST}
+  ${UNIT_TEST_LIBRARIES}
+  Qt5::Core
+  Qt5::DBus
+  Qt5::Test
+)
+
+add_test(
+  ${UNTAR_TEST}
+  ${UNTAR_TEST}
+)
 
 
 #
@@ -102,30 +148,57 @@
 endforeach(funcname)
 
 #
-# keeper-tar-create-test
-#
-
-set(
-  KEEPER_TAR_CREATE_TEST
-  keeper-tar-create-test
-)
-
-add_executable(
-  ${KEEPER_TAR_CREATE_TEST}
-  keeper-tar-create-test.cpp
-)
-
-target_link_libraries(
-  ${KEEPER_TAR_CREATE_TEST}
-  ${UNIT_TEST_LIBRARIES}
-  Qt5::Core
-  Qt5::DBus
-  Qt5::Test
-)
-
-add_test(
-  ${KEEPER_TAR_CREATE_TEST}
-  ${KEEPER_TAR_CREATE_TEST}
+# keeper-tar-test
+#
+
+set(
+  KEEPER_TAR_TEST
+  keeper-tar-test
+)
+
+add_executable(
+  ${KEEPER_TAR_TEST}
+  keeper-tar-test.cpp
+)
+
+target_link_libraries(
+  ${KEEPER_TAR_TEST}
+  ${UNIT_TEST_LIBRARIES}
+  Qt5::Core
+  Qt5::DBus
+  Qt5::Test
+)
+
+add_test(
+  ${KEEPER_TAR_TEST}
+  ${KEEPER_TAR_TEST}
+)
+
+#
+# keeper-untar-test
+#
+
+set(
+  KEEPER_UNTAR_TEST
+  keeper-untar-test
+)
+
+add_executable(
+  ${KEEPER_UNTAR_TEST}
+  keeper-untar-test.cpp
+)
+
+target_link_libraries(
+  ${KEEPER_UNTAR_TEST}
+  ${UNIT_TEST_LIBRARIES}
+  Qt5::Core
+  Qt5::DBus
+  Qt5::Test
+)
+
+add_test(
+  ${KEEPER_UNTAR_TEST}
+  ${KEEPER_UNTAR_TEST}
 )
 
 #
@@ -134,8 +207,10 @@
 set(
   COVERAGE_TEST_TARGETS
   ${COVERAGE_TEST_TARGETS}
+  ${UNTAR_TEST}
   ${TAR_CREATOR_TEST}
   ${TAR_CREATOR_LIBARCHIVE_FAILURE_TEST}
-  ${KEEPER_TAR_CREATE_TEST}
+  ${KEEPER_TAR_TEST}
+  ${KEEPER_UNTAR_TEST}
   PARENT_SCOPE
 )

=== renamed file 'tests/unit/tar/keeper-tar-create-test.cpp' => 'tests/unit/tar/keeper-tar-test.cpp'
=== added file 'tests/unit/tar/keeper-untar-test.cpp'
--- tests/unit/tar/keeper-untar-test.cpp	1970-01-01 00:00:00 +0000
+++ tests/unit/tar/keeper-untar-test.cpp	2016-12-16 19:41:21 +0000
@@ -0,0 +1,232 @@
+/*
+ * Copyright 2016 Canonical Ltd.
+ *
+ * This program is free software: you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 3, as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranties of
+ * MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+ * PURPOSE.  See the GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Authors:
+ *   Charles Kerr <[e-mail]>
+ */
+
+#include "tests/utils/file-utils.h"
+#include "tests/utils/keeper-dbusmock-fixture.h"
+
+#include "tar/tar-creator.h"
+
+#include <gtest/gtest.h>
+
+#include <QString>
+#include <QTemporaryDir>
+
+
+/***
+****
+***/
+
+class KeeperUntarFixture: public KeeperDBusMockFixture
+{
+    using parent = KeeperDBusMockFixture;
+
+    void SetUp() override
+    {
+        parent::SetUp();
+
+        qsrand(unsigned(time(nullptr)));
+    }
+
+    void TearDown() override
+    {
+    }
+
+protected:
+
+    std::vector<char> tar_directory_into_memory(QString const & in_path)
+    {
+        std::vector<char> blob;
+
+        auto const currentPath = QDir::currentPath();
+        EXPECT_TRUE(QDir::setCurrent(in_path));
+
+        QDir const indir(in_path);
+        QStringList files;
+        for (auto file : FileUtils::getFilesRecursively(in_path))
+            files += indir.relativeFilePath(file);
+
+        TarCreator tar_creator(files, false);
+        std::vector<char> step;
+        while (tar_creator.step(step))
+            blob.insert(blob.end(), step.begin(), step.end());
+
+        EXPECT_TRUE(QDir::setCurrent(currentPath));
+
+        return blob;
+    }
+
+    QMap<QString,QVariant> build_folder_restore_choice(
+        QTemporaryDir& source,
+        QTemporaryDir& target,
+        char const* helper_exec,
+        std::vector<char> const& blob)
+    {
+
+        return QMap<QString,QVariant>{
+            { KEY_NAME, QDir(source.path()).dirName() },
+            { KEY_TYPE, QStringLiteral("folder") },
+            { KEY_SUBTYPE, target.path() },
+            { KEY_HELPER, QString::fromUtf8(helper_exec) },
+            { KEY_SIZE, quint64(blob.size()) },
+            { KEY_CTIME, quint64(time(nullptr)) },
+            { KEY_BLOB, QByteArray{&blob.front(), int(blob.size())} }
+        };
+    }
+
+    void restore(QString const& uuid)
+    {
+        QDBusReply<void> reply = user_iface_->call("StartRestore", QStringList{uuid});
+        ASSERT_TRUE(reply.isValid()) << qPrintable(reply.error().message());
+        ASSERT_TRUE(wait_for_tasks_to_finish());
+    }
+};
+
+/***
+****
+***/
+
+TEST_F(KeeperUntarFixture, RestoreRun)
+{
+    static constexpr int n_runs {2};
+
+    for (int i=0; i<n_runs; ++i)
+    {
+        // build a directory full of random files
+        QTemporaryDir in;
+        FileUtils::fillTemporaryDirectory(in.path());
+
+        // tar it up
+        auto const blob = tar_directory_into_memory(in.path());
+
+        // tell keeper that's a restore choice
+        QTemporaryDir out;
+        const auto uuid = add_restore_choice(build_folder_restore_choice(in, out, KU_INVOKE, blob));
+
+        // now run the restore
+        restore(uuid);
+
+        // after restore, the source and restore dirs should match
+        EXPECT_TRUE(FileUtils::compareDirectories(in.path(), out.path()));
+
+        // if the test failed, keep the artifacts for a human to look at
+        auto const passed = ::testing::UnitTest::GetInstance()->current_test_info()->result()->Passed();
+        in.setAutoRemove(passed);
+        out.setAutoRemove(passed);
+    }
+}
+
+/***
+****
+***/
+
+TEST_F(KeeperUntarFixture, BadArgNoBus)
+{
+    static constexpr int n_runs {1};
+
+    for (int i=0; i<n_runs; ++i)
+    {
+        // build a directory full of random files
+        QTemporaryDir in;
+        FileUtils::fillTemporaryDirectory(in.path());
+
+        // tar it up
+        auto const blob = tar_directory_into_memory(in.path());
+
+        // tell keeper that's a restore choice
+        QTemporaryDir out;
+        const auto uuid = add_restore_choice(build_folder_restore_choice(in, out, KU_INVOKE_NOBUS, blob));
+
+        // now run the restore
+        restore(uuid);
+
+        // confirm that the backup ended in error
+        const auto state = user_iface_->state();
+        const auto& properties = state[uuid];
+        EXPECT_EQ(QString::fromUtf8("failed"), properties.value(KEY_ACTION))
+            << qPrintable(properties.value(KEY_ACTION).toString());
+        EXPECT_FALSE(properties.value(KEY_ERROR).toString().isEmpty());
+    }
+}
+
+/***
+****
+***/
+
+TEST_F(KeeperUntarFixture, BadData)
+{
+    static constexpr int n_runs {1};
+
+    for (int i=0; i<n_runs; ++i)
+    {
+        // build a directory full of random files
+        QTemporaryDir in;
+        FileUtils::fillTemporaryDirectory(in.path());
+
+        // make a junk blob
+        std::vector<char> blob { 'n', 'o', 't', 'a', 't', 'a', 'r' };
+
+        // tell keeper that's a restore choice
+        QTemporaryDir out;
+        const auto uuid = add_restore_choice(build_folder_restore_choice(in, out, KU_INVOKE, blob));
+
+        // now run the restore
+        restore(uuid);
+
+        // confirm that the backup ended in error
+        const auto state = user_iface_->state();
+        const auto& properties = state[uuid];
+        EXPECT_EQ(QString::fromUtf8("failed"), properties.value(KEY_ACTION))
+            << qPrintable(properties.value(KEY_ACTION).toString());
+        EXPECT_FALSE(properties.value(KEY_ERROR).toString().isEmpty());
+    }
+}
+
+/***
+****
+***/
+
+TEST_F(KeeperUntarFixture, IncompleteData)
+{
+    static constexpr int n_runs {1};
+
+    for (int i=0; i<n_runs; ++i)
+    {
+        // build a directory full of random files
+        QTemporaryDir in;
+        FileUtils::fillTemporaryDirectory(in.path());
+
+        // make a truncated backup of it
+        auto blob = tar_directory_into_memory(in.path());
+        blob.resize(511);
+
+        // tell keeper that's a restore choice
+        QTemporaryDir out;
+        const auto uuid = add_restore_choice(build_folder_restore_choice(in, out, KU_INVOKE, blob));
+
+        // now run the restore
+        restore(uuid);
+
+        // confirm that the backup ended in error
+        const auto state = user_iface_->state();
+        const auto& properties = state[uuid];
+        EXPECT_EQ(QString::fromUtf8("failed"), properties.value(KEY_ACTION))
+            << qPrintable(properties.value(KEY_ACTION).toString());
+        EXPECT_FALSE(properties.value(KEY_ERROR).toString().isEmpty());
+    }
+}

=== added file 'tests/unit/tar/ku-invoke-nobus.sh.in'
--- tests/unit/tar/ku-invoke-nobus.sh.in	1970-01-01 00:00:00 +0000
+++ tests/unit/tar/ku-invoke-nobus.sh.in	2016-12-16 19:41:21 +0000
@@ -0,0 +1,1 @@
+@KEEPER_UNTAR_BIN@

=== added file 'tests/unit/tar/ku-invoke.sh.in'
--- tests/unit/tar/ku-invoke.sh.in	1970-01-01 00:00:00 +0000
+++ tests/unit/tar/ku-invoke.sh.in	2016-12-16 19:41:21 +0000
@@ -0,0 +1,1 @@
+@KEEPER_UNTAR_BIN@ -a /com/canonical/keeper/helper

=== added file 'tests/unit/tar/untar-test.cpp'
--- tests/unit/tar/untar-test.cpp	1970-01-01 00:00:00 +0000
+++ tests/unit/tar/untar-test.cpp	2016-12-16 19:41:21 +0000
@@ -0,0 +1,120 @@
+/*
+ * Copyright 2016 Canonical Ltd.
+ *
+ * This program is free software: you can redistribute it and/or modify it
+ * under the terms of the GNU General Public License version 3, as published
+ * by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranties of
+ * MERCHANTABILITY, SATISFACTORY QUALITY, or FITNESS FOR A PARTICULAR
+ * PURPOSE.  See the GNU General Public License for more details.
+ *
+ * You should have received a copy of the GNU General Public License along
+ * with this program.  If not, see <http://www.gnu.org/licenses/>.
+ *
+ * Authors:
+ *   Charles Kerr <[e-mail]>
+ */
+
+#include "tests/utils/file-utils.h"
+
+#include "tar/tar-creator.h"
+#include "tar/untar.h"
+
+#include <gtest/gtest.h>
+
+#include <QDebug>
+#include <QDir>
+#include <QFile>
+#include <QFileInfo>
+#include <QProcess>
+#include <QString>
+#include <QTemporaryDir>
+
+#include <algorithm>
+#include <array>
+#include <cstdio>
+
+class UntarFixture: public ::testing::Test
+{
+protected:
+
+    void SetUp() override
+    {
+        qsrand(unsigned(time(nullptr)));
+    }
+
+    void TearDown() override
+    {
+    }
+
+};
+
+/***
+****
+***/
+
+TEST_F(UntarFixture, Untar)
+{
+    static constexpr int n_runs {5};
+    static constexpr std::array<size_t,4> step_sizes = { 1024, 2048, 4096, INT_MAX };
+    //static constexpr std::array<int,1> step_sizes = { 1024 };
+
+    for (int i=0; i<n_runs; ++i)
+    {
+        // build a directory full of random files
+        QTemporaryDir in;
+        QDir indir(in.path());
+        //FileUtils::fillTemporaryDirectory(in.path());
+        FileUtils::fillTemporaryDirectory(in.path(), 3, 3, 4096, 1);
+
+        // tar it up
+        std::vector<char> contents;
+        {
+            EXPECT_TRUE(QDir::setCurrent(in.path()));
+            QStringList files;
+            for (auto file : FileUtils::getFilesRecursively(in.path()))
+                files += indir.relativeFilePath(file);
+            TarCreator tar_creator(files, false);
+            std::vector<char> step;
+            while (tar_creator.step(step))
+                contents.insert(contents.end(), step.begin(), step.end());
+        }
+
+        // walk through an untar test for each of the step sizes
+        for (auto const& step_size : step_sizes)
+        {
+            char const * walk = &contents.front();
+            auto n_left = contents.size();
+
+            // untar it
+            QTemporaryDir out;
+            QDir outdir(out.path());
+
+            {
+                Untar untar(out.path().toStdString());
+                do
+                {
+                    auto const current_step_size = std::min(step_size, n_left);
+                    EXPECT_TRUE(untar.step(walk, current_step_size));
+                    n_left -= current_step_size;
+                    walk += current_step_size;
+                }
+                while(n_left > 0);
+                EXPECT_TRUE(untar.finish());
+            }
+
+            // compare it to the original
+            EXPECT_TRUE(FileUtils::compareDirectories(in.path(), out.path()));
+
+            // if the test failed, keep the outdir for manual inspection
+            auto const passed = ::testing::UnitTest::GetInstance()->current_test_info()->result()->Passed();
+            out.setAutoRemove(passed);
+        }
+
+        // if the test failed, keep the indir for manual inspection
+        auto const passed = ::testing::UnitTest::GetInstance()->current_test_info()->result()->Passed();
+        in.setAutoRemove(passed);
+    }
+}

=== modified file 'tests/utils/file-utils.cpp'
--- tests/utils/file-utils.cpp	2016-11-22 09:37:40 +0000
+++ tests/utils/file-utils.cpp	2016-12-16 19:41:21 +0000
@@ -38,7 +38,8 @@
 create_dummy_string()
 {
     // NB we want to exercise long filenames, but this cutoff length is arbitrary
-    static constexpr int MAX_BASENAME_LEN {200};
+    //static constexpr int MAX_BASENAME_LEN {200};
+    static constexpr int MAX_BASENAME_LEN {10};
     auto const filename_len = std::max(10, qrand() % MAX_BASENAME_LEN);
     QString str;
     for (int i=0; i<filename_len; ++i)

=== modified file 'tests/utils/file-utils.h'
--- tests/utils/file-utils.h	2016-09-12 15:28:06 +0000
+++ tests/utils/file-utils.h	2016-12-16 19:41:21 +0000
@@ -33,4 +33,4 @@
     bool checkPathIsDir(QString const & dirPath);
 
     QStringList getFilesRecursively(QString const & dirPath);
-};
+}

=== modified file 'tests/utils/keeper-dbusmock-fixture.h'
--- tests/utils/keeper-dbusmock-fixture.h	2016-09-15 16:02:54 +0000
+++ tests/utils/keeper-dbusmock-fixture.h	2016-12-16 19:41:21 +0000
@@ -142,6 +142,7 @@
             bool all_done = true;
             for(const auto& properties : state) {
                 const auto action = properties.value(KEY_ACTION);
+                qInfo() << Q_FUNC_INFO << "action is" << action;
                 bool task_done = (action == ACTION_CANCELLED) || (action == ACTION_FAILED) || (action == ACTION_COMPLETE);
                 if (!task_done)
                     all_done = false;
@@ -160,6 +161,14 @@
         return uuid;
     }
 
+    QString add_restore_choice(const QMap<QString,QVariant>& properties)
+    {
+        const auto uuid = QUuid::createUuid().toString();
+        auto msg = mock_iface_->call(QStringLiteral("AddRestoreChoice"), uuid, properties);
+        EXPECT_NE(QDBusMessage::ErrorMessage, msg.type()) << qPrintable(msg.errorMessage());
+        return uuid;
+    }
+
     void fail_next_helper_start()
     {
         auto msg = mock_iface_->call(QStringLiteral("FailNextHelperStart"));

=== modified file 'tests/utils/main.cpp'
--- tests/utils/main.cpp	2016-08-02 00:04:53 +0000
+++ tests/utils/main.cpp	2016-12-16 19:41:21 +0000
@@ -52,7 +52,7 @@
 
     qInstallMessageHandler(util::loggingFunction);
 
-    qsrand(time(nullptr));
+    qsrand(unsigned(time(nullptr)));
 
     QCoreApplication application(argc, argv);
     DBusMock::registerMetaTypes();