Merge lp:~mwrius/deja-dup/GCS into lp:deja-dup/34

[Marius Nuennerich]

Add support for Google Cloud Storage.

[Michael Terry (mterry)]

Oh wow, neat!

But I'd actually be more interested in finishing the Google Drive backend. Very few desktop users are familiar with GCS.

[Marius Nuennerich (mwrius)]

I'm interested in that too. Can you point me to bugs or such that describe
what is wrong with it?

As the code for GCS is already there, how can I help to get it included
upstream?

2015-11-05 18:26 GMT+00:00 Michael Terry <email address hidden>:

> Oh wow, neat!
>
> But I'd actually be more interested in finishing the Google Drive
> backend. Very few desktop users are familiar with GCS.
> --
> https://code.launchpad.net/~mwrius/deja-dup/GCS/+merge/276188
> You are the owner of lp:~mwrius/deja-dup/GCS.
>

[Marius Nuennerich (mwrius)]

Is there anything missing to get this merged?

[Michael Terry (mterry)]

No, this looks good. I was just hesitant to add support for something called "Google Cloud Storage" which will be confusing next to "Google Drive" when we flip the switch to enable that. But if Google is OK with it... Plus, when we do that, maybe we can add a label like "only for advanced users" or some such thing to make it clearer. As if asking for a "bucket" and "secret access key" isn't intimidating enough.

As for why Google Drive support isn't finished, it's because at the time, Duplicity's Google Drive plugin was a bit in flux. The API it used was about to be deprecated by Google, and I was waiting for that to settle. It probably has now... I need to look at that. But the code is all there in deja-dup, just hidden from the UI for now.

But this branch as-is is fine, thank you! Sorry for the long delay. I'll probably move your credit in AUTHORS into the main block at the top, but that's it. Thanks!

review: Approve

[Michael Terry (mterry)]

I also switched the default order of backends to prefer S3 ahead of GCS. S3 is slightly easier for users since it autogenerates the bucket name for them and is currently still a bigger name. Ideally we'd finish Google Drive support and that can be the new favorite default. It's much simpler for the user than either S3 or GCS.

=== modified file 'AUTHORS'
--- AUTHORS	2014-10-25 13:52:17 +0000
+++ AUTHORS	2015-10-29 20:58:01 +0000
@@ -17,6 +17,10 @@
 Copyright: 2008–2013 Rosetta Contributors and Canonical Ltd
 License: GPL-3+
 
+Files: libdeja/BackendGCS.vala deja-dup/widgets/ConfigLocationGCS.vala
+Copyright: 2015 Marius Nünnerich <[e-mail]>
+License: GPL-3+
+
 Files: libdeja/uriutils.c libdeja/uriutils.h
 Copyright: 2006–2007 Red Hat, Inc
 License: GPL-3+

=== modified file 'data/org.gnome.DejaDup.gschema.xml.in'
--- data/org.gnome.DejaDup.gschema.xml.in	2014-04-29 02:38:47 +0000
+++ data/org.gnome.DejaDup.gschema.xml.in	2015-10-29 20:58:01 +0000
@@ -65,6 +65,7 @@
       <choices>
         <choice value='auto'/>
         <choice value='file'/>
+        <choice value='gcs'/>
         <choice value='gdrive'/>
         <choice value='rackspace'/>
         <choice value='s3'/>
@@ -76,6 +77,7 @@
     </key>
     <child name="rackspace" schema="org.gnome.DejaDup.Rackspace"/>
     <child name="s3" schema="org.gnome.DejaDup.S3"/>
+    <child name="gcs" schema="org.gnome.DejaDup.GCS"/>
     <child name="gdrive" schema="org.gnome.DejaDup.GDrive"/>
     <child name="file" schema="org.gnome.DejaDup.File"/>
   </schema>
@@ -96,6 +98,23 @@
       <_description>An optional folder name to store files in.  This folder will be created in the chosen bucket.</_description>
     </key>
   </schema>
+  <schema id="org.gnome.DejaDup.GCS" path="/org/gnome/deja-dup/gcs/">
+    <key name="id" type="s">
+      <default>''</default>
+      <_summary>Google Cloud Storage Access Key ID</_summary>
+      <_description>Your Google Cloud Storage Access Key Identifier.  This acts as your Google Cloud Storage username.</_description>
+    </key>
+    <key name="bucket" type="s">
+      <default>''</default>
+      <_summary>The Google Cloud Storage bucket name to use</_summary>
+      <_description>Which Google Cloud Storage bucket to store files in.  This does not need to exist already.  Only legal hostname strings are valid.</_description>
+    </key>
+    <key name="folder" type="s">
+      <default>'$HOSTNAME'</default>
+      <_summary>The Google Cloud Storage folder</_summary>
+      <_description>An optional folder name to store files in.  This folder will be created in the chosen bucket.</_description>
+    </key>
+  </schema>
   <schema id="org.gnome.DejaDup.GDrive" path="/org/gnome/deja-dup/gdrive/">
     <key name="email" type="s">
       <default>''</default>

=== modified file 'debian/control'
--- debian/control	2014-09-20 14:51:29 +0000
+++ debian/control	2015-10-29 20:58:01 +0000
@@ -36,6 +36,7 @@
             policykit-1,
 Suggests: deja-dup-backend-cloudfiles,
           deja-dup-backend-s3,
+          deja-dup-backend-gcs,
 Description: Back up your files
  Déjà Dup is a simple backup tool. It hides the complexity of backing up the
  Right Way (encrypted, off-site, and regular) and uses duplicity as the
@@ -73,6 +74,18 @@
  .
  This package adds Rackspace Cloudfiles support to Déjà Dup.
 
+Package: deja-dup-backend-gcs
+Architecture: all
+Depends: ${misc:Depends},
+         deja-dup,
+         python-boto (>= 2.20),
+Description: Google Cloud Storage support for Déjà Dup
+ Déjà Dup is a simple backup tool. It hides the complexity of backing up the
+ Right Way (encrypted, off-site, and regular) and uses duplicity as the
+ backend.
+ .
+ This package adds Google Cloud Storage support to Déjà Dup.
+
 #Package: deja-dup-backend-gdrive
 #Architecture: all
 #Depends: ${misc:Depends},

=== modified file 'deja-dup/widgets/CMakeLists.txt'
--- deja-dup/widgets/CMakeLists.txt	2014-04-29 02:38:47 +0000
+++ deja-dup/widgets/CMakeLists.txt	2015-10-29 20:58:01 +0000
@@ -31,6 +31,7 @@
                        ConfigLocationDAV.vala
                        ConfigLocationFile.vala
                        ConfigLocationFTP.vala
+                       ConfigLocationGCS.vala
                        ConfigLocationGDrive.vala
                        ConfigLocationRackspace.vala
                        ConfigLocationS3.vala

=== modified file 'deja-dup/widgets/ConfigLocation.vala'
--- deja-dup/widgets/ConfigLocation.vala	2014-04-29 02:38:47 +0000
+++ deja-dup/widgets/ConfigLocation.vala	2015-10-29 20:58:01 +0000
@@ -61,6 +61,7 @@
   int index_ftp;
   int index_dav;
   int index_s3 = -2;
+  int index_gcs = -2;
   int index_gdrive = -2;
   int index_rackspace = -2;
   int index_u1 = -2;
@@ -112,6 +113,7 @@
     // Insert cloud providers
     insert_u1();
     insert_s3();
+    insert_gcs();
     insert_gdrive();
     insert_rackspace();
 
@@ -180,6 +182,14 @@
                               ref index_s3, insert_s3);
   }
 
+  void insert_gcs() {
+    insert_cloud_if_available("gcs", BackendGCS.get_checker(),
+                              new ThemedIcon("deja-dup-cloud"),
+                              _("Google Cloud Storage"),
+                              new ConfigLocationGCS(label_sizes),
+                              ref index_gcs, insert_gcs);
+  }
+
   void insert_gdrive() {
     insert_cloud_if_available("gdrive", BackendGDrive.get_checker(),
                               new ThemedIcon("deja-dup-cloud"),
@@ -424,6 +434,8 @@
     var backend = Backend.get_default_type();
     if (backend == "s3")
       index = index_s3;
+    else if (backend == "gcs")
+      index = index_gcs;
     else if (backend == "gdrive")
       index = index_gdrive;
     else if (backend == "rackspace")
@@ -516,6 +528,8 @@
 
     if (index == index_s3)
       settings.set_string(BACKEND_KEY, "s3");
+    else if (index == index_gcs)
+      settings.set_string(BACKEND_KEY, "gcs");
     else if (index == index_gdrive)
       settings.set_string(BACKEND_KEY, "gdrive");
     else if (index == index_rackspace)

=== added file 'deja-dup/widgets/ConfigLocationGCS.vala'
--- deja-dup/widgets/ConfigLocationGCS.vala	1970-01-01 00:00:00 +0000
+++ deja-dup/widgets/ConfigLocationGCS.vala	2015-10-29 20:58:01 +0000
@@ -0,0 +1,41 @@
+/* -*- Mode: Vala; indent-tabs-mode: nil; tab-width: 2 -*- */
+/*
+    This file is part of Déjà Dup.
+    For copyright information, see AUTHORS.
+
+    Déjà Dup is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    Déjà Dup is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with Déjà Dup.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+using GLib;
+
+namespace DejaDup {
+
+public class ConfigLocationGCS : ConfigLocationTable
+{
+  public ConfigLocationGCS(Gtk.SizeGroup sg) {
+    Object(label_sizes: sg);
+  }
+
+  construct {
+    add_widget(_("Google Cloud Storage Access Key I_D"),
+               new ConfigEntry(DejaDup.GCS_ID_KEY, DejaDup.GCS_ROOT));
+    add_widget(_("_Bucket"),
+               new ConfigEntry(DejaDup.GCS_BUCKET_KEY, DejaDup.GCS_ROOT));
+    add_widget(_("_Folder"),
+               new ConfigFolder(DejaDup.GCS_FOLDER_KEY, DejaDup.GCS_ROOT));
+  }
+}
+
+}
+

=== modified file 'libdeja/Backend.vala'
--- libdeja/Backend.vala	2014-04-29 02:38:47 +0000
+++ libdeja/Backend.vala	2015-10-29 20:58:01 +0000
@@ -57,6 +57,7 @@
 
     if (backend != "auto" &&
         backend != "s3" &&
+        backend != "gcs" &&
         backend != "gdrive" &&
         backend != "rackspace" &&
         backend != "u1" &&
@@ -71,6 +72,8 @@
     var backend_name = get_default_type();
     if (backend_name == "s3")
       return new BackendS3();
+    else if (backend_name == "gcs")
+      return new BackendGCS();
     else if (backend_name == "gdrive")
       return new BackendGDrive();
     else if (backend_name == "u1")

=== modified file 'libdeja/BackendAuto.vala'
--- libdeja/BackendAuto.vala	2014-04-29 02:38:47 +0000
+++ libdeja/BackendAuto.vala	2015-10-29 20:58:01 +0000
@@ -50,6 +50,7 @@
 
   static bool started = false;
   static bool done = false;
+  Checker gcs_checker;
   Checker gdrive_checker;
   Checker s3checker;
   construct {
@@ -59,10 +60,13 @@
       started = true;
       ref(); // Give us time to finish
 
-      // List is (in order): gdrive, s3, file
+      // List is (in order): gdrive, gcs, s3, file
       gdrive_checker = BackendGDrive.get_checker();
       gdrive_checker.notify["complete"].connect(examine_checkers);
 
+      gcs_checker = BackendGCS.get_checker();
+      gcs_checker.notify["complete"].connect(examine_checkers);
+
       s3checker = BackendS3.get_checker();
       s3checker.notify["complete"].connect(examine_checkers);
 
@@ -78,11 +82,15 @@
     if (gdrive_checker.complete) {
       if (gdrive_checker.available)
         finish("gdrive");
-      else if (s3checker.complete) {
-        if (s3checker.available)
-          finish("s3");
-        else
-          finish("file");
+      else if (gcs_checker.complete) {
+        if (gcs_checker.available)
+          finish("gcs");
+        else if (s3checker.complete) {
+          if (s3checker.available)
+            finish("s3");
+          else
+            finish("file");
+        }
       }
     }
   }

=== added file 'libdeja/BackendGCS.vala'
--- libdeja/BackendGCS.vala	1970-01-01 00:00:00 +0000
+++ libdeja/BackendGCS.vala	2015-10-29 20:58:01 +0000
@@ -0,0 +1,173 @@
+/* -*- Mode: Vala; indent-tabs-mode: nil; tab-width: 2 -*- */
+/*
+    This file is part of Déjà Dup.
+    For copyright information, see AUTHORS.
+
+    Déjà Dup is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    Déjà Dup is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with Déjà Dup.  If not, see <http://www.gnu.org/licenses/>.
+*/
+
+using GLib;
+
+namespace DejaDup {
+
+public const string GCS_ROOT = "GCS";
+public const string GCS_ID_KEY = "id";
+public const string GCS_BUCKET_KEY = "bucket";
+public const string GCS_FOLDER_KEY = "folder";
+
+const string GCS_SERVER = "www.googleapis.com";
+
+public class BackendGCS : Backend
+{
+  public static Checker get_checker() {
+    return PythonChecker.get_checker("boto");
+  }
+
+  public override Backend clone() {
+    return new BackendGCS();
+  }
+  
+  public override bool is_native() {
+    return false;
+  }
+  
+  public override Icon? get_icon() {
+    return new ThemedIcon("deja-dup-cloud");
+  }
+
+  public override async bool is_ready(out string when) {
+    when = _("Backup will begin when a network connection becomes available.");
+    return yield Network.get().can_reach ("http://%s/".printf(GCS_SERVER));
+  }
+
+  public override string get_location(ref bool as_root)
+  {
+    var settings = get_settings(GCS_ROOT);
+    
+    var bucket = settings.get_string(GCS_BUCKET_KEY);
+    var folder = get_folder_key(settings, GCS_FOLDER_KEY);
+
+    return "gs://%s/%s".printf(bucket, folder);
+  }
+  
+  public override string get_location_pretty()
+  {
+    var settings = get_settings(GCS_ROOT);
+    var bucket = settings.get_string(GCS_BUCKET_KEY);
+    var folder = get_folder_key(settings, GCS_FOLDER_KEY);
+    if (folder == "")
+      return _("Google Cloud Storage");
+    else
+      // Translators: %s/%s is a folder.
+      return _("%s/%s on Google Cloud Storage").printf(bucket, folder);
+  }
+  
+  string settings_id;
+  string id;
+  string secret_key;
+  public override async void get_envp() throws Error
+  {
+    var settings = get_settings(GCS_ROOT);
+    settings_id = settings.get_string(GCS_ID_KEY);
+    id = settings_id == null ? "" : settings_id;
+    
+    if (id != "" && secret_key != null) {
+      // We've already been run before and got the key
+      got_secret_key();
+      return;
+    }
+    
+    if (id != "") {
+      // First, try user's keyring
+      try {
+        secret_key = yield Secret.password_lookup(Secret.SCHEMA_COMPAT_NETWORK,
+                                                  null, 
+                                                  "user", id,
+                                                  "server", GCS_SERVER,
+                                                  "protocol", "https");
+        if (secret_key != null) {
+          got_secret_key();
+          return;
+        }
+      }
+      catch (Error e) {
+        // fall through to ask_password below
+      }
+    }
+
+    // Didn't find it, so ask user
+    ask_password();
+  }
+
+  async void got_password_reply(MountOperation mount_op, MountOperationResult result)
+  {
+    if (result != MountOperationResult.HANDLED) {
+      envp_ready(false, new List<string>(), _("Permission denied"));
+      return;
+    }
+
+    id = mount_op.username;
+    secret_key = mount_op.password;
+
+    // Save it
+    var remember = mount_op.password_save;
+    if (remember != PasswordSave.NEVER) {
+      string where = (remember == PasswordSave.FOR_SESSION) ?
+                     Secret.COLLECTION_SESSION : Secret.COLLECTION_DEFAULT;
+      try {
+        yield Secret.password_store(Secret.SCHEMA_COMPAT_NETWORK,
+                                    where,
+                                    "%s@%s".printf(id, GCS_SERVER),
+                                    secret_key,
+                                    null,
+                                    "user", id,
+                                    "server", GCS_SERVER,
+                                    "protocol", "https");
+      }
+      catch (Error e) {
+        warning("%s\n", e.message);
+      }
+    }
+
+    got_secret_key();
+  }
+
+  void ask_password() {
+    mount_op.set("label_help", _("You can sign up for a Google Cloud Storage account <a href=\"%s\">online</a>. Remember to enable Interoperability and create keys.").printf("http://cloud.google.com"));
+    mount_op.set("label_title", _("Connect to Google Cloud Storage"));
+    mount_op.set("label_username", _("_Access key ID"));
+    mount_op.set("label_password", _("_Secret access key"));
+    mount_op.set("label_show_password", _("S_how secret access key"));
+    mount_op.set("label_remember_password", _("_Remember secret access key"));
+    mount_op.reply.connect(got_password_reply);
+    mount_op.ask_password("", id, "",
+                          AskPasswordFlags.NEED_PASSWORD |
+                          AskPasswordFlags.NEED_USERNAME |
+                          AskPasswordFlags.SAVING_SUPPORTED);
+  }
+  
+  void got_secret_key() {
+    var settings = get_settings(GCS_ROOT);
+    if (id != settings_id)
+      settings.set_string(GCS_ID_KEY, id);
+    
+    List<string> envp = new List<string>();
+    envp.append("GS_ACCESS_KEY_ID=%s".printf(id));
+    envp.append("GS_SECRET_ACCESS_KEY=%s".printf(secret_key));
+    envp_ready(true, envp);
+  }
+}
+
+} // end namespace
+

=== modified file 'libdeja/CMakeLists.txt'
--- libdeja/CMakeLists.txt	2014-04-29 02:38:47 +0000
+++ libdeja/CMakeLists.txt	2015-10-29 20:58:01 +0000
@@ -20,6 +20,7 @@
                        Backend.vala
                        BackendAuto.vala
                        BackendFile.vala
+                       BackendGCS.vala
                        BackendGDrive.vala
                        BackendRackspace.vala
                        BackendS3.vala