Merge lp:~unity-api-team/unity-scopes-api/leverage-net-cpp-streaming-api into lp:unity-scopes-api/devel

[Marcus Tomlinson]

Leverage the streaming API introduced in net-cpp 1.2.0.

[Marcus Tomlinson (marcustomlinson)]

This is a copy of lp:~thomas-voss/unity-scopes-api/leverage-net-cpp-streaming-api, with buffering of partial json chucks added to SmartScopesClient.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:578
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1433/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/143/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/143/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/143/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1433/rebuild

review: Needs Fixing (continuous-integration)

[Thomas Voß (thomas-voss)]

LGTM.

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-autolanding/649/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-autolanding/76/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-autolanding/75/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-autolanding/75/console

review: Needs Fixing (continuous-integration)

[Marcus Tomlinson (marcustomlinson)]

Need to wait for net-cpp to land before we can go ahead with this one.

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:579
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1435/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/145/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/145/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/145/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1435/rebuild

review: Needs Fixing (continuous-integration)

[Marcus Tomlinson (marcustomlinson)]

Re-approving as net-cpp has now landed

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-autolanding/652/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-autolanding/79/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-autolanding/78/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-autolanding/78/console

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-autolanding/653/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-autolanding/80/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-autolanding/79/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-autolanding/79/console

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:580
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1443/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/153/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/153/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/153/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1443/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:581
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1447/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/157/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/157/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/157/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1447/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:581
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1448/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/158/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/158/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/158/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1448/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:582
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1452/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/162/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/162/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/162/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1452/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

FAILED: Continuous integration, rev:584
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1453/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/163/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/163/console
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/163/console

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1453/rebuild

review: Needs Fixing (continuous-integration)

[PS Jenkins bot (ps-jenkins)]

PASSED: Continuous integration, rev:585
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-ci/1454/
Executed test runs:
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-ci/164
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/164
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-ci/164/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-ci/164

Click here to trigger a rebuild:
http://s-jenkins.ubuntu-ci:8080/job/unity-scopes-api-devel-ci/1454/rebuild

review: Approve (continuous-integration)

[Michi Henning (michihenning)]

LGTM!

review: Approve

[PS Jenkins bot (ps-jenkins)]

FAILED: Autolanding.
More details in the following jenkins job:
http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-autolanding/666/
Executed test runs:
    FAILURE: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-amd64-autolanding/93/console
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-autolanding/92
        deb: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-armhf-autolanding/92/artifact/work/output/*zip*/output.zip
    SUCCESS: http://jenkins.qa.ubuntu.com/job/unity-scopes-api-devel-vivid-i386-autolanding/92

review: Needs Fixing (continuous-integration)

[Marcus Tomlinson (marcustomlinson)]

Oh gosh, benchmark test failure... Here goes another 2 hours of build time...

[PS Jenkins bot (ps-jenkins)]

review: Approve (continuous-integration)

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2015-05-15 06:05:48 +0000
+++ CMakeLists.txt	2015-05-18 06:26:24 +0000
@@ -70,7 +70,7 @@
 pkg_check_modules(LIBACCOUNTS libaccounts-glib REQUIRED)
 pkg_check_modules(LIBSIGNON libsignon-glib REQUIRED)
 pkg_check_modules(ZMQLIB libzmq REQUIRED)
-pkg_check_modules(NET_CPP net-cpp REQUIRED)
+pkg_check_modules(NET_CPP net-cpp>=1.2.0 REQUIRED)
 
 find_library(ZMQPPLIB zmqpp)
 if(NOT ZMQPPLIB)

=== modified file 'debian/control'
--- debian/control	2015-05-15 00:25:07 +0000
+++ debian/control	2015-05-18 06:26:24 +0000
@@ -20,7 +20,7 @@
                libdbustest1-dev,
                libjsoncpp-dev,
                liblttng-ust-dev,
-               libnet-cpp-dev,
+               libnet-cpp-dev (>= 1.2.0),
                libprocess-cpp-dev (>= 1.0.1),
                libsignon-glib-dev,
                libunity-api-dev (>= 7.80.7~),

=== modified file 'include/unity/scopes/internal/smartscopes/HttpClientNetCpp.h'
--- include/unity/scopes/internal/smartscopes/HttpClientNetCpp.h	2015-01-16 06:13:57 +0000
+++ include/unity/scopes/internal/smartscopes/HttpClientNetCpp.h	2015-05-18 06:26:24 +0000
@@ -30,7 +30,7 @@
 {
 namespace http
 {
-class Client;
+class StreamingClient;
 }
 }
 }
@@ -63,7 +63,7 @@
     void cancel_get(unsigned int session_id) override;
 
     unsigned int no_reply_timeout;
-    std::shared_ptr<core::net::http::Client> client;
+    std::shared_ptr<core::net::http::StreamingClient> client;
     std::thread worker;
 };
 

=== modified file 'include/unity/scopes/internal/smartscopes/SmartScopesClient.h'
--- include/unity/scopes/internal/smartscopes/SmartScopesClient.h	2015-01-28 09:03:59 +0000
+++ include/unity/scopes/internal/smartscopes/SmartScopesClient.h	2015-05-18 06:26:24 +0000
@@ -215,8 +215,10 @@
     std::shared_ptr<DepartmentInfo> parse_departments(JsonNodeInterface::SPtr node);
     Filters parse_filters(JsonNodeInterface::SPtr node);
     FilterState parse_filter_state(JsonNodeInterface::SPtr node);
-    void parse_line(std::string const& json, SearchReplyHandler const& handler);
-    void parse_line(std::string const& json, PreviewReplyHandler const& handler);
+
+    std::string handle_chunk(const std::string& chunk, std::function<void(const std::string&)> line_handler);
+    void handle_line(std::string const& json, SearchReplyHandler const& handler);
+    void handle_line(std::string const& json, PreviewReplyHandler const& handler);
 
     std::vector<std::string> extract_json_stream(std::string const& json_stream);
 

=== modified file 'src/scopes/internal/smartscopes/HttpClientNetCpp.cpp'
--- src/scopes/internal/smartscopes/HttpClientNetCpp.cpp	2015-01-19 09:15:06 +0000
+++ src/scopes/internal/smartscopes/HttpClientNetCpp.cpp	2015-05-18 06:26:24 +0000
@@ -20,8 +20,8 @@
 
 #include <unity/UnityExceptions.h>
 
-#include <core/net/http/client.h>
-#include <core/net/http/request.h>
+#include <core/net/http/streaming_client.h>
+#include <core/net/http/streaming_request.h>
 #include <core/net/http/response.h>
 #include <core/net/http/status.h>
 
@@ -102,7 +102,7 @@
 
 HttpClientNetCpp::HttpClientNetCpp(unsigned int no_reply_timeout)
     : no_reply_timeout{no_reply_timeout},
-      client{http::make_client()},
+      client{http::make_streaming_client()},
       worker([this]() { client->run(); })
 {
 }
@@ -129,7 +129,7 @@
     }
     http_config.header = http_header;
 
-    auto request = client->get(http_config);
+    auto request = client->streaming_get(http_config);
     request->set_timeout(std::chrono::milliseconds{no_reply_timeout});
 
     auto promise = std::make_shared<std::promise<void>>();
@@ -157,12 +157,8 @@
                         }
                         else
                         {
-                            std::istringstream in(response.body);
-                            std::string line;
-                            while (std::getline(in, line))
-                            {
-                                line_data(line);
-                            }
+                            // call line_data with empty string to signal end of chunked data
+                            line_data("");
                             promise->set_value();
                         }
                     })
@@ -170,7 +166,11 @@
                     {
                         unity::ResourceException re(e.what());
                         promise->set_exception(std::make_exception_ptr(re));
-                    }));
+                    }),
+                [line_data](const std::string& const_data)
+                {
+                    line_data(const_data);
+                });
 
     return std::make_shared<HttpResponseHandle>(
                 shared_from_this(),

=== modified file 'src/scopes/internal/smartscopes/SmartScopesClient.cpp'
--- src/scopes/internal/smartscopes/SmartScopesClient.cpp	2015-04-10 03:55:25 +0000
+++ src/scopes/internal/smartscopes/SmartScopesClient.cpp	2015-05-18 06:26:24 +0000
@@ -472,18 +472,17 @@
         headers.push_back(std::make_pair("User-Agent", user_agent_hdr));
     }
 
-    auto reponse_mutex = std::make_shared<std::mutex>();
-    query_results_[search_id] = http_client_->get(search_uri.str(), [this, handler, reponse_mutex](std::string const& line_data)
+    auto tmp_data = std::make_shared<std::string>();
+    query_results_[search_id] = http_client_->get(search_uri.str(), [this, tmp_data, handler](std::string const& chunk)
     {
-        std::lock_guard<std::mutex> lock(*reponse_mutex);
-        try
-        {
-            parse_line(line_data, handler);
-        }
-        catch (std::exception const &e)
-        {
-            BOOST_LOG_SEV(logger_, Logger::Error) << "SmartScopesClient.search(): Failed to parse: " << e.what();
-        }
+        // prepend any leftover data from the previous handle_chunk call
+        std::string data = *tmp_data + (chunk.empty() ? "\n" : chunk);
+
+        // store the leftover data from the handle_chunk call into tmp_data
+        *tmp_data = handle_chunk(data, [this, handler](const std::string& line)
+        {
+            handle_line(line, handler);
+        });
     }, headers);
 
     return SearchHandle::UPtr(new SearchHandle(search_id, shared_from_this()));
@@ -540,24 +539,54 @@
 
     BOOST_LOG_SEV(logger_, Logger::Info) << "SmartScopesClient.preview(): GET " << preview_uri.str();
 
-    auto reponse_mutex = std::make_shared<std::mutex>();
-    query_results_[preview_id] = http_client_->get(preview_uri.str(), [this, handler, reponse_mutex](std::string const& line_data)
+    auto tmp_data = std::make_shared<std::string>();
+    query_results_[preview_id] = http_client_->get(preview_uri.str(), [this, tmp_data, handler](std::string const& chunk)
     {
-        std::lock_guard<std::mutex> lock(*reponse_mutex);
-        try
-        {
-            parse_line(line_data, handler);
-        }
-        catch (std::exception const &e)
-        {
-            BOOST_LOG_SEV(logger_, Logger::Error) << "SmartScopesClient.preview(): Failed to parse: " << e.what();
-        }
+        // prepend any leftover data from the previous handle_chunk call
+        std::string data = *tmp_data + (chunk.empty() ? "\n" : chunk);
+
+        // store the leftover data from the handle_chunk call into tmp_data
+        *tmp_data = handle_chunk(data, [this, handler](const std::string& line)
+        {
+            handle_line(line, handler);
+        });
     }, headers);
 
     return PreviewHandle::UPtr(new PreviewHandle(preview_id, shared_from_this()));
 }
 
-void SmartScopesClient::parse_line(std::string const& json, PreviewReplyHandler const& handler)
+std::string SmartScopesClient::handle_chunk(const std::string& chunk, std::function<void(const std::string&)> line_handler)
+{
+    // According to the docs, we expect:
+    // The response will have Content-Type
+    // application/json, it will be a chunked response, in practice a series of
+    // “\r\n” delimited lines, each containing one JSON object, with the
+    // possible forms, matching what currently can be pushed into a reply in the
+    // new scopes API
+    static constexpr const char separator{'\n'};
+
+    // read data line-by-line calling line_handler() for each
+    auto newline_pos = 0;
+    auto endline_pos = chunk.find(separator);
+    while (endline_pos != std::string::npos)
+    {
+        try
+        {
+            line_handler(chunk.substr(newline_pos, endline_pos - newline_pos));
+        }
+        catch (std::exception const &e)
+        {
+            BOOST_LOG_SEV(logger_, Logger::Error) << "SmartScopesClient.handle_chunk(): Failed to parse line: " << e.what();
+        }
+        newline_pos = endline_pos + 1;
+        endline_pos = chunk.find(separator, newline_pos);
+    }
+
+    // return the leftover data
+    return chunk.substr(newline_pos, chunk.size() - newline_pos);
+}
+
+void SmartScopesClient::handle_line(std::string const& json, PreviewReplyHandler const& handler)
 {
     JsonNodeInterface::SPtr root_node;
     JsonNodeInterface::SPtr child_node;
@@ -605,7 +634,7 @@
     }
 }
 
-void SmartScopesClient::parse_line(std::string const& json, SearchReplyHandler const& handler)
+void SmartScopesClient::handle_line(std::string const& json, SearchReplyHandler const& handler)
 {
     JsonNodeInterface::SPtr root_node;
     JsonNodeInterface::SPtr child_node;