Merge lp:~charlesk/keeper/fix-slow-tar-creator-test into lp:keeper/devel

Proposed by Charles Kerr
Status: Merged
Approved by: Charles Kerr
Approved revision: 143
Merged at revision: 122
Proposed branch: lp:~charlesk/keeper/fix-slow-tar-creator-test
Merge into: lp:keeper/devel
Prerequisite: lp:~xavi-garcia-mena/keeper/restore
Diff against target: 98 lines (+31/-29)
1 file modified
src/tar/tar-creator.cpp (+31/-29)
To merge this branch: bzr merge lp:~charlesk/keeper/fix-slow-tar-creator-test
Reviewer Review Type Date Requested Status
Xavi Garcia (community) Approve
unity-api-1-bot continuous-integration Approve
Review via email: mp+311984@code.launchpad.net

Commit message

Fix TarCreator::step() to block less

Description of the change

Have TarCreator::step() take smaller steps so that in practice the function won't block.

Also, fix side bug: step() wasn't handling for the case of archive_write_data() returning a non-error status of having written less than the full buffer size.

To post a comment you must log in.
Revision history for this message
unity-api-1-bot (unity-api-1-bot) wrote :

FAILED: Continuous integration, rev:143
No commit message was specified in the merge proposal. Click on the following link and set the commit message (if you want a jenkins rebuild you need to trigger it yourself):
https://code.launchpad.net/~charlesk/keeper/fix-slow-tar-creator-test/+merge/311984/+edit-commit-message

https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/143/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build/1199
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-0-fetch/1206
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/992
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/992/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/992
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/992/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/992
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/992/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/992
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/992/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/992
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/992/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/992
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/992/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/143/rebuild

review: Needs Fixing (continuous-integration)
Revision history for this message
unity-api-1-bot (unity-api-1-bot) wrote :

PASSED: Continuous integration, rev:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/144/
Executed test runs:
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build/1206
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-0-fetch/1213
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/999
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=xenial+overlay/999/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/999
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=amd64,release=zesty/999/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/999
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=xenial+overlay/999/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/999
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=armhf,release=zesty/999/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/999
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=xenial+overlay/999/artifact/output/*zip*/output.zip
    SUCCESS: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/999
        deb: https://jenkins.canonical.com/unity-api-1/job/build-2-binpkg/arch=i386,release=zesty/999/artifact/output/*zip*/output.zip

Click here to trigger a rebuild:
https://jenkins.canonical.com/unity-api-1/job/lp-keeper-ci/144/rebuild

review: Approve (continuous-integration)
Revision history for this message
Xavi Garcia (xavi-garcia-mena) wrote :

Thanks for this, Charles...
It works much better that it did.

review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'src/tar/tar-creator.cpp'
--- src/tar/tar-creator.cpp 2016-09-05 18:38:36 +0000
+++ src/tar/tar-creator.cpp 2016-11-28 21:38:06 +0000
@@ -78,24 +78,20 @@
78 step_filenum_ = -1;78 step_filenum_ = -1;
79 }79 }
8080
81 for(;;)81 // if we don't have a file we're working on, then get one
82 if (!step_file_)
82 {83 {
83 // if we don't have a file we're working on, then get one84 if (step_filenum_ >= filenames_.size()) // tried to read past the end
84 if (!step_file_)85 {
85 {86 success = false;
86 if (step_filenum_ >= filenames_.size()) // tried to read past the end87 }
87 {88 // step to next file
88 success = false;89 else if (++step_filenum_ == filenames_.size()) // we made it to the end!
89 break;90 {
90 }91 archive_write_close(step_archive_.get());
9192 }
92 // step to next file93 else
93 if (++step_filenum_ == filenames_.size()) // we made it to the end!94 {
94 {
95 archive_write_close(step_archive_.get());
96 break;
97 }
98
99 // write the file's header95 // write the file's header
100 const auto& filename = filenames_[step_filenum_];96 const auto& filename = filenames_[step_filenum_];
101 add_file_header_to_archive(step_archive_.get(), filename);97 add_file_header_to_archive(step_archive_.get(), filename);
@@ -104,15 +100,24 @@
104 step_file_.reset(new QFile(filename));100 step_file_.reset(new QFile(filename));
105 step_file_->open(QIODevice::ReadOnly);101 step_file_->open(QIODevice::ReadOnly);
106 }102 }
103 }
107104
105 if (step_file_)
106 {
108 static constexpr int BUFSIZE {1024*10};107 static constexpr int BUFSIZE {1024*10};
109 char inbuf[BUFSIZE];108 char inbuf[BUFSIZE];
110 const auto n = step_file_->read(inbuf, sizeof(inbuf));109 auto inbuf_len = step_file_->read(inbuf, sizeof(inbuf));
111 if (n > 0) // got data110 if (inbuf_len > 0) // got data
112 {111 {
113 for(;;) {112 decltype(inbuf_len) offset = 0;
114 if (archive_write_data(step_archive_.get(), inbuf, size_t(n)) != -1)113 while(offset < inbuf_len) {
115 break;114 auto const n_written = archive_write_data(step_archive_.get(), inbuf+offset, inbuf_len-offset);
115 if (n_written > 0) {
116 offset += n_written;
117 if (offset == inbuf_len)
118 break;
119 continue;
120 }
116 const auto err = archive_errno(step_archive_.get());121 const auto err = archive_errno(step_archive_.get());
117 if (err == ARCHIVE_RETRY)122 if (err == ARCHIVE_RETRY)
118 continue;123 continue;
@@ -125,22 +130,19 @@
125 throw std::runtime_error(errstr.toStdString());130 throw std::runtime_error(errstr.toStdString());
126 }131 }
127 }132 }
128 else if (n < 0) // read error133 else if (inbuf_len < 0) // read error
129 {134 {
130 success = false;135 success = false;
131 auto errstr = QStringLiteral("read()ing %1 returned %2 (%3)")136 auto errstr = QStringLiteral("read()ing %1 returned %2 (%3)")
132 .arg(step_file_->fileName())137 .arg(step_file_->fileName())
133 .arg(n)138 .arg(inbuf_len)
134 .arg(step_file_->errorString());139 .arg(step_file_->errorString());
135 qWarning() << errstr;140 qWarning() << errstr;
136 throw std::runtime_error(errstr.toStdString());141 throw std::runtime_error(errstr.toStdString());
137 }142 }
138 else if (step_file_->atEnd()) // eof143
139 {144 if (step_file_->atEnd()) // if we're done with the file, close it
140 // loop to next file
141 step_file_.reset();145 step_file_.reset();
142 continue;
143 }
144 }146 }
145147
146 std::swap(fillme,step_buf_);148 std::swap(fillme,step_buf_);

Subscribers

People subscribed via source and target branches

to all changes: