[FFE] CouchDB 1.4.0 needed to work with Erlang 16.b.1

Bug #1212481 reported by Jason Gerard DeRose
48
This bug affects 8 people
Affects Status Importance Assigned to Milestone
couchdb (Ubuntu)
Fix Released
High
Jason Gerard DeRose

Bug Description

CouchDB 1.2.0 doesn't work at all with Erlang 16.b.1, so the current CouchDB package in Saucy is completely broken and should be removed if it can't be upgraded.

To work with Erlang 16.b.1, we'll have to upgrade to the recently released CouchDB 1.4.0. The needed compatibility fixes are extensive and include upgrading CouchDB's internal mochiweb server, so backporting the needed changes to CouchDB 1.2.0 isn't practical.

Following is an overview of the testing I've done, results, and known issues:

== Unit Tests ==

CouchDB 1.4.0 now has quite extensive unit tests, which I'm running during the build. These tests are (generally) passing fine when building with pbuilder, and when building in a PPA (i386 and amd64).

Note I've not tried a build on armhf yet.

There are a few unit tests that occasionally fail when building on Raring, but so far I haven't encountered this when building on Saucy. It is however possible to get successful builds on Raring, you just might have to retry a build or two.

== Reverse Dependencies ==

I've built several CouchDB consumers against my proposed CouchDB package in a PPA, on both Saucy and Raring. These reverse dependencies have extensive CouchDB using unit tests and are therefor a good measure of the correctness of my proposed CouchDB 1.4.0 package, and the quality of the upstream CouchDB 1.4.0 release.

UserCouch: https://launchpad.net/usercouch
  * Good coverage of key configuration permutations (bind address, port, basic auth, oauth, file_compression)
  * Good coverage of SSL options, both for httpd and the replicator
  * Provides good confidence that CouchDB actually starts, and starts in a timely manner, across all config permutations
  * IPv4 and IPv6 tests

Microfiber: https://launchpad.net/microfiber
  * Good coverage of core DB, doc, and attachment REST APIs
  * Deep tests for _bulk_docs API, testing both "non-atomic" and "all-or-nothing" update semantics
  * Basic view tests
  * Basic replication tests
  * IPv4 and IPv6 tests

Dmedia: https://launchpad.net/dmedia
  * Complex, demanding application with lots of "real-world" test scenarios
  * Extensive view tests with complex map functions (but _count, _sum, and _stats are the only reduce functions Dmedia uses)
  * Full-stack replication tests with SSL (and client-side SSL certs)
  * Tests for complex conflict scenarios

== Manual Testing ==

I've done extensive manual testing with Dmedia. I peered 3 different computers and imported around 20k files into Dmedia, made sure all metadata got replicated between all peers. I ran through my standard battery of abuse using things like PurgeAll and DowngradeAll, and in all scenarios the library successfully converged to its equilibrium state in a reasonable amount of time.

I've also done extensive manual testing with the `couchdb` binary package and its new upstart job. I confirmed that:
  1) The daemon is started when the package is installed
  2) The daemon starts during the boot sequence after a reboot and a cold boot
  3) All daemon processes are killed with `sudo stop couchdb`
  4) You can start the daemon with `sudo start couchdb`
  5) `sudo restart couchdb` works as expected
  6) Upstart respawn works as expected if I manually `kill $PID` on the daemon process
  7) All daemon processes are killed with `sudo apt-get remove couchdb`
  8) `sudo apt-get purge couchdb` removes /var/lib/couchdb and /var/log/couchdb

== Upgrade Testing ==

I tested the upgrade from 1.2.0 when only `couchdb-bin` is installed, and when `couchdb` and `couchdb-bin` are installed. Both work as expected, although these tests were done on Raring. I haven't yet done an upgrade test via a Raring=>Saucy update, but that's difficult to do when 1.4.0 is still only in a PPA (update process can't be properly simulated as the PPA will be disabled during the update).

A remaining issue is that sometimes the `couchdb` 1.2.0 daemon is not successfully terminated during the package upgrade. (Well, it sort of is, and then Erlang respawns more processes that don't ever get killed.) So you'll end up with the new, correct 1.4.0 processes running, plus some lingering 1.2.0 garbage processes running. This is due to fundamental problems in the upstream init.d script, and this is a big part of why I switched to Upstart.

A work-around is to stop couchdb before you upgrade:
  sudo service couchdb stop

Or simply reboot after you upgrade (which most will do anyway).

This is probably a difficult issue to solve as the problem is that the previous package version isn't correctly shutting down its CouchDB daemon. On the upside, no such problems exist with the Upstart job in my 1.4.0 package.

Tags: saucy

Related branches

CVE References

description: updated
Revision history for this message
Chad Miller (cmiller) wrote :

Debhelper is the way, and you're using it properly. Kill CDBS.

I like and approve of just taking it over and doing it right, Jason.

I'm not a MOTU either, but I will look over the packaging if and when you want.

Revision history for this message
Micah Gersten (micahg) wrote :

Have you spoken with the Debian maintainer about co-maintaining it in Debian? In general we prefer to not differ too much if possible, so if the Debian package is using CDBS, we should probably be doing the same. If 1.4 can run with an 8 line Debhelper file, maybe the Debian maintain would consider switching?
Debian has an RC bug for the erlang issue, so it needs to be solve one way or another: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=714263
If you have a patch for the Debian bug for erlang16b + 1.2.x, that could probably get sponsored pretty easily.
An upgrade to 1.4.x probably just needs a conversation with the Debian maintainer.

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Thanks, Chad! I'll fix the things I already know aren't right, and then ping you for a review.

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Micah,

Yes, over a year ago I sent a few emails to the debian maintainer, never heard back. Maybe email isn't the best way to reach him, but I don't have the time for much process either.

It's not practical for 1.2.x (or 1.3.x) to work with Erlang 16b... it requires a big, non trivial changes, including upgrading their internal copy of mochiweb. So realistically, it's get 1.4.0 directly into Ubuntu before the feature freeze, or have no CouchDB is Saucy.

I think the CouchDB debian maintainer doesn't have the time or has perhaps lost interest. The current version in Debian anything is CouchDB 1.2.0, which was released in Apr 2012. Since then there has been:

1.2.1: Jan 2013
1.2.2: Apr 2013
1.3.0: Apr 2013
1.3.1: Jun 2013

I'm not slighting the maintainer here... there's a lot more to life than maintaining Debian packages, and I'm assuming he's been generously *volunteering* his time. He's already given me an abundance to work with.

But at the end of the day, my goal isn't maintaining CouchDB. I'm building a product on Ubuntu that happens to use CouchDB, and so I absolutely need CouchDB in Saucy and will do whatever it takes to make that happen, even if I'm maintaining it myself in a PPA (which doesn't actually take that much time). I've only stepped in when things haven't worked out on their own.

As I'm doing this packaging work anyway, I think Ubuntu might as well include it in Universe. Better than having no package at all, or one so broken everyone wishes it wasn't there. But it's really hard for me to justify the time it would take to coordinate with debian, get consensus on the packages changes I need, etc. Even if I could, I don't think that could happen before Saucy is released, yet alone before the feature freeze.

Anyway, blah blah blah... bed time for me =)

Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in couchdb (Ubuntu):
status: New → Confirmed
Changed in couchdb (Ubuntu):
assignee: nobody → Jason Gerard DeRose (jderose)
Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Quick update: I've now managed to build all the Novacut components against my latest 1.4.0 package in Saucy. Having all the Novacut unit tests pass in the build environment makes me pretty confident about the correctness of the packaging, and the general quality of 1.4.0.

The Novacut Daily PPA now also has a 1.4.0 package for Raring:
https://launchpad.net/~novacut/+archive/daily?field.series_filter=raring

I haven't decided whether or not I'll push this out to the Novacut Stable PPA when Novacut 13.08 is released, but I figured this is a way to help it get more testing ASAP.

Revision history for this message
janl (jan-apache) wrote :

FWIW, the CouchDB developers support jderose’s plan. We are happy to help in any way we can.

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Thanks, Jan!

The biggest thing I need is an official CouchDB 1.4.0 release tarball, ideally before the feature freeze on August 29th :)

Also, if you wouldn't mind looking over my dependencies to see if I'm missing anything:
http://bazaar.launchpad.net/~jderose/+junk/couchdb/view/head:/debian/control

And checking whether there are any other options you recommend I used when calling ./configure:
http://bazaar.launchpad.net/~jderose/+junk/couchdb/view/head:/debian/rules#L6

(Obviously I wont be calling ./bootstrap with the release tarball.)

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Using the upstream 1.4.0-rc.1 tarball, I've been working on the UDD merge against Saucy in this branch:

https://code.launchpad.net/~jderose/ubuntu/saucy/couchdb/1.4.0

I'm still not super UDD savvy, so I'm not 100% confident I didn't goof anywhere.

The diff of just debian/ looks pretty sane and understandable, I think, and I'll continue to refine things while waiting on the final upstream 1.4.0 tarball.

Chad, I think it's ready for review, whenever you have time. There are still three small issues that Debian would probably frown upon:

1) I'm not using the shared libjs-jquery package because it's only at version 1.7.2, whereas the internal CouchDB jquery is now at version 1.8.3; I don't think it's worth my time (or anyone else's) to try and make sure Futon, etc works fantastically against 1.7.2, to say nothing of porting it to 1.7.2

2) CouchDB now has built-in docs in Futon (pretty cool, really). The docs are built using Sphinx, but they use a somewhat odd config or layout that breaks dh_sphinxdoc, so for now, I'm not depending on ${sphinxdoc:Depends}, am not properly symlinking to the shared files in libjs-sphinxdoc

3) It would be much better to build against the shared libsnappy-dev rather than CouchDB's internal copy, but I haven't figured out how to do that yet

Oh, and I was thinking I should probably split the static files in /usr/share/couchdb out into an Architecture: all package called `couchdb-common` or somesuch, as this directory is getting fairly hefty these days (almost 5 MB)

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Another update: so I went again and split /usr/share/couchdb out into a new `couchdb-common` package, as that wasn't much work and makes sense considering how large the data and docs in there have become.

I have builds for Raring and Saucy in the Novacut Daily PPA:
https://launchpad.net/~novacut/+archive/daily

And here's my current changelog:

couchdb (1.4.0~rc.1-0ubuntu1) saucy; urgency=low

  * New upstream release (LP: #1022515)
  * Switch from CDBS to pure debhelper (compat 9)
  * Bump Standards-Version to 3.9.4
  * Use an Upstart job instead of the upstream SysV init.d script
  * Remove Build-Depends: cdbs, libreadline-dev
  * Add Build-Depends: erlang-os-mon, erlang-syntax-tools, python-sphinx,
    texlive-latex-base, texlive-latex-recommended, texlive-latex-extra,
    texlive-fonts-recommended, texinfo
  * Remove couchdb-bin Depends: procps, lsb-base (needed for SysV init.d script)
  * Remove couchdb-bin Depends: libjs-jquery (1.7.2 is in Saucy, but the
    internal CouchDB jquery is now at version 1.8.3)
  * Simplify Erlang couchdb-bin Depends to just ${erlang-abi:Depends}, ${erlang:Depends}
  * Add couchdb Depends: upstart
  * Remove deprecated couchdb-bin.postinst, couchdb-bin.postrm
  * Thanks to the Upstart job, couchdb.postrm no longer needs `sleep 3` hack,
    nor needs to `rm -r -f "/var/run/couchdb"`
  * Stop using versioned database_dir /var/lib/couchdb/VERSION as this isn't
    done upstream and CouchDB is no longer considered alpha software
  * Remove README.Debian, README.source as they're no longer applicable
  * Drop patches superseded upstream for CVE-2012-5649, CVE-2012-5650:
    - improve_parsing_of_mochiweb_relative_paths.patch
    - improve_script_url_validation.patch
    - include_a_comment_before_jsonp_output.patch
  * Because of the switch to Upstart, drop unneeded SysV init.d script patches:
    - force-reload.patch
    - couchdb_own_rundir.patch
    - wait_for_couchdb_stop.patch
  * Drop couchdb_sighup.patch, superseded upstream
  * Drop logrotate_as_couchdb.patch as it doesn't make sense for the CouchDB
    daemon to be able to modify its own archived log files
  * Move static data and docs in "/usr/share/couchdb" from `couchdb-bin` into
    new `couchdb-common` Architecture:all package
  * Add couchdb-bin Depends: couchdb-common (= ${source:Version})

Revision history for this message
Chad Miller (cmiller) wrote :

Debdiff attached.

Changed in couchdb (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Micah Gersten (micahg) wrote :

This has now been uploaded to Debian unstable, I'd suggest updating this bug for an Feature Freeze exception and let's merge from Debian.

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Micah,

I noticed the 1.4.0 package in Debian, and it seems like some of my packaging work played into it (so I'm glad that my work was perhaps useful there too).

However, I'd argue the at this point merging from Debian is pedantic and provides no benefit to Ubuntu users. There is precious little time till Saucy is released, and I can either spend that time further testing and perfecting my current package, or doing lots tedious merging and conflict resolution.

I'd also argue that my package is currently in a much higher quality state than the Debian 1.4.0-3 package:

1) I run the upstream unit tests during the build, the Debian package doesn't

2) In the first two 1.4.0 revs of the Debian package, the CouchDB system daemon wouldn't even start, which suggests to me it's not being tested very thoroughly:
http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=721322

3) My package has been verified by successfully building several reverse-dependency against it in the unforgiving Ubuntu build environment... reverse-dependencies with extensive, demanding CouchDB unit tests

4) I've extensively tested the 1.2.0 to 1.4.0 upgrade on Ubuntu in many scenarios (the only problem I found was that sometimes the 1.2.0 system daemon doesn't actually get killed, because the upstream init.d script is just kinda broken, despite several loving patches from the Debian maintainer... which is why I feel moving to Upstart is very important for the Ubuntu package).

I have no problem working with the Debian maintainer to get the Debian and Ubuntu packages closer over time, ideally to reach a point where we're using the exact same package (well, aside from Upstart, obviously that doesn't apply in Debian).

But I strongly feel the time to do that is not now, two days after the feature freeze. That's something to work on for 14.04 LTS.

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Okay, I guess pedantic or not, I have a version that merges 1.4.0-3 from Debian here:

https://code.launchpad.net/~jderose/ubuntu/saucy/couchdb/1.4.0-from-debian

This reduces the diff (from Debian) for of a lot of things that don't have any technical impact (debian/copyright, debian/watch, etc), but it uses the exact debian/control and debian/rules I was using before.

From what I can tell, there are a number of things that are currently wrong in lp:debian/couchdb, like it's not building the Sphinx docs, which it really should as they are now prominently exposed in the Futon UI. I'll look those changes over more and see if there is anything I should incorporate.

But for now, I want err on the side of caution and stick with what I've already tested extensively.

Revision history for this message
Micah Gersten (micahg) wrote :

This bug still needs a Feature Freeze exception before it can be uploaded: https://wiki.ubuntu.com/FreezeExceptionProcess

Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Micah, is it best to modify this bug or file a new one?

Revision history for this message
Micah Gersten (micahg) wrote :

Modifying this one is fine (just update the description appropriately)

Changed in couchdb (Ubuntu):
status: In Progress → New
summary: - Saucy: CouchDB 1.4.0 needed to work with Erlang 16.b.1
+ [FFE] CouchDB 1.4.0 needed to work with Erlang 16.b.1
description: updated
Revision history for this message
Jason Gerard DeRose (jderose) wrote :
Revision history for this message
Scott Kitterman (kitterman) wrote :

Ack. Approved. It seems pretty clearly better to go forward.

Would it be possible to SRU 1.2 to fix the upgrade problem?

Changed in couchdb (Ubuntu):
importance: Undecided → High
status: New → Triaged
Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Scott, thanks for the Ack!

Assuming someone can figure out why CouchDB 1.2.0 isn't getting completely terminated during the package upgrade, I think this would be a great thing to SRU. But I know it's a tricky issue that the Debian maintainer has battled a lot, and so have I. I'll see what I can come up with, but this is a long-standing issue... so I'm not holding my breath.

Note that I'm still a lot more comfortable merging this branch, the one I worked on before 1.4.0 was in Debian unstable:
https://code.launchpad.net/~jderose/ubuntu/saucy/couchdb/1.4.0

The other branch was done in a big rush and hasn't been extensively tested:
https://code.launchpad.net/~jderose/ubuntu/saucy/couchdb/1.4.0-from-debian

I'm pretty sure there are no consequential differences between the two, but I feel the 2nd branch introduces lots last-minute risk without a clear benefit to Ubuntu users during the Saucy lifetime (as Saucy will only be supported for 9 months anyway).

I promise to be good and work with the Debian maintainer on getting the Ubuntu and Debian CouchDB packages much closer for 14.04 LTS, where easy merging from Debian is important for security fixes during the 5 years of support =)

Revision history for this message
Andrew Starr-Bochicchio (andrewsomething) wrote :

Hi Jason,

I'm inclined to go ahead and upload lp:~jderose/ubuntu/saucy/couchdb/1.4.0 It LGTM and it's certainly the way forward.

One last question, have you done any testing of python-couchdb against this version?

Revision history for this message
Andrew Starr-Bochicchio (andrewsomething) wrote : Re: [Bug 1212481] Re: [FFE] CouchDB 1.4.0 needed to work with Erlang 16.b.1

On Mon, Sep 16, 2013 at 2:36 PM, Andrew Starr-Bochicchio
<email address hidden> wrote:
> One last question, have you done any testing of python-couchdb against
> this version?

I tested this myself. All of python-couchdb's unittests pass against
the new version.

Uploading now. Thanks for all your work on this!

-- Andrew Starr-Bochicchio

   Ubuntu Developer <https://launchpad.net/~andrewsomething>
   Debian Developer <http://qa.debian.org/developer.php?login=asb>
   PGP/GPG Key ID: D53FDCB1

Revision history for this message
Andrew Starr-Bochicchio (andrewsomething) wrote :

This has been uploaded, but couchdb-common is in NEW:

https://launchpad.net/ubuntu/saucy/+queue

Changed in couchdb (Ubuntu):
status: Triaged → Fix Committed
Changed in couchdb (Ubuntu):
status: Fix Committed → Fix Released
Revision history for this message
Jason Gerard DeRose (jderose) wrote :

Thanks, Andrew!

BTW, I didn't do any testing with python-couchdb as I'm not very familiar with it these days. couchdbkit is another good reverse dependency to test against.

Revision history for this message
Hendy Irawan (ceefour) wrote :

Seems like CouchDB 1.4.0 on 13.10 (I'm using Mint 16 though) crashes even on a clean install :(

dmesg shows:

[ 2107.500012] init: couchdb main process (11934) terminated with status 1
[ 2107.500061] init: couchdb main process ended, respawning
[ 2107.829258] init: couchdb main process (11976) terminated with status 1
[ 2107.829338] init: couchdb main process ended, respawning
[ 2108.162798] init: couchdb main process (12017) terminated with status 1
[ 2108.162871] init: couchdb main process ended, respawning
[ 2108.522668] init: couchdb main process (12058) terminated with status 1
[ 2108.522710] init: couchdb main process ended, respawning
[ 2108.834284] init: couchdb main process (12098) terminated with status 1
[ 2108.834304] init: couchdb main process ended, respawning
[ 2109.173867] init: couchdb main process (12138) terminated with status 1
[ 2109.173931] init: couchdb main process ended, respawning
[ 2109.534902] init: couchdb main process (12181) terminated with status 1
[ 2109.534928] init: couchdb main process ended, respawning
[ 2109.875314] init: couchdb main process (12221) terminated with status 1
[ 2109.875339] init: couchdb main process ended, respawning
[ 2110.200497] init: couchdb main process (12263) terminated with status 1
[ 2110.200518] init: couchdb main process ended, respawning
[ 2110.514901] init: couchdb main process (12305) terminated with status 1
[ 2110.514924] init: couchdb main process ended, respawning
[ 2110.870927] init: couchdb main process (12346) terminated with status 1
[ 2110.870964] init: couchdb respawning too fast, stopped

but there's nothing in /var/log/couchdb

starting manually gives:

> sudo -u couchdb couchdb
{"init terminating in do_boot",{{badmatch,{error,{bad_return,{{couch_app,start,[normal,["/etc/couchdb/default.ini","/etc/couchdb/local.ini"]]},{'EXIT',{{badmatch,{error,{error,enoent}}},[{couch_server_sup,start_server,1,[{file,"couch_server_sup.erl"},{line,56}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,269}]}]}}}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_it,1,[]},{init,start_em,1,[]}]}}
init terminating in do_boot ()

is there existing bug report for this? using couchdb 1.4.0-0ubuntu1

Revision history for this message
Hendy Irawan (ceefour) wrote :

I reported bug #1262220

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.