lp:couchdb

Author: Nick V.
Author Date: 2024-04-24 04:35:56 UTC

Remove replication job supervisor

Use the scheduler as the job supervisor, since the scheduler is already a fancy
supervisor, with its own backoff logic, process monitoring, etc.

This simplifies the job starting/stopping logic and fixes a bug where the
simple_one_for_one supervisor could restart a job, but the scheduler would
consider it not running, and try to start another job with the same replication
ID on the same node. Since jobs register themselves in pg, the second job would
keep crashing with duplicate_job error the first time it tried to checkpoint.

Author: Nick V.
Author Date: 2024-04-23 20:02:58 UTC

Modernize couch replicator supervisor

Use maps and a few helper function to make it less verbose.

Otherwise it should be the exact same specs as before.

References:
  https://www.erlang.org/doc/design_principles/sup_princ.html
  https://www.erlang.org/doc/man/supervisor
  https://www.erlang.org/docs/17/man/supervisor.html

Author: Robert Newson
Author Date: 2024-04-17 15:33:10 UTC

Optional account lockout facility on repeated auth failure

Author: Nick Vatamaniuc
Author Date: 2022-11-26 05:14:01 UTC

Add QuickJS as a Javascript engine option

https://bellard.org/quickjs

Some benefits over SM:

 * Small. We're using 6 or so C files vs 700+ SM91 C++ files.

 * Built with Apache CouchDB as opposed having to maintain a separate SM
   package, like for RHEL9, for instance, where they dropped support for SM
   already. (see https://github.com/apache/couchdb/issues/4154).

 * Embedding friendly. Designed from ground-up for embedding. SM has been
   updating the C++ API such that we have to keep copy-pasting new versions of
   our C++ code every year or so. (see
   https://github.com/apache/couchdb/pull/4305).

 * Easy to modify to accept Spidermonkey 1.8.5 top level functions for
   map/reduce code so we don't have have to parse the JS, AST transform it, and
   then re-compile it.

 * Configurable runtime feature set - can disable workers, promises and other
   API and features which may not work well in a backend JS environment. Some
   users may want more, some may want to disable even Date(time) features to
   hedge again Spectre-style attacks (spectreattack.com).

 * Allows granular time (reduction) tracking if we wanted to provide a runtime
   allowance for each function.

 * Better sandboxing. Creating a whole JSRuntime takes only 300 microseconds, so
   we can afford to do that on reset. JSRuntimes cannot share JS data or object
   between them.

 * Seems to be faster in preliminary benchmarking with small
   concurrent VDU and view builds:
     https://gist.github.com/nickva/ed239651114794ebb138b1f16c5f6758
   Results seem promising:
     - 4x faster than SM 1.8.5
     - 5x faster than SM 91
     - 6x reduced memory usage per couchjs process (5MB vs 30MB)

 * Allows compiling JS bytecode ahead of time a C array of bytes.

QuickJS can be built alongside Spidermonkey and toggled on/off at runtime:

```
./configure --dev --js-engine=quickjs
```

This makes it the default engine. But Spidermonkey can still be set in the
config option.

```
[couchdb]
js_engine = spidermonkey | quickjs
```

To test individual views, without switching the default use the
`javascript_quickjs` language in the design docs. To keep using Spidermonkey
engine after switching the default, can use `javascript_spidermonkey` language
in design docs. However, language selection will reset the view and the view
will have to be rebuilt.

It's also possible to build without Spidermonkey support completely by using:
```
./configure --disable-spidermonkey
```

Issue: https://github.com/apache/couchdb/issues/4448

Author: Russell Branca
Author Date: 2024-04-11 22:17:15 UTC

Add dedicated monitor processes option

Author: Russell Branca
Author Date: 2024-03-05 20:35:06 UTC

Fix mango special cursor test

Author: Russell Branca
Author Date: 2024-02-12 22:34:04 UTC

Limit resource usage output and fix keys

Author: Nick V.
Author Date: 2024-01-11 00:12:28 UTC

Pick up dev/run defaults from install.mk

To avoid having to specify --enable-nouveau and --with-nouveau twice.

Once it's configured, dev/run should keep starting it. Until we run configure again to disable.

Author: Ronny Berndt
Author Date: 2024-01-03 15:57:26 UTC

Re-wrote snap installation guide lines for 3.3 (#4940) (#4941)

Updated installation guide lines for 3.3.3.

---------

Co-authored-by: Simon Klassen <6997477+sklassen@users.noreply.github.com>

Author: Ronny Berndt
Author Date: 2024-01-01 13:53:17 UTC

Add Single Source of Truth configuration system

Initial ideas.

Author: Russell Branca
Author Date: 2023-12-14 20:25:00 UTC

Error on unexpected rexi_call messages

Author: Russell Branca
Author Date: 2023-12-05 23:07:35 UTC

Switch to test_util:mock(config)

Author: janl
Author Date: 2023-11-28 15:09:00 UTC

feat(doc): What’s new in 3.4.0?

Author: Robert Newson
Author Date: 2023-11-28 16:56:00 UTC

DRAFT nouveau deb/rpm packaging

Author: janl
Author Date: 2023-11-27 14:17:13 UTC

doc: take out w + n params, folks get confused.

Author: Nick V.
Author Date: 2023-03-02 20:30:10 UTC

Restrict Deno to write to stdout only, read stdin and main.js only.

Author: Will Holley
Author Date: 2023-11-02 09:26:11 UTC

mango: fix $beginsWith range

In the intial implementation of $beginsWith, the range calculation
for json indexes mistakenly appends an integer with the size of
8 bits which gets maxed out at FF, rather than building a binary
with an extra 3 bytes at the end.

This commit fixes the `mango_idx_view:range/5` by correctly appending
the `U+FFFF` code point to create a utf-8 encoded binary. Additionally,
the Erlang `utf8` binary type ensures the result
is a valid utf8 string. If `Arg` is not a utf8 binary, this will
throw a badarg error.

We expect `Arg` strings to be a valid utf8 but, to be safe,
`mango_selector:norm_ops/1` is enhanced to verify
that any argument to `$beginsWith` is a utf8 string.

Author: Robert Newson
Author Date: 2023-10-04 21:36:35 UTC

store update seq of docs

Author: Robert Newson
Author Date: 2023-09-29 14:28:05 UTC

Support index version hint for greater consistency when paginating

Author: Robert Newson
Author Date: 2023-09-26 11:29:27 UTC

use text prefix in regex to speed up query

for selector;

{"selector":{"_id":{"$regex":"doc.+"}}}

before;

{
  "include_docs": true,
  "view_type": "map",
  "reduce": false,
  "partition": null,
  "start_key": [],
  "end_key": [
    "<MAX>"
  ],
  "direction": "fwd",
  "stable": false,
  "update": true,
  "conflicts": "undefined"
}

after;

{
  "include_docs": true,
  "view_type": "map",
  "reduce": false,
  "partition": null,
  "start_key": [
    "doc"
  ],
  "end_key": [
    "doc�",
    "<MAX>"
  ],
  "direction": "fwd",
  "stable": false,
  "update": true,
  "conflicts": "undefined"
}

closes: https://github.com/apache/couchdb/issues/4775

Author: Robert Newson
Author Date: 2023-08-30 23:17:30 UTC

EXPERIMENTAL: COS-backed directory

read caching will be essential. do this properly

Author: janl
Author Date: 2023-01-18 15:00:02 UTC

chore(readme): update mac dev install instructions

Author: Robert Newson
Author Date: 2023-08-22 12:21:40 UTC

A _find request can run for a very long time (on large databases when the
selector matches no index) and this continues even if the client disconnects.

We want to stop the fabric work when there is no client to receive the
result. fabric_streams already kills the workers if the coordinating process
dies but in this circumstance it does not.

this commit enhances (and renames) the existing cleanup process to be a watchdog. If
enabled, the watchdog needs to be kicked regularly (by whatever activity we
think indicates its worth continuing) or it will terminate the process it is
watching, and kill the worker processes also.

Currently only mango_httpd:handle_find_req enables the watchdog and it only
kicks the watchdog when it enqueues a row to be returned (i.e, only on selector
matches).

Author: janl
Author Date: 2023-08-17 10:52:11 UTC

chore: address various rerview notes by @rnewson

Author: Robert Newson
Author Date: 2023-08-07 08:29:50 UTC

Update README.md

fix markdown on enable nouveau line

Author: Robert Newson
Author Date: 2023-08-03 17:32:56 UTC

enhance smoosh to cleanup search indexes when ddocs change

Author: Robert Newson
Author Date: 2023-08-03 12:44:19 UTC

switch to fast_pbkdf2 for hashing

Author: Ronny Berndt
Author Date: 2023-07-27 16:59:39 UTC

Fix auth hashs test

Create a utility function for construction the AuthSession value

Author: Robert Newson
Author Date: 2023-07-21 20:51:52 UTC

upgrade to gradle 8.2.1

Author: Robert Newson
Author Date: 2023-07-21 20:30:14 UTC

run haproxy and nouveau without shell

Author: Ronny Berndt
Author Date: 2023-07-10 17:44:33 UTC

Align Makefile/Makefile.win

Author: Robert Newson
Author Date: 2023-06-26 20:29:08 UTC

support Websocket protocol for continuous response

This isn't full Websocket protocol as mochiweb doesn't have
the plumbing but it's still useful.

The endpoint will not return a pong if the client sends a ping
but if the heartbeat parameter is used the server will send pongs
at that right as a keep-alive.

If the server encounters an error it sends a Close message with
the reason before closing the socket.

Author: Robert Newson
Author Date: 2023-05-12 13:54:36 UTC

add nouveau and java tools to dev container

Author: Robert Newson
Author Date: 2023-05-11 05:33:46 UTC

fix dreyfus after 'Improve nouveau mango integration'

dreyfus/clouseau needs "string" type when indexing, so
make a separate add_default_field_nouveau function

Author: Russell Branca
Author Date: 2023-05-09 17:40:14 UTC

Match on #log_entry{}

Author: Robert Newson
Author Date: 2023-05-06 13:40:22 UTC

Merge remote-tracking branch 'origin/remove-content-md5-header' into remove-md5-entirely

Author: Robert Newson
Author Date: 2023-05-06 08:01:30 UTC

WIP extend syntax parser for geo

Author: Russell Branca
Author Date: 2023-05-04 21:07:50 UTC

Use proper Stats handle

Author: Nick V.
Author Date: 2023-05-02 16:34:42 UTC

CVE-2023-2626 details doc update

Author: Robert Newson
Author Date: 2023-05-01 21:22:09 UTC

fix ken_server:nouveau_updated

Author: Robert Newson
Author Date: 2023-04-22 10:30:25 UTC

doc linting

Author: Nick V.
Author Date: 2023-04-16 06:41:06 UTC

Release 3.2.3 version bump

Author: Robert Newson
Author Date: 2023-04-06 11:39:56 UTC

they're called hits

Author: Robert Newson
Author Date: 2023-04-06 07:08:03 UTC

trail ws

Author: Robert Newson
Author Date: 2023-04-04 19:54:30 UTC

track active search requests

Author: Robert Newson
Author Date: 2023-04-04 09:32:52 UTC

WIP send await time in response header - dreyfus

Author: Robert Newson
Author Date: 2023-03-22 13:14:30 UTC

capture original stack trace

Author: Will Holley
Author Date: 2023-03-20 18:30:04 UTC

Add units to metrics descriptions

Author: Jay Doane
Author Date: 2023-03-09 06:43:45 UTC

Refactor

Author: Ronny Berndt
Author Date: 2022-12-07 12:21:19 UTC

Merge branch 'spidermonkey_version_nif' of github.com:apache/couchdb into spidermonkey_version_nif

Author: Nick V.
Author Date: 2023-01-26 05:09:25 UTC

Fix flaky chttpd_changes_test

We made too strong of an assumption there that even in the case of Q=8 we'd
always have 2 pending changes, which is incorrect. The tests on Windows
apprently revealed an error where it returned 0. So let's relax the assumption
to assert that pending would be >= 0 and something less than 7.

Author: Ronny Berndt
Author Date: 2023-01-20 08:43:12 UTC

Merge branch 'main' into jenkins-add-windows-pipeline

Author: Gabor Pali
Author Date: 2023-01-17 09:54:19 UTC

docs(mango): match description of `$mod` with reality

The remainder argument for the `$mod` operator can be zero, while
its documentation suggests otherwise. It actually covers a very
realistic use case where divisibility is expressed.

Neither related restrictions could be identified in the sources
[1] nor MongoDB forbids this [2]. Tests also seem to exercise this
specific case [3]. Thanks @iilyak for checking on these.

[1] https://github.com/apache/couchdb/blob/adf17140e81d0b74f2b2ecdea48fc4f702832eaf/src/mango/src/mango_selector.erl#L512:L513
[2] https://www.mongodb.com/docs/manual/reference/operator/query/mod/
[3] https://github.com/apache/couchdb/blob/0059b8f90e58e10b199a4b768a06a762d12a30d3/src/mango/test/03-operator-test.py#L58

Author: janl
Author Date: 2023-01-18 15:01:58 UTC

ci(mac): re-enable mac CI for the full build

Author: janl
Author Date: 2023-01-09 16:14:46 UTC

ci: re-enable mac builder

Author: janl
Author Date: 2023-01-06 09:39:52 UTC

chore: protect 3.3.x and 3.2.x branches

Author: janl
Author Date: 2022-12-20 10:29:45 UTC

Revert "Reuse installed rebar and rebar3 for mix"

This reverts commit b5da5b66b8e2b2f0fe86ca142f1c45a63b5f9e0f.

Author: janl
Author Date: 2022-12-19 18:24:13 UTC

fix: debugging leftover from [checks notes] 2021

Author: Ronny Berndt
Author Date: 2022-12-19 16:09:38 UTC

Fix Powershell warnings

Author: janl
Author Date: 2022-11-12 07:25:53 UTC

fix: add function signature change to new open_docs_rev/3

Author: Nick V.
Author Date: 2022-12-11 05:48:01 UTC

try compiling for sm 102

Author: Russell Branca
Author Date: 2022-12-05 18:16:25 UTC

Add temporary marker

Author: Jay Doane
Author Date: 2022-10-16 00:23:51 UTC

Make test prometheus port explicit

These tests currently pass without this config parameter being set
because it gets that configuration from a test template. However, this
is fragile, and it's better practice to keep everything in one place.

This makes the configuration explicit.

Author: Robert Newson
Author Date: 2022-10-17 10:45:54 UTC

Clarify logging around elections and introduce depose/1 for testing

Author: Robert Newson
Author Date: 2022-10-10 14:43:51 UTC

Introduce update param

Author: Jay Doane
Author Date: 2022-09-01 20:38:44 UTC

Maybe return bad_content_type on _session POST

Currently, when POSTing to `/_session` with a Content-Type header
other than either `application/x-www-form-urlencoded` or
`application/json`, the error response can be surprising.

This changes the response to 415 `bad_content_type` when it's not one
of the above.

Author: Russell Branca
Author Date: 2022-08-08 22:55:07 UTC

Whitespace

Author: ILYA Khlopotov
Author Date: 2022-07-21 16:22:39 UTC

Fix undefined variable

Author: Russell Branca
Author Date: 2022-07-07 21:29:32 UTC

Fix cherry-pick merge bugs

Author: Robert Newson
Author Date: 2022-05-05 13:39:37 UTC

Database Encryption Support

CouchDB can optionally encrypt databases and views.

We use AES in Counter Mode, which ensures we can encrypt and decrypt
any section of the file without padding or alignment. The ciphertext
is the same length as the plaintext. This mode provides
confidentiality but not authentication.

Key management is configurable, a system administrator can write a
module implementing the couch_encryption_manager behaviour with any
implementation.

Author: Robert Newson
Author Date: 2022-05-14 19:28:05 UTC

Integrate raft algorithm (WIP)

couch_raft.erl is a complete implementation of the raft algorithm but
currently only manages an in-memory state machine and log.

Preliminary work is also here to add a new btree inside the `.couch`
files, which will be the real raft log. The intent is that log entries
can be removed from this log and applied to by_id and by_seq trees
atomically.

raft log is preserved over compaction in the same manner as local
docs, all entries are slurped into memory and written in one
pass. This should be fine as the log should stay short, committed
entries can be promptly removed. It's probably not fine for local
docs, though...

Anyway, it's progress and hopefully we're going somewhere cool.

Author: Nick V.
Author Date: 2022-06-07 20:30:44 UTC

Post main -> fdbmain move

Remove .asf.yaml and rename some CI references to point to fdbmain

Author: Nick V.
Author Date: 2022-06-07 20:37:52 UTC

Replace 3.x branch with a moved README marker file

Use main and fdbmain from now on

Author: Robert Newson
Author Date: 2022-05-09 10:26:47 UTC

use top bits of the IV as a nonce?

Author: janl
Author Date: 2022-04-09 08:28:41 UTC

fix: consistent use of new config sections makes tests less flakey

Author: Jay Doane
Author Date: 2022-03-26 19:43:43 UTC

Give smoosh eunit coverage

Author: Jay Doane
Author Date: 2022-03-23 19:09:09 UTC

Increase timeout for smoosh tests

As seen in recent CI:

    smoosh_tests:75: should_enqueue...*timed out*
    smoosh_tests:83: should_persist_queue...*timed out*

Author: Nick V.
Author Date: 2022-02-22 16:28:17 UTC

Try using rebar3

 * Use 3.15.2 version which is compatible with Erlang 20
 * Fauxton and docs converted to be used as rebar3 deps
 * Use get-deps only instead of update-deps

Author: Robert Newson
Author Date: 2022-02-28 19:04:28 UTC

Prevent users from modifying the reserved _replication_* fields

Only the replicator is permitted to do this. This PR enhances the
existing VDU to prohibit users from modifying the fields, in case the
user mistakenly believes that such edits are understood as
instructions to the replicator.

Author: Adam Kocoloski
Author Date: 2022-02-28 13:22:22 UTC

Very puzzling, this

Author: Francesco Pasa
Author Date: 2022-02-17 21:26:22 UTC

Autodetect spidermonkey version in ./configure

I have struggled with this point when building
couchDB, because it's written nowhere in the documentation
that you need to configure the version properly.

It seems that other people had the problem as well:
https://couchdb.slack.com/archives/C49LEE7NW/p1632249155263400

The updated configure script automatically finds
out the installed library version. It also introduces
a dependency on `sed`, `ldconfig` and `head`,
and remove the default to version 1.8.5 which is old
and not in most distributions anymore.

The argument was also removed from the CI scripts
because it does not exist anymore.

Author: Adam Kocoloski
Author Date: 2022-01-19 04:02:32 UTC

Apply new formatting from erlfmt

Author: Adam Kocoloski
Author Date: 2022-01-20 03:38:00 UTC

Merge branch 'main' into update-erlfmt

Author: Robert Newson
Author Date: 2021-12-22 16:32:11 UTC

Update test to match new behaviour

Author: Robert Newson
Author Date: 2021-12-06 13:55:34 UTC

Add SameSite setting when clearing session cookie

Author: janl
Author Date: 2021-11-26 10:19:34 UTC

fix memleak in couch_peruser, patch by @rnewson

Author: Russell Branca
Author Date: 2021-10-28 20:57:24 UTC

Fix segmented cache setup

Author: Tony Sun
Author Date: 2021-10-21 00:07:06 UTC

fix case_clause when HitId and DocId do not match when include_docs=true

Sometimes, a clouseau index may become out of sync with the db or
become corrupt under certain conditions (heap exhaustion, garbage collection).
This leads to certain HitIds that get returned which may not exist
in the current db. Before this fix, we assume the Hit's Id will
automatically match an Id return from:

fabric:all_docs(DbName, fun callback/2, [], [{keys, DocIds}, {include_docs, true}]).

When the document does not exist, {error, false} is returned for the row.
This fix accounts for this scenario and possible other scenarios
by returning an error to the user and also logging the error when the
Hit Id does not match what is returned from the _all_docs call.

Author: Russell Branca
Author Date: 2021-09-27 20:09:23 UTC

Revert "HACK: enable parallel readers"

This reverts commit 01305b518dda09330e924e29ab4c37b983033173.

Author: Nick V.
Author Date: 2021-09-24 21:15:43 UTC

Bump 3.1.2-RC versions

Author: janl
Author Date: 2021-09-07 14:26:05 UTC

feat(views): cache function compilation. Closes #3517.

Since supporting SpiderMonkey versions > 1.8.5 we compile design
doc functions of the form `function(args) { /* impl */ }` into a
form that is recognise by newer JS engines.

For reduce views, this means a transpilation happens on each
reduce call over the couchjs protocol, which is once for every
level in the b+tree plus one final rereduce across all shards.

down reduce view indexing/querying.

This patch adds caching to the compilation function. This is
implemented by way of producing a SHA-256 hash of all incoming
JS functions and caching them in a global object in the memory
of a `couchjs` process.

The cache is cleared when a `add_fun` message is received, which
happens before new map functions from a new ddoc are loaded into
`couchjs`. This ensures that only functions from a single view &
security context are ever loaded into the cache.

SHA-256 was chosen because it is producing collisions that are
also valid JS functions is unlikley.

This specific SHA-256 implementation was chosen because:
- it is favourably licensed (MIT)
- taken from the Deno (https://deno.land) project, (h/t Martin
  Sonnenholzer for the tip)so we can be reasonably assured this
  has been tested thoroughly.

Author: Russell Branca
Author Date: 2021-08-24 01:19:50 UTC

Fix some tests

Author: Russell Branca
Author Date: 2021-06-24 23:58:21 UTC

Alternative approach for mem3 shard creation

Author: Robert Newson
Author Date: 2021-05-11 22:20:20 UTC

Ensure no more than one response per request

Due to a retry loop in erlfb:transactional couchdb might try to send
multiple http responses for a single request which is clearly an
error.

This PR ensures the second attempt is prevented, closing the TCP
socket instead.

Author: Robert Newson
Author Date: 2021-05-05 21:56:49 UTC

hide key material behind zero-arity function

Author: Adam Kocoloski
Author Date: 2021-03-03 03:17:47 UTC

Require subscribers to wait until indexer finishes

This clause allowed a subscriber to start reading a view as soon as the
indexer made it past the sequence of interest. The trouble with that
approach is the resulting view is not directly related to any snapshot
of the underlying DB. Waiting until the indexer finishes allows us to
provide better semantics, where the view observes a consistent snapshot
of the database at some point in time >= the requested sequence.

In order to see this work through the view reader should explicitly set
the read version of FDB to match the commit version introduced by the
indexer, to avoid seeing partial results from a follow-on indexing job.

Author: Robert Newson
Author Date: 2021-02-15 09:17:34 UTC

Preserve max_dbs_open division during config change

And prevent max_dbs_open going below 1.

Author: jiangph
Author Date: 2021-01-26 04:57:18 UTC

introduce prometheus endpoint

Author: Robert Newson
Author Date: 2020-11-20 11:40:12 UTC

reuse GRV within an http request

Author: Robert Newson
Author Date: 2020-12-20 14:46:46 UTC

treat 408 as a retryable error condition (#3303)