couchdb:chewbranca-ioq-experiments-main-rebase

Last commit made on 2022-07-07
Get this branch:
git clone -b chewbranca-ioq-experiments-main-rebase https://git.launchpad.net/couchdb

Branch merges

Branch information

Name:
chewbranca-ioq-experiments-main-rebase
Repository:
lp:couchdb

Recent commits

a19d6ed... by Russell Branca <email address hidden>

Fix cherry-pick merge bugs

de07f48... by Russell Branca <email address hidden>

Bring in IOQ

897fbd6... by Russell Branca <email address hidden>

Add couch_file cache

Initial cherry-pick on top of main branch

29ac785... by Nick V.

Optimize couch_util:to_hex/1

When profiling [1] a cluster acting as a replication source, noticed a lot of
time spent in `couch_util:to_hex/1`. That function is used to emit revision ids
amongst other things. When processing a million documents with a 1000 revisions
each, it ends up in the hotpath, so to speak.

Remembering that in Erlang/OTP 24 there is a new `binary:encode_hex/1` [2]
function, decided to benchmark ours vs the OTP implementation. It turns ours is
slower [3] so let's try to use the OTP one.

One difference from the OTP's version is ours emits lower case hex letters,
while the OTP one emits upper case ones. That's why the lookup table had all
the "A"s replaced with "a"s, "B"s with "b"s, etc.

As a bonus, replaced a few calls to `couch_util:to_hex/1` wrapped in `?l2b/1`
or `list_to_binary/1` with just a single call to `couch_util:to_hex_bin/1`.

Existing `couch_util:to_hex/1` version, returning a list ,was left as is and
just calls `to_hex_bin/1` internally and converts the result to a list.

This altered the peak memory usage for the key tree stemming test so had to
alter the magic constants there a bit to avoid flakiness.

[1]
```
> ... eprof:analyze(total,[{sort, time}, {filter, [{time, 1000}, {calls, 100}]}]).

FUNCTION CALLS % TIME [uS / CALLS]
-------- ----- ------- ---- [----------]
...
couch_doc:revid_to_str/1 1165641 1.71 402860 [ 0.35]
couch_key_tree:get_key_leafs_simple/4 1304102 1.85 435700 [ 0.33]
erlang:list_to_integer/2 873172 2.00 471140 [ 0.54]
gen_server:loop/7 62209 2.11 496932 [ 7.99]
couch_key_tree:map_simple/3 1829235 2.36 554429 [ 0.30]
couch_util:nibble_to_hex/1 37334650 16.67 3917127 [ 0.10]
couch_util:to_hex/1 19834192 34.37 8077050 [ 0.41]
--------------------------------------------------------------- -------- ------- -------- [----------]
Total: 91072005 100.00% 23503072 [ 0.26]
```

[2] https://www.erlang.org/doc/man/binary.html#encode_hex-1

[3]

```
% ~/src/erlperf/erlperf 'hex:to_hex1(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).' 'hex:to_hex3(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>).'
Code || QPS Time Rel
hex:to_hex3(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>). 1 2746 Ki 364 ns 100%
hex:to_hex1(<<210,90,95,232,68,185,66,248,160,33,184,103,181,221,158,96>>). 1 1593 Ki 627 ns 58%
```

(`to_hex1/1` is the existing version and `to_hex3/1` is the OTP version).

e41465e... by Nick V.

Add an option to let custodian always use [cluster] n value

In cases when the application prevents creating dbs with non-default n values
(n=1, n=2) setting `[custodian] use_cluster_n_as_expected_n = true` can help
detect cases when the shard map itself was manpulated and by mistake ended up
with less than `[cluster] n` copies for some ranges.

By default the behavior is unchanged and n value will be deduced from the
number of copies indicated in the shard map documents.

eb2f8d9... by Nick Vatamaniuc <email address hidden>

Add Erlang 25 to PR CI pipeline and Ubuntu Jammy to full CI

ea5df65... by Nick V.

Bring back POWER full builds

2c351d6... by Nick V.

Update vm.args for Erlang 23+

 * file IO is using dirty IO schedulers and not the async IO thread pool

 * `-smp enable` does nothing, it's always enalbed

 * kernel polling is always enabled

```
% erl +SDio 16
Erlang/OTP 23 [erts-11.2.2.13] [source] [64-bit] [smp:12:12] [ds:12:12:16] [async-threads:1]

> erlang:system_info(kernel_poll).
true

> erlang:system_info(dirty_io_schedulers).
16
``

c605e04... by Nick V.

Fix Elixir 13 compatibility

Elixir 13 is the only one which seems to work with Erlang 25.

But we can't use Elixir 13 because of a few compatibility issues so have to fix
this. That involves upgrading credo. However, as soon as credo was upgraded it
noticed a bunch of issues which also had to be fixed. Some of those were:

 * Enum.map + join should be Enum.map_join
 * Invalid module name in teardown `%___MODULE__` -> `%__MODULE__`
 * `Mix.Config` is deprecated, use `Config`
 * `db_test.ex` looks like a test module to credo, so renamed it, otherwise it has to have an .exs extension
 * ibrowse had to be compiled so it could be properly analysed

Tested on macos (intel) with asdf installed elixir and erlang:
```
  25.0.1 / 1.13.4-otp-25
  24.3.4.1 / 1.13.4-otp-24
  23.3.4.14 / 1.10.4-otp-23
```

b424ad1... by Ronny Berndt

Merge pull request #4080 from apache/change-irc-url

Small url fixes