Merge ~racb/ubuntu/+source/memcached:upstream-bump-1.5.10 into ubuntu/+source/memcached:ubuntu/devel

Proposed by Robie Basak
Status: Merged
Merged at revision: 05743ddcb21ad054c5102c4cd3660bea10c6f17e
Proposed branch: ~racb/ubuntu/+source/memcached:upstream-bump-1.5.10
Merge into: ubuntu/+source/memcached:ubuntu/devel
Diff against target: 3924 lines (+1746/-336)
46 files modified
INSTALL (+370/-0)
Makefile.in (+2/-2)
README.md (+2/-2)
config.h.in (+7/-1)
configure (+80/-11)
configure.ac (+39/-3)
crawler.c (+8/-1)
crc32c.c (+72/-2)
debian/changelog (+8/-0)
debian/control (+2/-1)
doc/Makefile (+3/-3)
doc/memcached.1 (+2/-2)
doc/protocol.txt (+2/-0)
extstore.c (+97/-33)
extstore.h (+14/-2)
items.c (+8/-16)
linux_priv.c (+46/-4)
logger.c (+12/-3)
logger.h (+1/-0)
memcached.c (+155/-87)
memcached.h (+34/-2)
memcached.spec (+4/-4)
sasl_defs.c (+6/-2)
scripts/memcached-automove (+1/-1)
scripts/memcached-automove-extstore (+1/-2)
scripts/memcached-tool (+72/-58)
scripts/memcached-tool.1 (+5/-2)
slabs.c (+76/-3)
storage.c (+220/-28)
storage.h (+7/-1)
t/binary-extstore.t (+10/-2)
t/binary-sasl.t (+20/-7)
t/chunked-extstore.t (+35/-20)
t/error-extstore.t (+130/-0)
t/extstore-buckets.t (+1/-1)
t/extstore-jbod.t (+69/-0)
t/extstore.t (+37/-9)
t/flush-all.t (+1/-1)
t/issue_67.t (+19/-6)
t/lib/MemcachedTest.pm (+21/-1)
t/lru-crawler.t (+13/-1)
t/lru-maintainer.t (+12/-0)
t/misbehave.t (+10/-2)
thread.c (+5/-3)
timedrun.c (+6/-6)
version.m4 (+1/-1)
Reviewer Review Type Date Requested Status
Christian Ehrhardt  (community) Approve
Canonical Server Pending
Review via email: mp+352972@code.launchpad.net

Description of the change

This includes fixes for some issues which were causing build time test failures, preventing memcached from migrating to the release pocket. Thanks to upstream working closely with us, all the fixes are released upstream, so we just pull in the latest and it works.

After Debian takes this we should be able to sync.

To post a comment you must log in.
Revision history for this message
Robie Basak (racb) wrote :

A PPA build should be available in ppa:racb/experimental shortly.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
a few things inline - mostly suggestions for the changelog.

The bump itself seems correct to me - content change wise.
The md5sum of the upstream download matches:
$ md5sum memcached-1.5.10.tar.gz memcached_1.5.10.orig.tar.gz
8462616b554183a75845b03c56837cca memcached-1.5.10.tar.gz
8462616b554183a75845b03c56837cca memcached_1.5.10.orig.tar.gz

From:
http://www.memcached.org/files/memcached-1.5.10.tar.gz
https://launchpad.net/~racb/+archive/ubuntu/experimental/+sourcefiles/memcached/1.5.10-0ubuntu1~ppa1/memcached_1.5.10.orig.tar.gz

No other functional changes required.

The inline comments are all style-questions which you can decide either way.
I'll do some testing before a full +1

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I copied from your ppa that was identical to the proposed changes.
Testing from that ...

qa tests:
- I ran against the old version in cosmic 1.5.6-0ubuntu1 to know which might be known issues and if the test is repeatable
- of these memcdump had a fail, but all others worked
- after upgrading to the ppa version (upgrade worked) I reran the tests
- all tests that worked still work (no change on memcdump)

Running test: './test-memcached.py' distro: 'Ubuntu 18.10' kernel: '4.17.0-6.7 (Ubuntu 4.17.0-6.7-generic 4.17.9)' arch: 'amd64' uid: 0/0 SUDO_USER: 'ubuntu')
Skipping private tests
test_add (__main__.MemcachedTest)
Test adding data ... ok
test_cve_2011_4971 (__main__.MemcachedTest)
Test CVE-2011-4971 ... ok
test_default_tcp (__main__.MemcachedTest)
Test that memcached is listening on tcp port by default ... ok
test_default_udp (__main__.MemcachedTest)
Test that memcached is listening on udp port by default ... ok
test_memccapable (__main__.MemcachedTest)
Test memccapable ... ok
test_memccat (__main__.MemcachedTest)
Test memccat ... ok
test_memcdump (__main__.MemcachedTest)
Test memcdump ... FAIL
test_memcrm (__main__.MemcachedTest)
Test memcrm ... ok
test_memcstat (__main__.MemcachedTest)
Test memcstat ... ok
test_sample_data (__main__.MemcachedTest)
Test sample data ... ok
test_set (__main__.MemcachedTest)
Test setting data ... ok

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Autopkgtests are enqueued, will check results after lunch.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Autopkgtests are also good:
flask-limiter/1.0.1-1
gnocchi/4.2.4-0ubuntu3
memcached/1.5.10-0ubuntu1~ppa1
pgmemcache/2.3.0-4
python-limits/1.3-1

That said, code good, changelog ok (up to you to adapt my suggestions) and tests good - +1 on the MP

review: Approve
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1diff --git a/INSTALL b/INSTALL
2new file mode 100644
3index 0000000..2099840
4--- /dev/null
5+++ b/INSTALL
6@@ -0,0 +1,370 @@
7+Installation Instructions
8+*************************
9+
10+Copyright (C) 1994-1996, 1999-2002, 2004-2013 Free Software Foundation,
11+Inc.
12+
13+ Copying and distribution of this file, with or without modification,
14+are permitted in any medium without royalty provided the copyright
15+notice and this notice are preserved. This file is offered as-is,
16+without warranty of any kind.
17+
18+Basic Installation
19+==================
20+
21+ Briefly, the shell command `./configure && make && make install'
22+should configure, build, and install this package. The following
23+more-detailed instructions are generic; see the `README' file for
24+instructions specific to this package. Some packages provide this
25+`INSTALL' file but do not implement all of the features documented
26+below. The lack of an optional feature in a given package is not
27+necessarily a bug. More recommendations for GNU packages can be found
28+in *note Makefile Conventions: (standards)Makefile Conventions.
29+
30+ The `configure' shell script attempts to guess correct values for
31+various system-dependent variables used during compilation. It uses
32+those values to create a `Makefile' in each directory of the package.
33+It may also create one or more `.h' files containing system-dependent
34+definitions. Finally, it creates a shell script `config.status' that
35+you can run in the future to recreate the current configuration, and a
36+file `config.log' containing compiler output (useful mainly for
37+debugging `configure').
38+
39+ It can also use an optional file (typically called `config.cache'
40+and enabled with `--cache-file=config.cache' or simply `-C') that saves
41+the results of its tests to speed up reconfiguring. Caching is
42+disabled by default to prevent problems with accidental use of stale
43+cache files.
44+
45+ If you need to do unusual things to compile the package, please try
46+to figure out how `configure' could check whether to do them, and mail
47+diffs or instructions to the address given in the `README' so they can
48+be considered for the next release. If you are using the cache, and at
49+some point `config.cache' contains results you don't want to keep, you
50+may remove or edit it.
51+
52+ The file `configure.ac' (or `configure.in') is used to create
53+`configure' by a program called `autoconf'. You need `configure.ac' if
54+you want to change it or regenerate `configure' using a newer version
55+of `autoconf'.
56+
57+ The simplest way to compile this package is:
58+
59+ 1. `cd' to the directory containing the package's source code and type
60+ `./configure' to configure the package for your system.
61+
62+ Running `configure' might take a while. While running, it prints
63+ some messages telling which features it is checking for.
64+
65+ 2. Type `make' to compile the package.
66+
67+ 3. Optionally, type `make check' to run any self-tests that come with
68+ the package, generally using the just-built uninstalled binaries.
69+
70+ 4. Type `make install' to install the programs and any data files and
71+ documentation. When installing into a prefix owned by root, it is
72+ recommended that the package be configured and built as a regular
73+ user, and only the `make install' phase executed with root
74+ privileges.
75+
76+ 5. Optionally, type `make installcheck' to repeat any self-tests, but
77+ this time using the binaries in their final installed location.
78+ This target does not install anything. Running this target as a
79+ regular user, particularly if the prior `make install' required
80+ root privileges, verifies that the installation completed
81+ correctly.
82+
83+ 6. You can remove the program binaries and object files from the
84+ source code directory by typing `make clean'. To also remove the
85+ files that `configure' created (so you can compile the package for
86+ a different kind of computer), type `make distclean'. There is
87+ also a `make maintainer-clean' target, but that is intended mainly
88+ for the package's developers. If you use it, you may have to get
89+ all sorts of other programs in order to regenerate files that came
90+ with the distribution.
91+
92+ 7. Often, you can also type `make uninstall' to remove the installed
93+ files again. In practice, not all packages have tested that
94+ uninstallation works correctly, even though it is required by the
95+ GNU Coding Standards.
96+
97+ 8. Some packages, particularly those that use Automake, provide `make
98+ distcheck', which can by used by developers to test that all other
99+ targets like `make install' and `make uninstall' work correctly.
100+ This target is generally not run by end users.
101+
102+Compilers and Options
103+=====================
104+
105+ Some systems require unusual options for compilation or linking that
106+the `configure' script does not know about. Run `./configure --help'
107+for details on some of the pertinent environment variables.
108+
109+ You can give `configure' initial values for configuration parameters
110+by setting variables in the command line or in the environment. Here
111+is an example:
112+
113+ ./configure CC=c99 CFLAGS=-g LIBS=-lposix
114+
115+ *Note Defining Variables::, for more details.
116+
117+Compiling For Multiple Architectures
118+====================================
119+
120+ You can compile the package for more than one kind of computer at the
121+same time, by placing the object files for each architecture in their
122+own directory. To do this, you can use GNU `make'. `cd' to the
123+directory where you want the object files and executables to go and run
124+the `configure' script. `configure' automatically checks for the
125+source code in the directory that `configure' is in and in `..'. This
126+is known as a "VPATH" build.
127+
128+ With a non-GNU `make', it is safer to compile the package for one
129+architecture at a time in the source code directory. After you have
130+installed the package for one architecture, use `make distclean' before
131+reconfiguring for another architecture.
132+
133+ On MacOS X 10.5 and later systems, you can create libraries and
134+executables that work on multiple system types--known as "fat" or
135+"universal" binaries--by specifying multiple `-arch' options to the
136+compiler but only a single `-arch' option to the preprocessor. Like
137+this:
138+
139+ ./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
140+ CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
141+ CPP="gcc -E" CXXCPP="g++ -E"
142+
143+ This is not guaranteed to produce working output in all cases, you
144+may have to build one architecture at a time and combine the results
145+using the `lipo' tool if you have problems.
146+
147+Installation Names
148+==================
149+
150+ By default, `make install' installs the package's commands under
151+`/usr/local/bin', include files under `/usr/local/include', etc. You
152+can specify an installation prefix other than `/usr/local' by giving
153+`configure' the option `--prefix=PREFIX', where PREFIX must be an
154+absolute file name.
155+
156+ You can specify separate installation prefixes for
157+architecture-specific files and architecture-independent files. If you
158+pass the option `--exec-prefix=PREFIX' to `configure', the package uses
159+PREFIX as the prefix for installing programs and libraries.
160+Documentation and other data files still use the regular prefix.
161+
162+ In addition, if you use an unusual directory layout you can give
163+options like `--bindir=DIR' to specify different values for particular
164+kinds of files. Run `configure --help' for a list of the directories
165+you can set and what kinds of files go in them. In general, the
166+default for these options is expressed in terms of `${prefix}', so that
167+specifying just `--prefix' will affect all of the other directory
168+specifications that were not explicitly provided.
169+
170+ The most portable way to affect installation locations is to pass the
171+correct locations to `configure'; however, many packages provide one or
172+both of the following shortcuts of passing variable assignments to the
173+`make install' command line to change installation locations without
174+having to reconfigure or recompile.
175+
176+ The first method involves providing an override variable for each
177+affected directory. For example, `make install
178+prefix=/alternate/directory' will choose an alternate location for all
179+directory configuration variables that were expressed in terms of
180+`${prefix}'. Any directories that were specified during `configure',
181+but not in terms of `${prefix}', must each be overridden at install
182+time for the entire installation to be relocated. The approach of
183+makefile variable overrides for each directory variable is required by
184+the GNU Coding Standards, and ideally causes no recompilation.
185+However, some platforms have known limitations with the semantics of
186+shared libraries that end up requiring recompilation when using this
187+method, particularly noticeable in packages that use GNU Libtool.
188+
189+ The second method involves providing the `DESTDIR' variable. For
190+example, `make install DESTDIR=/alternate/directory' will prepend
191+`/alternate/directory' before all installation names. The approach of
192+`DESTDIR' overrides is not required by the GNU Coding Standards, and
193+does not work on platforms that have drive letters. On the other hand,
194+it does better at avoiding recompilation issues, and works well even
195+when some directory options were not specified in terms of `${prefix}'
196+at `configure' time.
197+
198+Optional Features
199+=================
200+
201+ If the package supports it, you can cause programs to be installed
202+with an extra prefix or suffix on their names by giving `configure' the
203+option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'.
204+
205+ Some packages pay attention to `--enable-FEATURE' options to
206+`configure', where FEATURE indicates an optional part of the package.
207+They may also pay attention to `--with-PACKAGE' options, where PACKAGE
208+is something like `gnu-as' or `x' (for the X Window System). The
209+`README' should mention any `--enable-' and `--with-' options that the
210+package recognizes.
211+
212+ For packages that use the X Window System, `configure' can usually
213+find the X include and library files automatically, but if it doesn't,
214+you can use the `configure' options `--x-includes=DIR' and
215+`--x-libraries=DIR' to specify their locations.
216+
217+ Some packages offer the ability to configure how verbose the
218+execution of `make' will be. For these packages, running `./configure
219+--enable-silent-rules' sets the default to minimal output, which can be
220+overridden with `make V=1'; while running `./configure
221+--disable-silent-rules' sets the default to verbose, which can be
222+overridden with `make V=0'.
223+
224+Particular systems
225+==================
226+
227+ On HP-UX, the default C compiler is not ANSI C compatible. If GNU
228+CC is not installed, it is recommended to use the following options in
229+order to use an ANSI C compiler:
230+
231+ ./configure CC="cc -Ae -D_XOPEN_SOURCE=500"
232+
233+and if that doesn't work, install pre-built binaries of GCC for HP-UX.
234+
235+ HP-UX `make' updates targets which have the same time stamps as
236+their prerequisites, which makes it generally unusable when shipped
237+generated files such as `configure' are involved. Use GNU `make'
238+instead.
239+
240+ On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot
241+parse its `<wchar.h>' header file. The option `-nodtk' can be used as
242+a workaround. If GNU CC is not installed, it is therefore recommended
243+to try
244+
245+ ./configure CC="cc"
246+
247+and if that doesn't work, try
248+
249+ ./configure CC="cc -nodtk"
250+
251+ On Solaris, don't put `/usr/ucb' early in your `PATH'. This
252+directory contains several dysfunctional programs; working variants of
253+these programs are available in `/usr/bin'. So, if you need `/usr/ucb'
254+in your `PATH', put it _after_ `/usr/bin'.
255+
256+ On Haiku, software installed for all users goes in `/boot/common',
257+not `/usr/local'. It is recommended to use the following options:
258+
259+ ./configure --prefix=/boot/common
260+
261+Specifying the System Type
262+==========================
263+
264+ There may be some features `configure' cannot figure out
265+automatically, but needs to determine by the type of machine the package
266+will run on. Usually, assuming the package is built to be run on the
267+_same_ architectures, `configure' can figure that out, but if it prints
268+a message saying it cannot guess the machine type, give it the
269+`--build=TYPE' option. TYPE can either be a short name for the system
270+type, such as `sun4', or a canonical name which has the form:
271+
272+ CPU-COMPANY-SYSTEM
273+
274+where SYSTEM can have one of these forms:
275+
276+ OS
277+ KERNEL-OS
278+
279+ See the file `config.sub' for the possible values of each field. If
280+`config.sub' isn't included in this package, then this package doesn't
281+need to know the machine type.
282+
283+ If you are _building_ compiler tools for cross-compiling, you should
284+use the option `--target=TYPE' to select the type of system they will
285+produce code for.
286+
287+ If you want to _use_ a cross compiler, that generates code for a
288+platform different from the build platform, you should specify the
289+"host" platform (i.e., that on which the generated programs will
290+eventually be run) with `--host=TYPE'.
291+
292+Sharing Defaults
293+================
294+
295+ If you want to set default values for `configure' scripts to share,
296+you can create a site shell script called `config.site' that gives
297+default values for variables like `CC', `cache_file', and `prefix'.
298+`configure' looks for `PREFIX/share/config.site' if it exists, then
299+`PREFIX/etc/config.site' if it exists. Or, you can set the
300+`CONFIG_SITE' environment variable to the location of the site script.
301+A warning: not all `configure' scripts look for a site script.
302+
303+Defining Variables
304+==================
305+
306+ Variables not defined in a site shell script can be set in the
307+environment passed to `configure'. However, some packages may run
308+configure again during the build, and the customized values of these
309+variables may be lost. In order to avoid this problem, you should set
310+them in the `configure' command line, using `VAR=value'. For example:
311+
312+ ./configure CC=/usr/local2/bin/gcc
313+
314+causes the specified `gcc' to be used as the C compiler (unless it is
315+overridden in the site shell script).
316+
317+Unfortunately, this technique does not work for `CONFIG_SHELL' due to
318+an Autoconf limitation. Until the limitation is lifted, you can use
319+this workaround:
320+
321+ CONFIG_SHELL=/bin/bash ./configure CONFIG_SHELL=/bin/bash
322+
323+`configure' Invocation
324+======================
325+
326+ `configure' recognizes the following options to control how it
327+operates.
328+
329+`--help'
330+`-h'
331+ Print a summary of all of the options to `configure', and exit.
332+
333+`--help=short'
334+`--help=recursive'
335+ Print a summary of the options unique to this package's
336+ `configure', and exit. The `short' variant lists options used
337+ only in the top level, while the `recursive' variant lists options
338+ also present in any nested packages.
339+
340+`--version'
341+`-V'
342+ Print the version of Autoconf used to generate the `configure'
343+ script, and exit.
344+
345+`--cache-file=FILE'
346+ Enable the cache: use and save the results of the tests in FILE,
347+ traditionally `config.cache'. FILE defaults to `/dev/null' to
348+ disable caching.
349+
350+`--config-cache'
351+`-C'
352+ Alias for `--cache-file=config.cache'.
353+
354+`--quiet'
355+`--silent'
356+`-q'
357+ Do not print messages saying which checks are being made. To
358+ suppress all normal output, redirect it to `/dev/null' (any error
359+ messages will still be shown).
360+
361+`--srcdir=DIR'
362+ Look for the package's source code in directory DIR. Usually
363+ `configure' can determine that directory automatically.
364+
365+`--prefix=DIR'
366+ Use DIR as the installation prefix. *note Installation Names::
367+ for more details, including other options available for fine-tuning
368+ the installation locations.
369+
370+`--no-create'
371+`-n'
372+ Run the configure checks, but stop before creating any output
373+ files.
374+
375+`configure' also accepts some other, not widely useful, options. Run
376+`configure --help' for more details.
377diff --git a/Makefile.in b/Makefile.in
378index 966b4d7..c125ace 100644
379--- a/Makefile.in
380+++ b/Makefile.in
381@@ -327,8 +327,8 @@ CTAGS = ctags
382 CSCOPE = cscope
383 DIST_SUBDIRS = $(SUBDIRS)
384 am__DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/config.h.in AUTHORS \
385- COPYING ChangeLog NEWS compile config.guess config.sub depcomp \
386- install-sh missing
387+ COPYING ChangeLog INSTALL NEWS compile config.guess config.sub \
388+ depcomp install-sh missing
389 DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
390 distdir = $(PACKAGE)-$(VERSION)
391 top_distdir = $(distdir)
392diff --git a/README.md b/README.md
393index 935e4cd..e330fbe 100644
394--- a/README.md
395+++ b/README.md
396@@ -18,8 +18,8 @@ list to ask questions, github issues aren't seen by everyone!
397 ## Dependencies
398
399 * libevent, http://www.monkey.org/~provos/libevent/ (libevent-dev)
400-* libseccomp, (optional, linux) - enables process restrictions for better
401- security.
402+* libseccomp, (optional, experimental, linux) - enables process restrictions for
403+ better security. Tested only on x86_64 architectures.
404
405 ## Environment
406
407diff --git a/config.h.in b/config.h.in
408index 9638dfa..3f3d42d 100644
409--- a/config.h.in
410+++ b/config.h.in
411@@ -1,5 +1,8 @@
412 /* config.h.in. Generated from configure.ac by autoheader. */
413
414+/* Set to nonzero if you want to enable ARMv8 crc32 */
415+#undef ARM_CRC32
416+
417 /* Set to nonzero if you want to include DTRACE */
418 #undef ENABLE_DTRACE
419
420@@ -15,7 +18,7 @@
421 /* machine is littleendian */
422 #undef ENDIAN_LITTLE
423
424-/* Set to nonzero if you want to enable extstorextstore */
425+/* Set to nonzero if you want to enable extstore */
426 #undef EXTSTORE
427
428 /* Define to 1 if support accept4 */
429@@ -66,6 +69,9 @@
430 /* Set to nonzero if your SASL implementation supports SASL_CB_GETCONF */
431 #undef HAVE_SASL_CB_GETCONF
432
433+/* Set to nonzero if your SASL implementation supports SASL_CB_GETCONFPATH */
434+#undef HAVE_SASL_CB_GETCONFPATH
435+
436 /* Define to 1 if you have the <sasl/sasl.h> header file. */
437 #undef HAVE_SASL_SASL_H
438
439diff --git a/configure b/configure
440index e479e60..9ccb4dd 100755
441--- a/configure
442+++ b/configure
443@@ -1,6 +1,6 @@
444 #! /bin/sh
445 # Guess values for system-dependent variables and create Makefiles.
446-# Generated by GNU Autoconf 2.69 for memcached 1.5.6.
447+# Generated by GNU Autoconf 2.69 for memcached 1.5.10.
448 #
449 # Report bugs to <memcached@googlegroups.com>.
450 #
451@@ -580,8 +580,8 @@ MAKEFLAGS=
452 # Identity of this package.
453 PACKAGE_NAME='memcached'
454 PACKAGE_TARNAME='memcached'
455-PACKAGE_VERSION='1.5.6'
456-PACKAGE_STRING='memcached 1.5.6'
457+PACKAGE_VERSION='1.5.10'
458+PACKAGE_STRING='memcached 1.5.10'
459 PACKAGE_BUGREPORT='memcached@googlegroups.com'
460 PACKAGE_URL=''
461
462@@ -643,6 +643,8 @@ PROFILER
463 PROFILER_LDFLAGS
464 ENABLE_SASL
465 DTRACEFLAGS
466+ENABLE_ARM_CRC32_FALSE
467+ENABLE_ARM_CRC32_TRUE
468 ENABLE_EXTSTORE_FALSE
469 ENABLE_EXTSTORE_TRUE
470 ENABLE_SASL_FALSE
471@@ -751,6 +753,7 @@ ac_user_opts='
472 enable_option_checking
473 enable_silent_rules
474 enable_dependency_tracking
475+enable_arm_crc32
476 enable_extstore
477 enable_seccomp
478 enable_sasl
479@@ -1320,7 +1323,7 @@ if test "$ac_init_help" = "long"; then
480 # Omit some internal or obsolete options to make the list less imposing.
481 # This message is too long to be a string in the A/UX 3.1 sh.
482 cat <<_ACEOF
483-\`configure' configures memcached 1.5.6 to adapt to many kinds of systems.
484+\`configure' configures memcached 1.5.10 to adapt to many kinds of systems.
485
486 Usage: $0 [OPTION]... [VAR=VALUE]...
487
488@@ -1391,7 +1394,7 @@ fi
489
490 if test -n "$ac_init_help"; then
491 case $ac_init_help in
492- short | recursive ) echo "Configuration of memcached 1.5.6:";;
493+ short | recursive ) echo "Configuration of memcached 1.5.10:";;
494 esac
495 cat <<\_ACEOF
496
497@@ -1405,8 +1408,9 @@ Optional Features:
498 do not reject slow dependency extractors
499 --disable-dependency-tracking
500 speeds up one-time build
501+ --enable-arm-crc32 Enable ARMv8 CRC32 instructions
502 --enable-extstore Enable external storage EXPERIMENTAL
503- --enable-seccomp Enable seccomp restrictions
504+ --enable-seccomp Enable seccomp restrictions EXPERIMENTAL
505 --enable-sasl Enable SASL authentication
506 --enable-sasl-pwdb Enable plaintext password db
507 --enable-dtrace Enable dtrace probes
508@@ -1495,7 +1499,7 @@ fi
509 test -n "$ac_init_help" && exit $ac_status
510 if $ac_init_version; then
511 cat <<\_ACEOF
512-memcached configure 1.5.6
513+memcached configure 1.5.10
514 generated by GNU Autoconf 2.69
515
516 Copyright (C) 2012 Free Software Foundation, Inc.
517@@ -1964,7 +1968,7 @@ cat >config.log <<_ACEOF
518 This file contains any messages produced by compilers while
519 running configure, to aid debugging if configure makes a mistake.
520
521-It was created by memcached $as_me 1.5.6, which was
522+It was created by memcached $as_me 1.5.10, which was
523 generated by GNU Autoconf 2.69. Invocation command line was
524
525 $ $0 $@
526@@ -2899,7 +2903,7 @@ fi
527
528 # Define the identity of the package.
529 PACKAGE='memcached'
530- VERSION='1.5.6'
531+ VERSION='1.5.10'
532
533
534 cat >>confdefs.h <<_ACEOF
535@@ -4701,6 +4705,12 @@ fi
536
537
538
539+# Check whether --enable-arm_crc32 was given.
540+if test "${enable_arm_crc32+set}" = set; then :
541+ enableval=$enable_arm_crc32;
542+fi
543+
544+
545 # Check whether --enable-extstore was given.
546 if test "${enable_extstore+set}" = set; then :
547 enableval=$enable_extstore;
548@@ -4733,6 +4743,8 @@ fi
549
550
551
552+
553+
554 for ac_header in sasl/sasl.h
555 do :
556 ac_fn_c_check_header_mongrel "$LINENO" "sasl/sasl.h" "ac_cv_header_sasl_sasl_h" "$ac_includes_default"
557@@ -4784,6 +4796,43 @@ $as_echo "#define HAVE_SASL_CB_GETCONF 1" >>confdefs.h
558 fi
559
560
561+ { $as_echo "$as_me:${as_lineno-$LINENO}: checking for SASL_CB_GETCONFPATH" >&5
562+$as_echo_n "checking for SASL_CB_GETCONFPATH... " >&6; }
563+if ${ac_cv_c_sasl_cb_getconfpath+:} false; then :
564+ $as_echo_n "(cached) " >&6
565+else
566+ cat confdefs.h - <<_ACEOF >conftest.$ac_ext
567+/* end confdefs.h. */
568+
569+#include <sasl/sasl.h>
570+
571+int
572+main ()
573+{
574+
575+unsigned long val = SASL_CB_GETCONFPATH;
576+
577+ ;
578+ return 0;
579+}
580+_ACEOF
581+if ac_fn_c_try_compile "$LINENO"; then :
582+ ac_cv_c_sasl_cb_getconfpath=yes
583+else
584+ ac_cv_c_sasl_cb_getconfpath=no
585+fi
586+rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
587+
588+fi
589+{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_c_sasl_cb_getconfpath" >&5
590+$as_echo "$ac_cv_c_sasl_cb_getconfpath" >&6; }
591+ if test "$ac_cv_c_sasl_cb_getconfpath" = "yes"; then :
592+
593+$as_echo "#define HAVE_SASL_CB_GETCONFPATH 1" >>confdefs.h
594+
595+fi
596+
597+
598 $as_echo "#define ENABLE_SASL 1" >>confdefs.h
599
600 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sasl_server_init" >&5
601@@ -4929,6 +4978,12 @@ $as_echo "#define EXTSTORE 1" >>confdefs.h
602
603 fi
604
605+if test "x$enable_arm_crc32" = "xyes"; then
606+
607+$as_echo "#define ARM_CRC32 1" >>confdefs.h
608+
609+fi
610+
611 if test "$build_dtrace" = "yes"; then
612 BUILD_DTRACE_TRUE=
613 BUILD_DTRACE_FALSE='#'
614@@ -4961,6 +5016,14 @@ else
615 ENABLE_EXTSTORE_FALSE=
616 fi
617
618+ if test "$enable_arm_crc32" = "yes"; then
619+ ENABLE_ARM_CRC32_TRUE=
620+ ENABLE_ARM_CRC32_FALSE='#'
621+else
622+ ENABLE_ARM_CRC32_TRUE='#'
623+ ENABLE_ARM_CRC32_FALSE=
624+fi
625+
626
627
628
629@@ -6203,6 +6266,7 @@ else
630
631 #include <stdlib.h>
632 #include <inttypes.h>
633+#pragma GCC optimize ("O0")
634
635 int
636 main ()
637@@ -6292,6 +6356,7 @@ have_gcc_64atomics=no
638 $as_echo_n "checking for GCC 64bit atomics... " >&6; }
639 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
640 /* end confdefs.h. */
641+#include <inttypes.h>
642
643 int
644 main ()
645@@ -6739,6 +6804,10 @@ if test -z "${ENABLE_EXTSTORE_TRUE}" && test -z "${ENABLE_EXTSTORE_FALSE}"; then
646 as_fn_error $? "conditional \"ENABLE_EXTSTORE\" was never defined.
647 Usually this means the macro was only invoked conditionally." "$LINENO" 5
648 fi
649+if test -z "${ENABLE_ARM_CRC32_TRUE}" && test -z "${ENABLE_ARM_CRC32_FALSE}"; then
650+ as_fn_error $? "conditional \"ENABLE_ARM_CRC32\" was never defined.
651+Usually this means the macro was only invoked conditionally." "$LINENO" 5
652+fi
653 if test -z "${BUILD_SOLARIS_PRIVS_TRUE}" && test -z "${BUILD_SOLARIS_PRIVS_FALSE}"; then
654 as_fn_error $? "conditional \"BUILD_SOLARIS_PRIVS\" was never defined.
655 Usually this means the macro was only invoked conditionally." "$LINENO" 5
656@@ -7156,7 +7225,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
657 # report actual input values of CONFIG_FILES etc. instead of their
658 # values after options handling.
659 ac_log="
660-This file was extended by memcached $as_me 1.5.6, which was
661+This file was extended by memcached $as_me 1.5.10, which was
662 generated by GNU Autoconf 2.69. Invocation command line was
663
664 CONFIG_FILES = $CONFIG_FILES
665@@ -7222,7 +7291,7 @@ _ACEOF
666 cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
667 ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
668 ac_cs_version="\\
669-memcached config.status 1.5.6
670+memcached config.status 1.5.10
671 configured by $0, generated by GNU Autoconf 2.69,
672 with options \\"\$ac_cs_config\\"
673
674diff --git a/configure.ac b/configure.ac
675index 9f05cfa..abcb8ae 100644
676--- a/configure.ac
677+++ b/configure.ac
678@@ -82,11 +82,14 @@ fi
679 AM_PROG_CC_C_O
680 AC_PROG_INSTALL
681
682+AC_ARG_ENABLE(arm_crc32,
683+ [AS_HELP_STRING([--enable-arm-crc32], [Enable ARMv8 CRC32 instructions])])
684+
685 AC_ARG_ENABLE(extstore,
686 [AS_HELP_STRING([--enable-extstore], [Enable external storage EXPERIMENTAL ])])
687
688 AC_ARG_ENABLE(seccomp,
689- [AS_HELP_STRING([--enable-seccomp],[Enable seccomp restrictions])])
690+ [AS_HELP_STRING([--enable-seccomp],[Enable seccomp restrictions EXPERIMENTAL])])
691
692 AC_ARG_ENABLE(sasl,
693 [AS_HELP_STRING([--enable-sasl],[Enable SASL authentication])])
694@@ -122,9 +125,33 @@ unsigned long val = SASL_CB_GETCONF;
695 [Set to nonzero if your SASL implementation supports SASL_CB_GETCONF])])
696 ])
697
698+dnl **********************************************************************
699+dnl DETECT_SASL_CB_GETCONFPATH
700+dnl
701+dnl check if we can use SASL_CB_GETCONFPATH
702+dnl **********************************************************************
703+AC_DEFUN([AC_C_DETECT_SASL_CB_GETCONFPATH],
704+[
705+ AC_CACHE_CHECK([for SASL_CB_GETCONFPATH],
706+ [ac_cv_c_sasl_cb_getconfpath],
707+ [AC_TRY_COMPILE(
708+ [
709+#include <sasl/sasl.h>
710+ ], [
711+unsigned long val = SASL_CB_GETCONFPATH;
712+ ],
713+ [ ac_cv_c_sasl_cb_getconfpath=yes ],
714+ [ ac_cv_c_sasl_cb_getconfpath=no ])
715+ ])
716+ AS_IF([test "$ac_cv_c_sasl_cb_getconfpath" = "yes"],
717+ [AC_DEFINE([HAVE_SASL_CB_GETCONFPATH], 1,
718+ [Set to nonzero if your SASL implementation supports SASL_CB_GETCONFPATH])])
719+])
720+
721 AC_CHECK_HEADERS([sasl/sasl.h])
722 if test "x$enable_sasl" = "xyes"; then
723 AC_C_DETECT_SASL_CB_GETCONF
724+ AC_C_DETECT_SASL_CB_GETCONFPATH
725 AC_DEFINE([ENABLE_SASL],1,[Set to nonzero if you want to include SASL])
726 AC_SEARCH_LIBS([sasl_server_init], [sasl2 sasl], [],
727 [
728@@ -160,13 +187,18 @@ if test "x$enable_dtrace" = "xyes"; then
729 fi
730
731 if test "x$enable_extstore" = "xyes"; then
732- AC_DEFINE([EXTSTORE],1,[Set to nonzero if you want to enable extstorextstore])
733+ AC_DEFINE([EXTSTORE],1,[Set to nonzero if you want to enable extstore])
734+fi
735+
736+if test "x$enable_arm_crc32" = "xyes"; then
737+ AC_DEFINE([ARM_CRC32],1,[Set to nonzero if you want to enable ARMv8 crc32])
738 fi
739
740 AM_CONDITIONAL([BUILD_DTRACE],[test "$build_dtrace" = "yes"])
741 AM_CONDITIONAL([DTRACE_INSTRUMENT_OBJ],[test "$dtrace_instrument_obj" = "yes"])
742 AM_CONDITIONAL([ENABLE_SASL],[test "$enable_sasl" = "yes"])
743 AM_CONDITIONAL([ENABLE_EXTSTORE],[test "$enable_extstore" = "yes"])
744+AM_CONDITIONAL([ENABLE_ARM_CRC32],[test "$enable_arm_crc32" = "yes"])
745
746 AC_SUBST(DTRACE)
747 AC_SUBST(DTRACEFLAGS)
748@@ -490,6 +522,8 @@ AC_CHECK_FUNCS(clock_gettime)
749 AC_CHECK_FUNCS([accept4], [AC_DEFINE(HAVE_ACCEPT4, 1, [Define to 1 if support accept4])])
750 AC_CHECK_FUNCS([getopt_long], [AC_DEFINE(HAVE_GETOPT_LONG, 1, [Define to 1 if support getopt_long])])
751
752+dnl Need to disable opt for alignment check. GCC is too clever and turns this
753+dnl into wide stores and no cmp under O2.
754 AC_DEFUN([AC_C_ALIGNMENT],
755 [AC_CACHE_CHECK(for alignment, ac_cv_c_alignment,
756 [
757@@ -497,6 +531,7 @@ AC_DEFUN([AC_C_ALIGNMENT],
758 [AC_LANG_PROGRAM([
759 #include <stdlib.h>
760 #include <inttypes.h>
761+#pragma GCC optimize ("O0")
762 ], [
763 char *buf = malloc(32);
764
765@@ -551,7 +586,8 @@ dnl Check for usage of 64bit atomics
766 dnl 32bit systems shouldn't have these.
767 have_gcc_64atomics=no
768 AC_MSG_CHECKING(for GCC 64bit atomics)
769-AC_TRY_LINK([],[
770+AC_TRY_LINK([#include <inttypes.h>
771+ ],[
772 uint64_t a;
773 uint64_t b;
774 b = __sync_add_and_fetch(&a, 1);
775diff --git a/crawler.c b/crawler.c
776index 9c81bba..a1a5ead 100644
777--- a/crawler.c
778+++ b/crawler.c
779@@ -218,7 +218,6 @@ static void crawler_expired_eval(crawler_module_t *cm, item *search, uint32_t hv
780 #endif
781 do_item_unlink_nolock(search, hv);
782 do_item_remove(search);
783- assert(search->slabs_clsid == 0);
784 } else {
785 s->seen++;
786 refcount_decr(search);
787@@ -533,6 +532,14 @@ static int do_lru_crawler_start(uint32_t id, uint32_t remaining) {
788 if (remaining == LRU_CRAWLER_CAP_REMAINING) {
789 remaining = do_get_lru_size(sid);
790 }
791+ /* Values for remaining:
792+ * remaining = 0
793+ * - scan all elements, until a NULL is reached
794+ * - if empty, NULL is reached right away
795+ * remaining = n + 1
796+ * - first n elements are parsed (or until a NULL is reached)
797+ */
798+ if (remaining) remaining++;
799 crawlers[sid].remaining = remaining;
800 crawlers[sid].slabs_clsid = sid;
801 crawlers[sid].reclaimed = 0;
802diff --git a/crc32c.c b/crc32c.c
803index 13deee2..a4296a7 100644
804--- a/crc32c.c
805+++ b/crc32c.c
806@@ -40,6 +40,10 @@
807 #include <stdint.h>
808 #include <unistd.h>
809 #include <pthread.h>
810+#include "config.h"
811+#if defined(__linux__) && defined(__aarch64__)
812+#include <sys/auxv.h>
813+#endif
814 #include "crc32c.h"
815
816 /* CRC-32C (iSCSI) polynomial in reversed bit order. */
817@@ -109,6 +113,60 @@ static uint32_t crc32c_sw(uint32_t crci, const void *buf, size_t len)
818 return (uint32_t)crc ^ 0xffffffff;
819 }
820
821+/* Hardware CRC support for aarch64 platform */
822+#if defined(__linux__) && defined(__aarch64__) && defined(ARM_CRC32)
823+
824+#define CRC32CX(crc, value) __asm__("crc32cx %w[c], %w[c], %x[v]":[c]"+r"(crc):[v]"r"(+value))
825+#define CRC32CW(crc, value) __asm__("crc32cw %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value))
826+#define CRC32CH(crc, value) __asm__("crc32ch %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value))
827+#define CRC32CB(crc, value) __asm__("crc32cb %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value))
828+
829+#ifndef HWCAP_CRC32
830+#define HWCAP_CRC32 (1 << 7)
831+#endif /* HWCAP for crc32 */
832+
833+static uint32_t crc32c_hw_aarch64(uint32_t crc, const void* buf, size_t len)
834+{
835+ const uint8_t* p_buf = buf;
836+ uint64_t crc64bit = crc;
837+ for (size_t i = 0; i < len / sizeof(uint64_t); i++) {
838+ CRC32CX(crc64bit, *(uint64_t*) p_buf);
839+ p_buf += sizeof(uint64_t);
840+ }
841+
842+ uint32_t crc32bit = (uint32_t) crc64bit;
843+ len &= sizeof(uint64_t) - 1;
844+ switch (len) {
845+ case 7:
846+ CRC32CB(crc32bit, *p_buf++);
847+ case 6:
848+ CRC32CH(crc32bit, *(uint16_t*) p_buf);
849+ p_buf += 2;
850+ case 4:
851+ CRC32CW(crc32bit, *(uint32_t*) p_buf);
852+ break;
853+ case 3:
854+ CRC32CB(crc32bit, *p_buf++);
855+ case 2:
856+ CRC32CH(crc32bit, *(uint16_t*) p_buf);
857+ break;
858+ case 5:
859+ CRC32CW(crc32bit, *(uint32_t*) p_buf);
860+ p_buf += 4;
861+ case 1:
862+ CRC32CB(crc32bit, *p_buf);
863+ break;
864+ case 0:
865+ break;
866+ }
867+
868+ return crc32bit;
869+}
870+#endif
871+
872+/* Apply if the platform is intel */
873+#if defined(__X86_64__)||defined(__x86_64__)||defined(__ia64__)
874+
875 /* Multiply a matrix times a vector over the Galois field of two elements,
876 GF(2). Each element is a bit in an unsigned integer. mat must have at
877 least as many entries as the power of two for most significant one bit in
878@@ -329,15 +387,27 @@ static uint32_t crc32c_hw(uint32_t crc, const void *buf, size_t len)
879 (have) = (ecx >> 20) & 1; \
880 } while (0)
881
882+#endif
883 /* Compute a CRC-32C. If the crc32 instruction is available, use the hardware
884 version. Otherwise, use the software version. */
885 void crc32c_init(void) {
886+ #if defined(__X86_64__)||defined(__x86_64__)||defined(__ia64__)
887 int sse42;
888-
889 SSE42(sse42);
890+
891 if (sse42) {
892 crc32c = crc32c_hw;
893- } else {
894+ } else
895+ #endif
896+ /* Check if CRC instructions supported by aarch64 */
897+ #if defined(__linux__) && defined(__aarch64__) && defined(ARM_CRC32)
898+ unsigned long hwcap = getauxval(AT_HWCAP);
899+
900+ if (hwcap & HWCAP_CRC32) {
901+ crc32c = crc32c_hw_aarch64;
902+ } else
903+ #endif
904+ {
905 crc32c = crc32c_sw;
906 }
907 }
908diff --git a/debian/changelog b/debian/changelog
909index 43c90a2..1397f89 100644
910--- a/debian/changelog
911+++ b/debian/changelog
912@@ -1,3 +1,11 @@
913+memcached (1.5.10-0ubuntu1) cosmic; urgency=medium
914+
915+ * New upstream release.
916+ * Includes fixes for various failures on various architectures
917+ (LP: #1780838).
918+
919+ -- Robie Basak <robie.basak@ubuntu.com> Mon, 13 Aug 2018 13:10:37 +0100
920+
921 memcached (1.5.6-1) unstable; urgency=medium
922
923 * New upstream release
924diff --git a/debian/control b/debian/control
925index 72e948f..2c63ea0 100644
926--- a/debian/control
927+++ b/debian/control
928@@ -1,7 +1,8 @@
929 Source: memcached
930 Section: web
931 Priority: optional
932-Maintainer: Guillaume Delacour <gui@iroqwa.org>
933+Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
934+XSBC-Original-Maintainer: Guillaume Delacour <gui@iroqwa.org>
935 Build-Depends: debhelper (>> 10), libevent-dev, libsasl2-dev, adduser,
936 autotools-dev, dh-autoreconf
937 Homepage: http://www.memcached.org/
938diff --git a/doc/Makefile b/doc/Makefile
939index 0337680..49cce71 100644
940--- a/doc/Makefile
941+++ b/doc/Makefile
942@@ -191,10 +191,10 @@ OBJEXT = o
943 PACKAGE = memcached
944 PACKAGE_BUGREPORT = memcached@googlegroups.com
945 PACKAGE_NAME = memcached
946-PACKAGE_STRING = memcached 1.5.6
947+PACKAGE_STRING = memcached 1.5.10
948 PACKAGE_TARNAME = memcached
949 PACKAGE_URL =
950-PACKAGE_VERSION = 1.5.6
951+PACKAGE_VERSION = 1.5.10
952 PATH_SEPARATOR = :
953 PROFILER = /usr/bin/gcov
954 PROFILER_FLAGS = -fprofile-arcs -ftest-coverage
955@@ -202,7 +202,7 @@ PROFILER_LDFLAGS = -lgcov
956 SET_MAKE =
957 SHELL = /bin/bash
958 STRIP =
959-VERSION = 1.5.6
960+VERSION = 1.5.10
961 XML2RFC = /usr/bin/xml2rfc
962 XSLTPROC = no
963 abs_builddir = /home/dormando/d/p/danga/git/memcached/doc
964diff --git a/doc/memcached.1 b/doc/memcached.1
965index caa53b7..04aca5f 100644
966--- a/doc/memcached.1
967+++ b/doc/memcached.1
968@@ -62,10 +62,10 @@ caches, so consult the README and memcached homepage for configuration
969 suggestions.
970 .TP
971 .B \-p, --port=<num>
972-Listen on TCP port <num>, the default is port 11211.
973+Listen on TCP port <num>, the default is port 11211. 0 means off.
974 .TP
975 .B \-U, --udp-port=<num>
976-Listen on UDP port <num>, the default is port 11211, 0 is off.
977+Listen on UDP port <num>, the default is port 0, which is off.
978 .TP
979 .B \-M, --disable-evictions
980 Disable automatic removal of items from the cache when out of memory.
981diff --git a/doc/protocol.txt b/doc/protocol.txt
982index 9c8e8bc..0984594 100644
983--- a/doc/protocol.txt
984+++ b/doc/protocol.txt
985@@ -849,6 +849,8 @@ other stats command.
986 | | bool | If yes, stores numbers from VALUE response |
987 | | | inside an item, using up to 24 bytes. |
988 | | | Small slowdown for ASCII get, faster sets. |
989+| drop_privileges | bool | If yes, and available, drop unused syscalls |
990+| | | (see seccomp on Linux, pledge on OpenBSD) |
991 |-------------------+----------+----------------------------------------------|
992
993
994diff --git a/extstore.c b/extstore.c
995index 02558c0..726435c 100644
996--- a/extstore.c
997+++ b/extstore.c
998@@ -62,6 +62,7 @@ typedef struct _store_page {
999 unsigned int allocated;
1000 unsigned int written; /* item offsets can be past written if wbuf not flushed */
1001 unsigned int bucket; /* which bucket the page is linked into */
1002+ unsigned int free_bucket; /* which bucket this page returns to when freed */
1003 int fd;
1004 unsigned short id;
1005 bool active; /* actively being written to */
1006@@ -95,6 +96,7 @@ struct store_engine {
1007 store_maint_thread *maint_thread;
1008 store_page *page_freelist;
1009 store_page **page_buckets; /* stack of pages currently allocated to each bucket */
1010+ store_page **free_page_buckets; /* stack of use-case isolated free pages */
1011 size_t page_size;
1012 unsigned int version; /* global version counter */
1013 unsigned int last_io_thread; /* round robin the IO threads */
1014@@ -102,6 +104,7 @@ struct store_engine {
1015 unsigned int page_count;
1016 unsigned int page_free; /* unallocated pages */
1017 unsigned int page_bucketcount; /* count of potential page buckets */
1018+ unsigned int free_page_bucketcount; /* count of free page buckets */
1019 unsigned int io_depth; /* FIXME: Might cache into thr struct */
1020 pthread_mutex_t stats_mutex;
1021 struct extstore_stats stats;
1022@@ -192,11 +195,11 @@ const char *extstore_err(enum extstore_res res) {
1023 return rv;
1024 }
1025
1026-void *extstore_init(char *fn, struct extstore_conf *cf,
1027+// TODO: #define's for DEFAULT_BUCKET, FREE_VERSION, etc
1028+void *extstore_init(struct extstore_conf_file *fh, struct extstore_conf *cf,
1029 enum extstore_res *res) {
1030 int i;
1031- int fd;
1032- uint64_t offset = 0;
1033+ struct extstore_conf_file *f = NULL;
1034 pthread_t thread;
1035
1036 if (cf->page_size % cf->wbuf_size != 0) {
1037@@ -227,43 +230,72 @@ void *extstore_init(char *fn, struct extstore_conf *cf,
1038 }
1039
1040 e->page_size = cf->page_size;
1041- fd = open(fn, O_RDWR | O_CREAT | O_TRUNC, 0644);
1042- if (fd < 0) {
1043- *res = EXTSTORE_INIT_OPEN_FAIL;
1044+ for (f = fh; f != NULL; f = f->next) {
1045+ f->fd = open(f->file, O_RDWR | O_CREAT | O_TRUNC, 0644);
1046+ if (f->fd < 0) {
1047+ *res = EXTSTORE_INIT_OPEN_FAIL;
1048 #ifdef EXTSTORE_DEBUG
1049- perror("open");
1050+ perror("open");
1051 #endif
1052- free(e);
1053- return NULL;
1054+ free(e);
1055+ return NULL;
1056+ }
1057+ e->page_count += f->page_count;
1058+ f->offset = 0;
1059 }
1060
1061- e->pages = calloc(cf->page_count, sizeof(store_page));
1062+ e->pages = calloc(e->page_count, sizeof(store_page));
1063 if (e->pages == NULL) {
1064 *res = EXTSTORE_INIT_OOM;
1065- close(fd);
1066+ // FIXME: loop-close. make error label
1067 free(e);
1068 return NULL;
1069 }
1070
1071- for (i = 0; i < cf->page_count; i++) {
1072+ // interleave the pages between devices
1073+ f = NULL; // start at the first device.
1074+ for (i = 0; i < e->page_count; i++) {
1075+ // find next device with available pages
1076+ while (1) {
1077+ // restart the loop
1078+ if (f == NULL || f->next == NULL) {
1079+ f = fh;
1080+ } else {
1081+ f = f->next;
1082+ }
1083+ if (f->page_count) {
1084+ f->page_count--;
1085+ break;
1086+ }
1087+ }
1088 pthread_mutex_init(&e->pages[i].mutex, NULL);
1089 e->pages[i].id = i;
1090- e->pages[i].fd = fd;
1091- e->pages[i].offset = offset;
1092+ e->pages[i].fd = f->fd;
1093+ e->pages[i].free_bucket = f->free_bucket;
1094+ e->pages[i].offset = f->offset;
1095 e->pages[i].free = true;
1096- offset += e->page_size;
1097+ f->offset += e->page_size;
1098 }
1099
1100- for (i = cf->page_count-1; i > 0; i--) {
1101- e->pages[i].next = e->page_freelist;
1102- e->page_freelist = &e->pages[i];
1103+ // free page buckets allows the app to organize devices by use case
1104+ e->free_page_buckets = calloc(cf->page_buckets, sizeof(store_page *));
1105+ e->page_bucketcount = cf->page_buckets;
1106+
1107+ for (i = e->page_count-1; i > 0; i--) {
1108 e->page_free++;
1109+ if (e->pages[i].free_bucket == 0) {
1110+ e->pages[i].next = e->page_freelist;
1111+ e->page_freelist = &e->pages[i];
1112+ } else {
1113+ int fb = e->pages[i].free_bucket;
1114+ e->pages[i].next = e->free_page_buckets[fb];
1115+ e->free_page_buckets[fb] = &e->pages[i];
1116+ }
1117 }
1118
1119 // 0 is magic "page is freed" version
1120 e->version = 1;
1121
1122- e->page_count = cf->page_count;
1123 // scratch data for stats. TODO: malloc failure handle
1124 e->stats.page_data =
1125 calloc(e->page_count, sizeof(struct extstore_page_data));
1126@@ -305,8 +337,12 @@ void *extstore_init(char *fn, struct extstore_conf *cf,
1127 e->maint_thread = calloc(1, sizeof(store_maint_thread));
1128 e->maint_thread->e = e;
1129 // FIXME: error handling
1130+ pthread_mutex_init(&e->maint_thread->mutex, NULL);
1131+ pthread_cond_init(&e->maint_thread->cond, NULL);
1132 pthread_create(&thread, NULL, extstore_maint_thread, e->maint_thread);
1133
1134+ extstore_run_maint(e);
1135+
1136 return (void *)e;
1137 }
1138
1139@@ -316,13 +352,25 @@ void extstore_run_maint(void *ptr) {
1140 }
1141
1142 // call with *e locked
1143-static store_page *_allocate_page(store_engine *e, unsigned int bucket) {
1144+static store_page *_allocate_page(store_engine *e, unsigned int bucket,
1145+ unsigned int free_bucket) {
1146 assert(!e->page_buckets[bucket] || e->page_buckets[bucket]->allocated == e->page_size);
1147- store_page *tmp = e->page_freelist;
1148- E_DEBUG("EXTSTORE: allocating new page\n");
1149- if (e->page_free > 0) {
1150- assert(e->page_freelist != NULL);
1151+ store_page *tmp = NULL;
1152+ // if a specific free bucket was requested, check there first
1153+ if (free_bucket != 0 && e->free_page_buckets[free_bucket] != NULL) {
1154+ assert(e->page_free > 0);
1155+ tmp = e->free_page_buckets[free_bucket];
1156+ e->free_page_buckets[free_bucket] = tmp->next;
1157+ }
1158+ // failing that, try the global list.
1159+ if (tmp == NULL && e->page_freelist != NULL) {
1160+ tmp = e->page_freelist;
1161 e->page_freelist = tmp->next;
1162+ }
1163+ E_DEBUG("EXTSTORE: allocating new page\n");
1164+ // page_freelist can be empty if the only free pages are specialized and
1165+ // we didn't just request one.
1166+ if (e->page_free > 0 && tmp != NULL) {
1167 tmp->next = e->page_buckets[bucket];
1168 e->page_buckets[bucket] = tmp;
1169 tmp->active = true;
1170@@ -432,7 +480,8 @@ static void _submit_wbuf(store_engine *e, store_page *p) {
1171 * new page. best if used from a background thread that can harmlessly retry.
1172 */
1173
1174-int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) {
1175+int extstore_write_request(void *ptr, unsigned int bucket,
1176+ unsigned int free_bucket, obj_io *io) {
1177 store_engine *e = (store_engine *)ptr;
1178 store_page *p;
1179 int ret = -1;
1180@@ -442,7 +491,7 @@ int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) {
1181 pthread_mutex_lock(&e->mutex);
1182 p = e->page_buckets[bucket];
1183 if (!p) {
1184- p = _allocate_page(e, bucket);
1185+ p = _allocate_page(e, bucket, free_bucket);
1186 }
1187 pthread_mutex_unlock(&e->mutex);
1188 if (!p)
1189@@ -456,7 +505,7 @@ int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) {
1190 ((!p->wbuf || p->wbuf->full) && p->allocated >= e->page_size)) {
1191 pthread_mutex_unlock(&p->mutex);
1192 pthread_mutex_lock(&e->mutex);
1193- _allocate_page(e, bucket);
1194+ _allocate_page(e, bucket, free_bucket);
1195 pthread_mutex_unlock(&e->mutex);
1196 return ret;
1197 }
1198@@ -510,7 +559,7 @@ void extstore_write(void *ptr, obj_io *io) {
1199 /* engine submit function; takes engine, item_io stack.
1200 * lock io_thread context and add stack?
1201 * signal io thread to wake.
1202- * return sucess.
1203+ * return success.
1204 */
1205 int extstore_submit(void *ptr, obj_io *io) {
1206 store_engine *e = (store_engine *)ptr;
1207@@ -603,7 +652,7 @@ void extstore_close_page(void *ptr, unsigned int page_id, uint64_t page_version)
1208
1209 /* Finds an attached wbuf that can satisfy the read.
1210 * Since wbufs can potentially be flushed to disk out of order, they are only
1211- * removed as the head of the list successfuly flushes to disk.
1212+ * removed as the head of the list successfully flushes to disk.
1213 */
1214 // call with *p locked
1215 // FIXME: protect from reading past wbuf
1216@@ -762,8 +811,14 @@ static void _free_page(store_engine *e, store_page *p) {
1217 p->closed = false;
1218 p->free = true;
1219 // add to page stack
1220- p->next = e->page_freelist;
1221- e->page_freelist = p;
1222+ // TODO: free_page_buckets first class and remove redundancy?
1223+ if (p->free_bucket != 0) {
1224+ p->next = e->free_page_buckets[p->free_bucket];
1225+ e->free_page_buckets[p->free_bucket] = p;
1226+ } else {
1227+ p->next = e->page_freelist;
1228+ e->page_freelist = p;
1229+ }
1230 e->page_free++;
1231 pthread_mutex_unlock(&e->mutex);
1232 }
1233@@ -795,7 +850,9 @@ static void *extstore_maint_thread(void *arg) {
1234
1235 pthread_cond_wait(&me->cond, &me->mutex);
1236 pthread_mutex_lock(&e->mutex);
1237- if (e->page_free == 0) {
1238+ // default freelist requires at least one page free.
1239+ // specialized freelists fall back to default once full.
1240+ if (e->page_free == 0 || e->page_freelist == NULL) {
1241 do_evict = true;
1242 }
1243 pthread_mutex_unlock(&e->mutex);
1244@@ -804,6 +861,7 @@ static void *extstore_maint_thread(void *arg) {
1245 for (i = 0; i < e->page_count; i++) {
1246 store_page *p = &e->pages[i];
1247 pthread_mutex_lock(&p->mutex);
1248+ pd[p->id].free_bucket = p->free_bucket;
1249 if (p->active || p->free) {
1250 pthread_mutex_unlock(&p->mutex);
1251 continue;
1252@@ -812,7 +870,13 @@ static void *extstore_maint_thread(void *arg) {
1253 pd[p->id].version = p->version;
1254 pd[p->id].bytes_used = p->bytes_used;
1255 pd[p->id].bucket = p->bucket;
1256- if (p->version < low_version) {
1257+ // low_version/low_page are only used in the eviction
1258+ // scenario. when we evict, it's only to fill the default page
1259+ // bucket again.
1260+ // TODO: experiment with allowing evicting up to a single page
1261+ // for any specific free bucket. this is *probably* required
1262+ // since it could cause a load bias on default-only devices?
1263+ if (p->free_bucket == 0 && p->version < low_version) {
1264 low_version = p->version;
1265 low_page = i;
1266 }
1267diff --git a/extstore.h b/extstore.h
1268index a466562..6814415 100644
1269--- a/extstore.h
1270+++ b/extstore.h
1271@@ -8,6 +8,7 @@ struct extstore_page_data {
1272 uint64_t version;
1273 uint64_t bytes_used;
1274 unsigned int bucket;
1275+ unsigned int free_bucket;
1276 };
1277
1278 /* Pages can have objects deleted from them at any time. This creates holes
1279@@ -43,12 +44,23 @@ struct extstore_conf {
1280 unsigned int page_size; // ideally 64-256M in size
1281 unsigned int page_count;
1282 unsigned int page_buckets; // number of different writeable pages
1283+ unsigned int free_page_buckets; // buckets of dedicated pages (see code)
1284 unsigned int wbuf_size; // must divide cleanly into page_size
1285 unsigned int wbuf_count; // this might get locked to "2 per active page"
1286 unsigned int io_threadcount;
1287 unsigned int io_depth; // with normal I/O, hits locks less. req'd for AIO
1288 };
1289
1290+struct extstore_conf_file {
1291+ unsigned int page_count;
1292+ char *file;
1293+ int fd; // internal usage
1294+ uint64_t offset; // internal usage
1295+ unsigned int bucket; // free page bucket
1296+ unsigned int free_bucket; // specialized free bucket
1297+ struct extstore_conf_file *next;
1298+};
1299+
1300 enum obj_io_mode {
1301 OBJ_IO_READ = 0,
1302 OBJ_IO_WRITE,
1303@@ -87,8 +99,8 @@ enum extstore_res {
1304 };
1305
1306 const char *extstore_err(enum extstore_res res);
1307-void *extstore_init(char *fn, struct extstore_conf *cf, enum extstore_res *res);
1308-int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io);
1309+void *extstore_init(struct extstore_conf_file *fh, struct extstore_conf *cf, enum extstore_res *res);
1310+int extstore_write_request(void *ptr, unsigned int bucket, unsigned int free_bucket, obj_io *io);
1311 void extstore_write(void *ptr, obj_io *io);
1312 int extstore_submit(void *ptr, obj_io *io);
1313 /* count are the number of objects being removed, bytes are the original
1314diff --git a/items.c b/items.c
1315index 0aefaf0..d4ce5a1 100644
1316--- a/items.c
1317+++ b/items.c
1318@@ -285,6 +285,13 @@ item *do_item_alloc(char *key, const size_t nkey, const unsigned int flags,
1319 if (settings.use_cas) {
1320 htotal += sizeof(uint64_t);
1321 }
1322+#ifdef NEED_ALIGN
1323+ // header chunk needs to be padded on some systems
1324+ int remain = htotal % 8;
1325+ if (remain != 0) {
1326+ htotal += 8 - remain;
1327+ }
1328+#endif
1329 hdr_id = slabs_clsid(htotal);
1330 it = do_item_alloc_pull(htotal, hdr_id);
1331 /* setting ITEM_CHUNKED is fine here because we aren't LINKED yet. */
1332@@ -336,7 +343,7 @@ item *do_item_alloc(char *key, const size_t nkey, const unsigned int flags,
1333
1334 /* Initialize internal chunk. */
1335 if (it->it_flags & ITEM_CHUNKED) {
1336- item_chunk *chunk = (item_chunk *) ITEM_data(it);
1337+ item_chunk *chunk = (item_chunk *) ITEM_schunk(it);
1338
1339 chunk->next = 0;
1340 chunk->prev = 0;
1341@@ -1538,7 +1545,6 @@ static void *lru_maintainer_thread(void *arg) {
1342 void *storage = arg;
1343 if (storage != NULL)
1344 sam = &slab_automove_extstore;
1345- int x;
1346 #endif
1347 int i;
1348 useconds_t to_sleep = MIN_LRU_MAINTAINER_SLEEP;
1349@@ -1592,20 +1598,6 @@ static void *lru_maintainer_thread(void *arg) {
1350 }
1351
1352 int did_moves = lru_maintainer_juggle(i);
1353-#ifdef EXTSTORE
1354- // Deeper loop to speed up pushing to storage.
1355- if (storage) {
1356- for (x = 0; x < 500; x++) {
1357- int found;
1358- found = lru_maintainer_store(storage, i);
1359- if (found) {
1360- did_moves += found;
1361- } else {
1362- break;
1363- }
1364- }
1365- }
1366-#endif
1367 if (did_moves == 0) {
1368 if (backoff_juggles[i] != 0) {
1369 backoff_juggles[i] += backoff_juggles[i] / 8;
1370diff --git a/linux_priv.c b/linux_priv.c
1371index 04155dd..cc9aef3 100644
1372--- a/linux_priv.c
1373+++ b/linux_priv.c
1374@@ -2,11 +2,11 @@
1375 #include <seccomp.h>
1376 #include <errno.h>
1377 #include <stdlib.h>
1378+#include <sys/ioctl.h>
1379 #include "memcached.h"
1380
1381-// In the future when the system is more tested this could be switched
1382-// to SCMP_ACT_KILL instead.
1383-#define DENY_ACTION SCMP_ACT_ERRNO(EACCES)
1384+// If anything crosses the policy, kill the process.
1385+#define DENY_ACTION SCMP_ACT_KILL
1386
1387 void drop_privileges(void) {
1388 scmp_filter_ctx ctx = seccomp_init(DENY_ACTION);
1389@@ -16,24 +16,41 @@ void drop_privileges(void) {
1390
1391 int rc = 0;
1392 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0);
1393+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0);
1394 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);
1395 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0);
1396+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_pwait), 0);
1397 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept4), 0);
1398 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept), 0);
1399 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
1400+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 0);
1401 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0);
1402 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mmap), 0);
1403 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0);
1404 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(shmctl), 0);
1405 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
1406+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0);
1407+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TIOCGWINSZ));
1408+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TCGETS));
1409+
1410+#if defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC)
1411+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clock_gettime), 0);
1412+#endif
1413
1414 #ifdef MEMCACHED_DEBUG
1415 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0);
1416 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0);
1417 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
1418+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(readv), 0);
1419 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0);
1420 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0);
1421 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0);
1422+
1423+ if (settings.relaxed_privileges) {
1424+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(openat), 0);
1425+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mkdir), 0);
1426+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(access), 0);
1427+ }
1428 #endif
1429
1430 if (rc != 0) {
1431@@ -45,8 +62,13 @@ void drop_privileges(void) {
1432 goto fail;
1433 }
1434
1435+ seccomp_release(ctx);
1436+ return;
1437+
1438 fail:
1439 seccomp_release(ctx);
1440+ fprintf(stderr, "Failed to set a seccomp profile on the main thread\n");
1441+ exit(EXIT_FAILURE);
1442 }
1443
1444 void drop_worker_privileges(void) {
1445@@ -57,10 +79,14 @@ void drop_worker_privileges(void) {
1446
1447 int rc = 0;
1448 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0);
1449+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0);
1450 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);
1451 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0);
1452+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_pwait), 0);
1453 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_ctl), 0);
1454+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(poll), 0);
1455 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
1456+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(readv), 0);
1457 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mprotect), 0);
1458 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpeername), 0);
1459 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0);
1460@@ -70,6 +96,8 @@ void drop_worker_privileges(void) {
1461 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mremap), 0);
1462 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0);
1463 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(recvfrom), 0);
1464+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0);
1465+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TIOCGWINSZ));
1466
1467 // for spawning the LRU crawler
1468 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clone), 0);
1469@@ -83,20 +111,29 @@ void drop_worker_privileges(void) {
1470
1471 if (settings.shutdown_command) {
1472 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tgkill), 0);
1473+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tkill), 0);
1474 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
1475 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0);
1476- rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0);
1477 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0);
1478 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(gettid), 0);
1479 }
1480
1481 if (settings.relaxed_privileges) {
1482+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(openat), 0);
1483+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mkdir), 0);
1484+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(access), 0);
1485 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0);
1486 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0);
1487 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0);
1488 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
1489+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 0);
1490 } else {
1491+ // stdout
1492 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 1));
1493+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 1, SCMP_A0(SCMP_CMP_EQ, 1));
1494+ // stderr
1495+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 2));
1496+ rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 1, SCMP_A0(SCMP_CMP_EQ, 2));
1497 }
1498
1499 if (rc != 0) {
1500@@ -108,6 +145,11 @@ void drop_worker_privileges(void) {
1501 goto fail;
1502 }
1503
1504+ seccomp_release(ctx);
1505+ return;
1506+
1507 fail:
1508 seccomp_release(ctx);
1509+ fprintf(stderr, "Failed to set a seccomp profile on a worker thread\n");
1510+ exit(EXIT_FAILURE);
1511 }
1512diff --git a/logger.c b/logger.c
1513index 7af9917..1322d7d 100644
1514--- a/logger.c
1515+++ b/logger.c
1516@@ -363,7 +363,7 @@ static int logger_thread_read(logger *l, struct logger_stats *ls) {
1517 } else {
1518 logger_thread_write_entry(e, ls, scratch, scratch_len);
1519 }
1520- pos += sizeof(logentry) + e->size;
1521+ pos += sizeof(logentry) + e->size + e->pad;
1522 }
1523 assert(pos <= size);
1524
1525@@ -699,8 +699,9 @@ enum logger_ret_type logger_log(logger *l, const enum log_entry_type event, cons
1526 l->dropped++;
1527 return LOGGER_RET_NOSPACE;
1528 }
1529- e->gid = logger_get_gid();
1530 e->event = d->subtype;
1531+ e->pad = 0;
1532+ e->gid = logger_get_gid();
1533 /* TODO: Could pass this down as an argument now that we're using
1534 * LOGGER_LOG() macro.
1535 */
1536@@ -751,11 +752,19 @@ enum logger_ret_type logger_log(logger *l, const enum log_entry_type event, cons
1537 rel_time_t sttl = va_arg(ap, rel_time_t);
1538 uint8_t sclsid = va_arg(ap, int);
1539 _logger_log_item_store(e, status, comm, skey, snkey, sttl, sclsid);
1540+ va_end(ap);
1541 break;
1542 }
1543
1544+#ifdef NEED_ALIGN
1545+ /* Need to ensure *next* request is aligned. */
1546+ if (sizeof(logentry) + e->size % 8 != 0) {
1547+ e->pad = 8 - (sizeof(logentry) + e->size % 8);
1548+ }
1549+#endif
1550+
1551 /* Push pointer forward by the actual amount required */
1552- if (bipbuf_push(buf, (sizeof(logentry) + e->size)) == 0) {
1553+ if (bipbuf_push(buf, (sizeof(logentry) + e->size + e->pad)) == 0) {
1554 fprintf(stderr, "LOGGER: Failed to bipbuf push a text entry\n");
1555 pthread_mutex_unlock(&l->mutex);
1556 return LOGGER_RET_ERR;
1557diff --git a/logger.h b/logger.h
1558index 3d4c44c..730c86f 100644
1559--- a/logger.h
1560+++ b/logger.h
1561@@ -100,6 +100,7 @@ struct logentry_item_store {
1562
1563 typedef struct _logentry {
1564 enum log_entry_subtype event;
1565+ uint8_t pad;
1566 uint16_t eflags;
1567 uint64_t gid;
1568 struct timeval tv; /* not monotonic! */
1569diff --git a/memcached.c b/memcached.c
1570index 7178666..621b317 100644
1571--- a/memcached.c
1572+++ b/memcached.c
1573@@ -271,7 +271,7 @@ static void settings_init(void) {
1574 settings.crawls_persleep = 1000;
1575 settings.logger_watcher_buf_size = LOGGER_WATCHER_BUF_SIZE;
1576 settings.logger_buf_size = LOGGER_BUF_SIZE;
1577- settings.drop_privileges = true;
1578+ settings.drop_privileges = false;
1579 #ifdef MEMCACHED_DEBUG
1580 settings.relaxed_privileges = false;
1581 #endif
1582@@ -612,6 +612,7 @@ conn *conn_new(const int sfd, enum conn_states init_state,
1583 c->iovused = 0;
1584 c->msgcurr = 0;
1585 c->msgused = 0;
1586+ c->sasl_started = false;
1587 c->authenticated = false;
1588 c->last_cmd_time = current_time; /* initialize for idle kicker */
1589 #ifdef EXTSTORE
1590@@ -648,8 +649,19 @@ static void recache_or_free(conn *c, io_wrap *wrap) {
1591 item *it;
1592 it = (item *)wrap->io.buf;
1593 bool do_free = true;
1594- // If request was ultimately a miss, unlink the header.
1595- if (wrap->miss) {
1596+ if (wrap->active) {
1597+ // If request never dispatched, free the read buffer but leave the
1598+ // item header alone.
1599+ do_free = false;
1600+ size_t ntotal = ITEM_ntotal(wrap->hdr_it);
1601+ slabs_free(it, ntotal, slabs_clsid(ntotal));
1602+ c->io_wrapleft--;
1603+ assert(c->io_wrapleft >= 0);
1604+ pthread_mutex_lock(&c->thread->stats.mutex);
1605+ c->thread->stats.get_aborted_extstore++;
1606+ pthread_mutex_unlock(&c->thread->stats.mutex);
1607+ } else if (wrap->miss) {
1608+ // If request was ultimately a miss, unlink the header.
1609 do_free = false;
1610 size_t ntotal = ITEM_ntotal(wrap->hdr_it);
1611 item_unlink(wrap->hdr_it);
1612@@ -1020,7 +1032,7 @@ static int add_iov(conn *c, const void *buf, int len) {
1613
1614 static int add_chunked_item_iovs(conn *c, item *it, int len) {
1615 assert(it->it_flags & ITEM_CHUNKED);
1616- item_chunk *ch = (item_chunk *) ITEM_data(it);
1617+ item_chunk *ch = (item_chunk *) ITEM_schunk(it);
1618 while (ch) {
1619 int todo = (len > ch->used) ? ch->used : len;
1620 if (add_iov(c, ch->data, todo) != 0) {
1621@@ -1640,13 +1652,8 @@ static void process_bin_get_or_touch(conn *c) {
1622 rsp->message.header.response.cas = htonll(ITEM_get_cas(it));
1623
1624 // add the flags
1625- if (settings.inline_ascii_response) {
1626- rsp->message.body.flags = htonl(strtoul(ITEM_suffix(it), NULL, 10));
1627- } else if (it->nsuffix > 0) {
1628- rsp->message.body.flags = htonl(*((uint32_t *)ITEM_suffix(it)));
1629- } else {
1630- rsp->message.body.flags = 0;
1631- }
1632+ FLAGS_CONV(settings.inline_ascii_response, it, rsp->message.body.flags);
1633+ rsp->message.body.flags = htonl(rsp->message.body.flags);
1634 add_iov(c, &rsp->message.body, sizeof(rsp->message.body));
1635
1636 if (should_return_key) {
1637@@ -1663,10 +1670,14 @@ static void process_bin_get_or_touch(conn *c) {
1638 iovcnt = 3;
1639 iovst = c->iovused - 2;
1640 }
1641- // FIXME: this can return an error, but code flow doesn't
1642- // allow bailing here.
1643- if (_get_extstore(c, it, iovst, iovcnt) != 0)
1644+
1645+ if (_get_extstore(c, it, iovst, iovcnt) != 0) {
1646+ pthread_mutex_lock(&c->thread->stats.mutex);
1647+ c->thread->stats.get_oom_extstore++;
1648+ pthread_mutex_unlock(&c->thread->stats.mutex);
1649+
1650 failed = true;
1651+ }
1652 } else if ((it->it_flags & ITEM_CHUNKED) == 0) {
1653 add_iov(c, ITEM_data(it), it->nbytes - 2);
1654 } else {
1655@@ -1686,10 +1697,11 @@ static void process_bin_get_or_touch(conn *c) {
1656 c->write_and_go = conn_new_cmd;
1657 /* Remember this command so we can garbage collect it later */
1658 #ifdef EXTSTORE
1659- if ((it->it_flags & ITEM_HDR) == 0) {
1660- c->item = it;
1661- } else {
1662+ if ((it->it_flags & ITEM_HDR) != 0 && should_return_value) {
1663+ // Only have extstore clean if header and returning value.
1664 c->item = NULL;
1665+ } else {
1666+ c->item = it;
1667 }
1668 #else
1669 c->item = it;
1670@@ -2105,8 +2117,16 @@ static void process_bin_complete_sasl_auth(conn *c) {
1671 result = sasl_server_start(c->sasl_conn, mech,
1672 challenge, vlen,
1673 &out, &outlen);
1674+ c->sasl_started = (result == SASL_OK || result == SASL_CONTINUE);
1675 break;
1676 case PROTOCOL_BINARY_CMD_SASL_STEP:
1677+ if (!c->sasl_started) {
1678+ if (settings.verbose) {
1679+ fprintf(stderr, "%d: SASL_STEP called but sasl_server_start "
1680+ "not called for this connection!\n", c->sfd);
1681+ }
1682+ break;
1683+ }
1684 result = sasl_server_step(c->sasl_conn,
1685 challenge, vlen,
1686 &out, &outlen);
1687@@ -2465,7 +2485,15 @@ static void process_bin_update(conn *c) {
1688 }
1689
1690 c->item = it;
1691+#ifdef NEED_ALIGN
1692+ if (it->it_flags & ITEM_CHUNKED) {
1693+ c->ritem = ITEM_schunk(it);
1694+ } else {
1695+ c->ritem = ITEM_data(it);
1696+ }
1697+#else
1698 c->ritem = ITEM_data(it);
1699+#endif
1700 c->rlbytes = vlen;
1701 conn_set_state(c, conn_nread);
1702 c->substate = bin_read_set_value;
1703@@ -2520,7 +2548,15 @@ static void process_bin_append_prepend(conn *c) {
1704 }
1705
1706 c->item = it;
1707+#ifdef NEED_ALIGN
1708+ if (it->it_flags & ITEM_CHUNKED) {
1709+ c->ritem = ITEM_schunk(it);
1710+ } else {
1711+ c->ritem = ITEM_data(it);
1712+ }
1713+#else
1714 c->ritem = ITEM_data(it);
1715+#endif
1716 c->rlbytes = vlen;
1717 conn_set_state(c, conn_nread);
1718 c->substate = bin_read_set_value;
1719@@ -2681,7 +2717,7 @@ static void complete_nread(conn *c) {
1720 /* Destination must always be chunked */
1721 /* This should be part of item.c */
1722 static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) {
1723- item_chunk *dch = (item_chunk *) ITEM_data(d_it);
1724+ item_chunk *dch = (item_chunk *) ITEM_schunk(d_it);
1725 /* Advance dch until we find free space */
1726 while (dch->size == dch->used) {
1727 if (dch->next) {
1728@@ -2693,7 +2729,7 @@ static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) {
1729
1730 if (s_it->it_flags & ITEM_CHUNKED) {
1731 int remain = len;
1732- item_chunk *sch = (item_chunk *) ITEM_data(s_it);
1733+ item_chunk *sch = (item_chunk *) ITEM_schunk(s_it);
1734 int copied = 0;
1735 /* Fills dch's to capacity, not straight copy sch in case data is
1736 * being added or removed (ie append/prepend)
1737@@ -2857,15 +2893,7 @@ enum store_item_type do_store_item(item *it, int comm, conn *c, const uint32_t h
1738 if (stored == NOT_STORED) {
1739 /* we have it and old_it here - alloc memory to hold both */
1740 /* flags was already lost - so recover them from ITEM_suffix(it) */
1741-
1742- if (settings.inline_ascii_response) {
1743- flags = (uint32_t) strtoul(ITEM_suffix(old_it), (char **) NULL, 10);
1744- } else if (old_it->nsuffix > 0) {
1745- flags = *((uint32_t *)ITEM_suffix(old_it));
1746- } else {
1747- flags = 0;
1748- }
1749-
1750+ FLAGS_CONV(settings.inline_ascii_response, old_it, flags);
1751 new_it = do_item_alloc(key, it->nkey, flags, old_it->exptime, it->nbytes + old_it->nbytes - 2 /* CRLF */);
1752
1753 /* copy data from it and old_it to new_it */
1754@@ -3105,6 +3133,8 @@ static void server_stats(ADD_STAT add_stats, conn *c) {
1755 #ifdef EXTSTORE
1756 if (c->thread->storage) {
1757 APPEND_STAT("get_extstore", "%llu", (unsigned long long)thread_stats.get_extstore);
1758+ APPEND_STAT("get_aborted_extstore", "%llu", (unsigned long long)thread_stats.get_aborted_extstore);
1759+ APPEND_STAT("get_oom_extstore", "%llu", (unsigned long long)thread_stats.get_oom_extstore);
1760 APPEND_STAT("recache_from_extstore", "%llu", (unsigned long long)thread_stats.recache_from_extstore);
1761 APPEND_STAT("miss_from_extstore", "%llu", (unsigned long long)thread_stats.miss_from_extstore);
1762 APPEND_STAT("badcrc_from_extstore", "%llu", (unsigned long long)thread_stats.badcrc_from_extstore);
1763@@ -3242,6 +3272,9 @@ static void process_stat_settings(ADD_STAT add_stats, void *c) {
1764 APPEND_STAT("worker_logbuf_size", "%u", settings.logger_buf_size);
1765 APPEND_STAT("track_sizes", "%s", item_stats_sizes_status() ? "yes" : "no");
1766 APPEND_STAT("inline_ascii_response", "%s", settings.inline_ascii_response ? "yes" : "no");
1767+#ifdef HAVE_DROP_PRIVILEGES
1768+ APPEND_STAT("drop_privileges", "%s", settings.drop_privileges ? "yes" : "no");
1769+#endif
1770 #ifdef EXTSTORE
1771 APPEND_STAT("ext_item_size", "%u", settings.ext_item_size);
1772 APPEND_STAT("ext_item_age", "%u", settings.ext_item_age);
1773@@ -3384,6 +3417,8 @@ static void process_extstore_stats(ADD_STAT add_stats, conn *c) {
1774 (unsigned long long) st.page_data[i].bytes_used);
1775 APPEND_NUM_STAT(i, "bucket", "%u",
1776 st.page_data[i].bucket);
1777+ APPEND_NUM_STAT(i, "free_bucket", "%u",
1778+ st.page_data[i].free_bucket);
1779 }
1780 }
1781 #endif
1782@@ -3583,14 +3618,14 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) {
1783 // item is chunked, crc the iov's
1784 if (io->iov != NULL) {
1785 // first iov is the header, which we don't use beyond crc
1786- crc2 = crc32c(0, (char *)io->iov[0].iov_base+32, io->iov[0].iov_len-32);
1787+ crc2 = crc32c(0, (char *)io->iov[0].iov_base+STORE_OFFSET, io->iov[0].iov_len-STORE_OFFSET);
1788 // make sure it's not sent. hack :(
1789 io->iov[0].iov_len = 0;
1790 for (x = 1; x < io->iovcnt; x++) {
1791 crc2 = crc32c(crc2, (char *)io->iov[x].iov_base, io->iov[x].iov_len);
1792 }
1793 } else {
1794- crc2 = crc32c(0, (char *)read_it+32, io->len-32);
1795+ crc2 = crc32c(0, (char *)read_it+STORE_OFFSET, io->len-STORE_OFFSET);
1796 }
1797
1798 if (crc != crc2) {
1799@@ -3639,6 +3674,7 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) {
1800 }
1801 }
1802 }
1803+ wrap->miss = false;
1804 // iov_len is already set
1805 // TODO: Should do that here instead and cuddle in the wrap object
1806 }
1807@@ -3657,23 +3693,20 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) {
1808
1809 // FIXME: This completely breaks UDP support.
1810 static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
1811+#ifdef NEED_ALIGN
1812+ item_hdr hdr;
1813+ memcpy(&hdr, ITEM_data(it), sizeof(hdr));
1814+#else
1815 item_hdr *hdr = (item_hdr *)ITEM_data(it);
1816+#endif
1817 size_t ntotal = ITEM_ntotal(it);
1818 unsigned int clsid = slabs_clsid(ntotal);
1819 item *new_it;
1820 bool chunked = false;
1821 if (ntotal > settings.slab_chunk_size_max) {
1822 // Pull a chunked item header.
1823- // FIXME: make a func. used in several places.
1824 uint32_t flags;
1825- if (settings.inline_ascii_response) {
1826- flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10);
1827- } else if (it->nsuffix > 0) {
1828- flags = *((uint32_t *)ITEM_suffix(it));
1829- } else {
1830- flags = 0;
1831- }
1832-
1833+ FLAGS_CONV(settings.inline_ascii_response, it, flags);
1834 new_it = item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, it->nbytes);
1835 assert(new_it == NULL || (new_it->it_flags & ITEM_CHUNKED));
1836 chunked = true;
1837@@ -3702,13 +3735,12 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
1838 if (chunked) {
1839 unsigned int ciovcnt = 1;
1840 size_t remain = new_it->nbytes;
1841- item_chunk *chunk = (item_chunk *) ITEM_data(new_it);
1842+ item_chunk *chunk = (item_chunk *) ITEM_schunk(new_it);
1843 io->io.iov = &c->iov[c->iovused];
1844 // fill the header so we can get the full data + crc back.
1845 add_iov(c, new_it, ITEM_ntotal(new_it) - new_it->nbytes);
1846 while (remain > 0) {
1847 chunk = do_item_alloc_chunk(chunk, remain);
1848- // TODO: counter bump
1849 if (chunk == NULL) {
1850 item_remove(new_it);
1851 do_cache_free(c->thread->io_cache, io);
1852@@ -3745,9 +3777,15 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
1853 io->io.data = (void *)io;
1854
1855 // Now, fill in io->io based on what was in our header.
1856+#ifdef NEED_ALIGN
1857+ io->io.page_version = hdr.page_version;
1858+ io->io.page_id = hdr.page_id;
1859+ io->io.offset = hdr.offset;
1860+#else
1861 io->io.page_version = hdr->page_version;
1862 io->io.page_id = hdr->page_id;
1863 io->io.offset = hdr->offset;
1864+#endif
1865 io->io.len = ntotal;
1866 io->io.mode = OBJ_IO_READ;
1867 io->io.cb = _get_extstore_cb;
1868@@ -3763,8 +3801,6 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
1869 return 0;
1870 }
1871 #endif
1872-// FIXME: the 'breaks' around memory malloc's should break all the way down,
1873-// fill ileft/suffixleft, then run conn_releaseitems()
1874 /* ntokens is overwritten here... shrug.. */
1875 static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, bool return_cas, bool should_touch) {
1876 char *key;
1877@@ -3776,6 +3812,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1878 char *suffix;
1879 int32_t exptime_int = 0;
1880 rel_time_t exptime = 0;
1881+ bool fail_length = false;
1882 assert(c != NULL);
1883
1884 if (should_touch) {
1885@@ -3795,14 +3832,8 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1886 nkey = key_token->length;
1887
1888 if (nkey > KEY_MAX_LENGTH) {
1889- out_string(c, "CLIENT_ERROR bad command line format");
1890- while (i-- > 0) {
1891- item_remove(*(c->ilist + i));
1892- if (return_cas || !settings.inline_ascii_response) {
1893- do_cache_free(c->thread->suffix_cache, *(c->suffixlist + i));
1894- }
1895- }
1896- return;
1897+ fail_length = true;
1898+ goto stop;
1899 }
1900
1901 it = limited_get(key, nkey, c, exptime, should_touch);
1902@@ -3812,7 +3843,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1903 if (it) {
1904 if (_ascii_get_expand_ilist(c, i) != 0) {
1905 item_remove(it);
1906- break; // FIXME: Should bail down to error.
1907+ goto stop;
1908 }
1909
1910 /*
1911@@ -3831,7 +3862,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1912 suffix = _ascii_get_suffix_buf(c, si);
1913 if (suffix == NULL) {
1914 item_remove(it);
1915- break;
1916+ goto stop;
1917 }
1918 si++;
1919 nbytes = it->nbytes;
1920@@ -3842,13 +3873,17 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1921 add_iov(c, suffix, suffix_len) != 0)
1922 {
1923 item_remove(it);
1924- break;
1925+ goto stop;
1926 }
1927 #ifdef EXTSTORE
1928 if (it->it_flags & ITEM_HDR) {
1929 if (_get_extstore(c, it, c->iovused-3, 4) != 0) {
1930+ pthread_mutex_lock(&c->thread->stats.mutex);
1931+ c->thread->stats.get_oom_extstore++;
1932+ pthread_mutex_unlock(&c->thread->stats.mutex);
1933+
1934 item_remove(it);
1935- break;
1936+ goto stop;
1937 }
1938 } else if ((it->it_flags & ITEM_CHUNKED) == 0) {
1939 #else
1940@@ -3857,7 +3892,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1941 add_iov(c, ITEM_data(it), it->nbytes);
1942 } else if (add_chunked_item_iovs(c, it, it->nbytes) != 0) {
1943 item_remove(it);
1944- break;
1945+ goto stop;
1946 }
1947 }
1948 else
1949@@ -3868,19 +3903,19 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1950 add_iov(c, ITEM_key(it), it->nkey) != 0)
1951 {
1952 item_remove(it);
1953- break;
1954+ goto stop;
1955 }
1956 if ((it->it_flags & ITEM_CHUNKED) == 0)
1957 {
1958 if (add_iov(c, ITEM_suffix(it), it->nsuffix + it->nbytes) != 0)
1959 {
1960 item_remove(it);
1961- break;
1962+ goto stop;
1963 }
1964 } else if (add_iov(c, ITEM_suffix(it), it->nsuffix) != 0 ||
1965 add_chunked_item_iovs(c, it, it->nbytes) != 0) {
1966 item_remove(it);
1967- break;
1968+ goto stop;
1969 }
1970 }
1971
1972@@ -3940,6 +3975,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1973 }
1974
1975 } while(key_token->value != NULL);
1976+stop:
1977
1978 c->icurr = c->ilist;
1979 c->ileft = i;
1980@@ -3958,7 +3994,12 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
1981 */
1982 if (key_token->value != NULL || add_iov(c, "END\r\n", 5) != 0
1983 || (IS_UDP(c->transport) && build_udp_headers(c) != 0)) {
1984- out_of_memory(c, "SERVER_ERROR out of memory writing get response");
1985+ if (fail_length) {
1986+ out_string(c, "CLIENT_ERROR bad command line format");
1987+ } else {
1988+ out_of_memory(c, "SERVER_ERROR out of memory writing get response");
1989+ }
1990+ conn_release_items(c);
1991 }
1992 else {
1993 conn_set_state(c, conn_mwrite);
1994@@ -4055,7 +4096,15 @@ static void process_update_command(conn *c, token_t *tokens, const size_t ntoken
1995 ITEM_set_cas(it, req_cas_id);
1996
1997 c->item = it;
1998+#ifdef NEED_ALIGN
1999+ if (it->it_flags & ITEM_CHUNKED) {
2000+ c->ritem = ITEM_schunk(it);
2001+ } else {
2002+ c->ritem = ITEM_data(it);
2003+ }
2004+#else
2005 c->ritem = ITEM_data(it);
2006+#endif
2007 c->rlbytes = it->nbytes;
2008 c->cmd = comm;
2009 conn_set_state(c, conn_nread);
2010@@ -4239,13 +4288,7 @@ enum delta_result_type do_add_delta(conn *c, const char *key, const size_t nkey,
2011 } else if (it->refcount > 1) {
2012 item *new_it;
2013 uint32_t flags;
2014- if (settings.inline_ascii_response) {
2015- flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10);
2016- } else if (it->nsuffix > 0) {
2017- flags = *((uint32_t *)ITEM_suffix(it));
2018- } else {
2019- flags = 0;
2020- }
2021+ FLAGS_CONV(settings.inline_ascii_response, it, flags);
2022 new_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, res + 2);
2023 if (new_it == 0) {
2024 do_item_remove(it);
2025@@ -5274,7 +5317,6 @@ static int read_into_chunked_item(conn *c) {
2026
2027 while (c->rlbytes > 0) {
2028 item_chunk *ch = (item_chunk *)c->ritem;
2029- assert(ch->used <= ch->size);
2030 if (ch->size == ch->used) {
2031 // FIXME: ch->next is currently always 0. remove this?
2032 if (ch->next) {
2033@@ -6156,7 +6198,7 @@ static void clock_handler(const int fd, const short which, void *arg) {
2034 static void usage(void) {
2035 printf(PACKAGE " " VERSION "\n");
2036 printf("-p, --port=<num> TCP port to listen on (default: 11211)\n"
2037- "-U, --udp-port=<num> UDP port to listen on (default: 11211, 0 is off)\n"
2038+ "-U, --udp-port=<num> UDP port to listen on (default: 0, off)\n"
2039 "-s, --unix-socket=<file> UNIX socket to listen on (disables network support)\n"
2040 "-A, --enable-shutdown enable ascii \"shutdown\" command\n"
2041 "-a, --unix-mask=<mask> access mask for UNIX socket, in octal (default: 0700)\n"
2042@@ -6236,7 +6278,8 @@ static void usage(void) {
2043 " currently: nothing\n"
2044 " - no_modern: uses defaults of previous major version (1.4.x)\n"
2045 #ifdef HAVE_DROP_PRIVILEGES
2046- " - no_drop_privileges: Disable drop_privileges in case it causes issues with\n"
2047+ " - drop_privileges: enable dropping extra syscall privileges\n"
2048+ " - no_drop_privileges: disable drop_privileges in case it causes issues with\n"
2049 " some customisation.\n"
2050 #ifdef MEMCACHED_DEBUG
2051 " - relaxed_privileges: Running tests requires extra privileges.\n"
2052@@ -6244,6 +6287,7 @@ static void usage(void) {
2053 #endif
2054 #ifdef EXTSTORE
2055 " - ext_path: file to write to for external storage.\n"
2056+ " ie: ext_path=/mnt/d1/extstore:1G\n"
2057 " - ext_page_size: size in megabytes of storage pages.\n"
2058 " - ext_wbuf_size: size in megabytes of page write buffers.\n"
2059 " - ext_threads: number of IO threads to run.\n"
2060@@ -6255,6 +6299,7 @@ static void usage(void) {
2061 " - ext_compact_under: compact when fewer than this many free pages\n"
2062 " - ext_drop_under: drop COLD items when fewer than this many free pages\n"
2063 " - ext_max_frag: max page fragmentation to tolerage\n"
2064+ " - slab_automove_freeratio: ratio of memory to hold free as buffer.\n"
2065 " (see doc/storage.txt for more info)\n"
2066 #endif
2067 );
2068@@ -6436,6 +6481,16 @@ static int enable_large_pages(void) {
2069 }
2070
2071 return ret;
2072+#elif defined(__linux__) && defined(MADV_HUGEPAGE)
2073+ /* check if transparent hugepages is compiled into the kernel */
2074+ struct stat st;
2075+ int ret = stat("/sys/kernel/mm/transparent_hugepage/enabled", &st);
2076+ if (ret || !(st.st_mode & S_IFREG)) {
2077+ fprintf(stderr, "Transparent huge pages support not detected.\n");
2078+ fprintf(stderr, "Will use default page size.\n");
2079+ return -1;
2080+ }
2081+ return 0;
2082 #else
2083 return -1;
2084 #endif
2085@@ -6534,7 +6589,7 @@ int main (int argc, char **argv) {
2086 bool slab_chunk_size_changed = false;
2087 #ifdef EXTSTORE
2088 void *storage = NULL;
2089- char *storage_file = NULL;
2090+ struct extstore_conf_file *storage_file = NULL;
2091 struct extstore_conf ext_cf;
2092 #endif
2093 char *subopts, *subopts_orig;
2094@@ -6575,12 +6630,12 @@ int main (int argc, char **argv) {
2095 NO_LRU_CRAWLER,
2096 NO_LRU_MAINTAINER,
2097 NO_DROP_PRIVILEGES,
2098+ DROP_PRIVILEGES,
2099 #ifdef MEMCACHED_DEBUG
2100 RELAXED_PRIVILEGES,
2101 #endif
2102 #ifdef EXTSTORE
2103 EXT_PAGE_SIZE,
2104- EXT_PAGE_COUNT,
2105 EXT_WBUF_SIZE,
2106 EXT_THREADS,
2107 EXT_IO_DEPTH,
2108@@ -6632,12 +6687,12 @@ int main (int argc, char **argv) {
2109 [NO_LRU_CRAWLER] = "no_lru_crawler",
2110 [NO_LRU_MAINTAINER] = "no_lru_maintainer",
2111 [NO_DROP_PRIVILEGES] = "no_drop_privileges",
2112+ [DROP_PRIVILEGES] = "drop_privileges",
2113 #ifdef MEMCACHED_DEBUG
2114 [RELAXED_PRIVILEGES] = "relaxed_privileges",
2115 #endif
2116 #ifdef EXTSTORE
2117 [EXT_PAGE_SIZE] = "ext_page_size",
2118- [EXT_PAGE_COUNT] = "ext_page_count",
2119 [EXT_WBUF_SIZE] = "ext_wbuf_size",
2120 [EXT_THREADS] = "ext_threads",
2121 [EXT_IO_DEPTH] = "ext_io_depth",
2122@@ -6677,7 +6732,6 @@ int main (int argc, char **argv) {
2123 settings.ext_drop_under = 0;
2124 settings.slab_automove_freeratio = 0.01;
2125 ext_cf.page_size = 1024 * 1024 * 64;
2126- ext_cf.page_count = 64;
2127 ext_cf.wbuf_size = settings.ext_wbuf_size;
2128 ext_cf.io_threadcount = 1;
2129 ext_cf.io_depth = 1;
2130@@ -7203,16 +7257,6 @@ int main (int argc, char **argv) {
2131 }
2132 ext_cf.page_size *= 1024 * 1024; /* megabytes */
2133 break;
2134- case EXT_PAGE_COUNT:
2135- if (subopts_value == NULL) {
2136- fprintf(stderr, "Missing ext_page_count argument\n");
2137- return 1;
2138- }
2139- if (!safe_strtoul(subopts_value, &ext_cf.page_count)) {
2140- fprintf(stderr, "could not parse argument to ext_page_count\n");
2141- return 1;
2142- }
2143- break;
2144 case EXT_WBUF_SIZE:
2145 if (subopts_value == NULL) {
2146 fprintf(stderr, "Missing ext_wbuf_size argument\n");
2147@@ -7329,7 +7373,20 @@ int main (int argc, char **argv) {
2148 settings.ext_drop_unread = true;
2149 break;
2150 case EXT_PATH:
2151- storage_file = strdup(subopts_value);
2152+ if (subopts_value) {
2153+ struct extstore_conf_file *tmp = storage_conf_parse(subopts_value, ext_cf.page_size);
2154+ if (tmp == NULL) {
2155+ fprintf(stderr, "failed to parse ext_path argument\n");
2156+ return 1;
2157+ }
2158+ if (storage_file != NULL) {
2159+ tmp->next = storage_file;
2160+ }
2161+ storage_file = tmp;
2162+ } else {
2163+ fprintf(stderr, "missing argument to ext_path, ie: ext_path=/d/file:5G\n");
2164+ return 1;
2165+ }
2166 break;
2167 #endif
2168 case MODERN:
2169@@ -7351,6 +7408,9 @@ int main (int argc, char **argv) {
2170 case NO_DROP_PRIVILEGES:
2171 settings.drop_privileges = false;
2172 break;
2173+ case DROP_PRIVILEGES:
2174+ settings.drop_privileges = true;
2175+ break;
2176 #ifdef MEMCACHED_DEBUG
2177 case RELAXED_PRIVILEGES:
2178 settings.relaxed_privileges = true;
2179@@ -7536,6 +7596,10 @@ int main (int argc, char **argv) {
2180 fprintf(stderr, "can't find the user %s to switch to\n", username);
2181 exit(EX_NOUSER);
2182 }
2183+ if (setgroups(0, NULL) < 0) {
2184+ fprintf(stderr, "failed to drop supplementary groups\n");
2185+ exit(EX_OSERR);
2186+ }
2187 if (setgid(pw->pw_gid) < 0 || setuid(pw->pw_uid) < 0) {
2188 fprintf(stderr, "failed to assume identity of user %s\n", username);
2189 exit(EX_OSERR);
2190@@ -7596,9 +7660,9 @@ int main (int argc, char **argv) {
2191 if (storage_file) {
2192 enum extstore_res eres;
2193 if (settings.ext_compact_under == 0) {
2194- settings.ext_compact_under = ext_cf.page_count / 4;
2195+ settings.ext_compact_under = storage_file->page_count / 4;
2196 /* Only rescues non-COLD items if below this threshold */
2197- settings.ext_drop_under = ext_cf.page_count / 4;
2198+ settings.ext_drop_under = storage_file->page_count / 4;
2199 }
2200 crc32c_init();
2201 /* Init free chunks to zero. */
2202@@ -7649,6 +7713,10 @@ int main (int argc, char **argv) {
2203 fprintf(stderr, "Failed to start storage compaction thread\n");
2204 exit(EXIT_FAILURE);
2205 }
2206+ if (storage && start_storage_write_thread(storage) != 0) {
2207+ fprintf(stderr, "Failed to start storage writer thread\n");
2208+ exit(EXIT_FAILURE);
2209+ }
2210
2211 if (start_lru_maintainer && start_lru_maintainer_thread(storage) != 0) {
2212 #else
2213diff --git a/memcached.h b/memcached.h
2214index 22893f5..a8dd59d 100644
2215--- a/memcached.h
2216+++ b/memcached.h
2217@@ -18,6 +18,7 @@
2218 #include <pthread.h>
2219 #include <unistd.h>
2220 #include <assert.h>
2221+#include <grp.h>
2222
2223 #include "itoa_ljust.h"
2224 #include "protocol_binary.h"
2225@@ -140,6 +141,17 @@
2226 #define APPEND_NUM_STAT(num, name, fmt, val) \
2227 APPEND_NUM_FMT_STAT("%d:%s", num, name, fmt, val)
2228
2229+/** Item client flag conversion */
2230+#define FLAGS_CONV(iar, it, flag) { \
2231+ if ((iar)) { \
2232+ flag = (uint32_t) strtoul(ITEM_suffix((it)), (char **) NULL, 10); \
2233+ } else if ((it)->nsuffix > 0) { \
2234+ flag = *((uint32_t *)ITEM_suffix((it))); \
2235+ } else { \
2236+ flag = 0; \
2237+ } \
2238+}
2239+
2240 /**
2241 * Callback for any function producing stats.
2242 *
2243@@ -274,6 +286,8 @@ struct slab_stats {
2244 #ifdef EXTSTORE
2245 #define EXTSTORE_THREAD_STATS_FIELDS \
2246 X(get_extstore) \
2247+ X(get_aborted_extstore) \
2248+ X(get_oom_extstore) \
2249 X(recache_from_extstore) \
2250 X(miss_from_extstore) \
2251 X(badcrc_from_extstore)
2252@@ -510,6 +524,23 @@ typedef struct _strchunk {
2253 uint8_t slabs_clsid; /* Same as above. */
2254 char data[];
2255 } item_chunk;
2256+
2257+#ifdef NEED_ALIGN
2258+static inline char *ITEM_schunk(item *it) {
2259+ int offset = it->nkey + 1 + it->nsuffix
2260+ + ((it->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0);
2261+ int remain = offset % 8;
2262+ if (remain != 0) {
2263+ offset += 8 - remain;
2264+ }
2265+ return ((char *) &(it->data)) + offset;
2266+}
2267+#else
2268+#define ITEM_schunk(item) ((char*) &((item)->data) + (item)->nkey + 1 \
2269+ + (item)->nsuffix \
2270+ + (((item)->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0))
2271+#endif
2272+
2273 #ifdef EXTSTORE
2274 typedef struct {
2275 unsigned int page_version; /* from IO header */
2276@@ -545,7 +576,7 @@ typedef struct _io_wrap {
2277 unsigned int iovec_data; /* specific index of data iovec */
2278 bool miss; /* signal a miss to unlink hdr_it */
2279 bool badcrc; /* signal a crc failure */
2280- bool active; // FIXME: canary for test. remove
2281+ bool active; /* tells if IO was dispatched or not */
2282 } io_wrap;
2283 #endif
2284 /**
2285@@ -554,6 +585,7 @@ typedef struct _io_wrap {
2286 struct conn {
2287 int sfd;
2288 sasl_conn_t *sasl_conn;
2289+ bool sasl_started;
2290 bool authenticated;
2291 enum conn_states state;
2292 enum bin_substates substate;
2293@@ -713,7 +745,7 @@ void sidethread_conn_close(conn *c);
2294
2295 /* Lock wrappers for cache functions that are called from main loop. */
2296 enum delta_result_type add_delta(conn *c, const char *key,
2297- const size_t nkey, const int incr,
2298+ const size_t nkey, bool incr,
2299 const int64_t delta, char *buf,
2300 uint64_t *cas);
2301 void accept_new_conns(const bool do_accept);
2302diff --git a/memcached.spec b/memcached.spec
2303index 697c38f..295b7d9 100644
2304--- a/memcached.spec
2305+++ b/memcached.spec
2306@@ -17,7 +17,7 @@ BuildRequires: systemd-units
2307 %endif
2308
2309 Name: memcached
2310-Version: 1.5.6
2311+Version: 1.5.10
2312 Release: 1%{?dist}
2313 Summary: High Performance, Distributed Memory Object Cache
2314
2315@@ -79,9 +79,9 @@ install -Dp -m0755 scripts/memcached.service %{buildroot}%{_unitdir}/%{name}.ser
2316 install -Dp -m0755 scripts/memcached@.service %{buildroot}%{_unitdir}/%{name}@.service
2317
2318 if [ %{safer_systemd} -gt 0 ]; then
2319- sed -e -i 's/^##safer##//g' %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service
2320+ sed -e 's/^##safer##//g' -i %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service
2321 else
2322- sed -e -i 's/^##safer##/#/g' %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service
2323+ sed -e 's/^##safer##/#/g' -i %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service
2324 fi
2325 %else
2326 install -Dp -m0755 scripts/memcached.sysv %{buildroot}%{_initrddir}/%{name}
2327@@ -187,7 +187,7 @@ exit 0
2328 - above suggestions from Bernard Johnson
2329
2330 * Mon May 7 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-2
2331-- Tidyness improvements suggested by Ruben Kerkhof in bugzilla #238994
2332+- Tidiness improvements suggested by Ruben Kerkhof in bugzilla #238994
2333
2334 * Fri May 4 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-1
2335 - Initial spec file created via rpmdev-newspec
2336diff --git a/sasl_defs.c b/sasl_defs.c
2337index 93a7b8b..c60d1bf 100644
2338--- a/sasl_defs.c
2339+++ b/sasl_defs.c
2340@@ -7,7 +7,7 @@
2341
2342 char my_sasl_hostname[1025];
2343
2344-#ifdef HAVE_SASL_CB_GETCONF
2345+#if defined(HAVE_SASL_CB_GETCONF) || defined(HAVE_SASL_CB_GETCONFPATH)
2346 /* The locations we may search for a SASL config file if the user didn't
2347 * specify one in the environment variable SASL_CONF_PATH
2348 */
2349@@ -82,7 +82,7 @@ static int sasl_server_userdb_checkpass(sasl_conn_t *conn,
2350 }
2351 #endif
2352
2353-#ifdef HAVE_SASL_CB_GETCONF
2354+#if defined(HAVE_SASL_CB_GETCONF) || defined(HAVE_SASL_CB_GETCONFPATH)
2355 static int sasl_getconf(void *context, const char **path)
2356 {
2357 *path = getenv("SASL_CONF_PATH");
2358@@ -152,6 +152,10 @@ static sasl_callback_t sasl_callbacks[] = {
2359
2360 #ifdef HAVE_SASL_CB_GETCONF
2361 { SASL_CB_GETCONF, sasl_getconf, NULL },
2362+#else
2363+#ifdef HAVE_SASL_CB_GETCONFPATH
2364+ { SASL_CB_GETCONFPATH, (sasl_callback_ft)sasl_getconf, NULL },
2365+#endif
2366 #endif
2367
2368 { SASL_CB_LIST_END, NULL, NULL }
2369diff --git a/scripts/memcached-automove b/scripts/memcached-automove
2370index f881129..4a3d3dd 100755
2371--- a/scripts/memcached-automove
2372+++ b/scripts/memcached-automove
2373@@ -54,7 +54,7 @@ def determine_move(history, diffs, totals):
2374 - use age as average over window. smooths over items falling out of WARM.
2375 also gives a little weight: if still evicting, a few more pages than
2376 necessary may be moved, pulling the classes closer together. Hopefully
2377- avoidnig ping-ponging.
2378+ avoiding ping-ponging.
2379 """
2380 # rotate windows
2381 history['w'].append({})
2382diff --git a/scripts/memcached-automove-extstore b/scripts/memcached-automove-extstore
2383index db66d8c..2bb0e35 100755
2384--- a/scripts/memcached-automove-extstore
2385+++ b/scripts/memcached-automove-extstore
2386@@ -64,7 +64,7 @@ def determine_move(history, stats, diffs, memfree):
2387 - if global pool is below minimum remove pages from oldest large class.
2388 - if global pool is above maximum, move pages to youngest large class.
2389 - extstore manages a desired number of free chunks in each slab class.
2390- - autmover adjusts above limits once per minute based on current sizes.
2391+ - automover adjusts above limits once per minute based on current sizes.
2392 - if youngest is below the age ratio limit of oldest, move a page to it.
2393 """
2394 # rotate windows
2395@@ -85,7 +85,6 @@ def determine_move(history, stats, diffs, memfree):
2396 print("global pool: [{}]".format(stats['slab_global_page_pool']))
2397
2398 pool_low = window_key_check(history, 'slab_pool_low')
2399- pool_high = window_key_check(history, 'slab_pool_high')
2400 for sid, slab in diffs.items():
2401 small_slab = False
2402 free_enough = False
2403diff --git a/scripts/memcached-tool b/scripts/memcached-tool
2404index 79f3770..cab36a5 100755
2405--- a/scripts/memcached-tool
2406+++ b/scripts/memcached-tool
2407@@ -20,6 +20,7 @@ use IO::Socket::INET;
2408 my $addr = shift;
2409 my $mode = shift || "display";
2410 my ($from, $to);
2411+my $limit;
2412
2413 if ($mode eq "display") {
2414 undef $mode if @ARGV;
2415@@ -30,7 +31,11 @@ if ($mode eq "display") {
2416 undef $mode if $to < 6 || $to > 17;
2417 print STDERR "ERROR: parameters out of range\n\n" unless $mode;
2418 } elsif ($mode eq 'dump') {
2419- ;
2420+ if (@ARGV) {
2421+ $limit = shift;
2422+ undef $mode if $limit < 1;
2423+ print STDERR "ERROR: invalid limit (should be a positive number)\n\n" unless $mode;
2424+ }
2425 } elsif ($mode eq 'stats') {
2426 ;
2427 } elsif ($mode eq 'settings') {
2428@@ -45,12 +50,12 @@ undef $mode if @ARGV;
2429
2430 die
2431 "Usage: memcached-tool <host[:port] | /path/to/socket> [mode]\n
2432- memcached-tool 10.0.0.5:11211 display # shows slabs
2433- memcached-tool 10.0.0.5:11211 # same. (default is display)
2434- memcached-tool 10.0.0.5:11211 stats # shows general stats
2435- memcached-tool 10.0.0.5:11211 settings # shows settings stats
2436- memcached-tool 10.0.0.5:11211 sizes # shows sizes stats
2437- memcached-tool 10.0.0.5:11211 dump # dumps keys and values
2438+ memcached-tool 10.0.0.5:11211 display # shows slabs
2439+ memcached-tool 10.0.0.5:11211 # same. (default is display)
2440+ memcached-tool 10.0.0.5:11211 stats # shows general stats
2441+ memcached-tool 10.0.0.5:11211 settings # shows settings stats
2442+ memcached-tool 10.0.0.5:11211 sizes # shows sizes stats
2443+ memcached-tool 10.0.0.5:11211 dump [limit] # dumps keys and values
2444
2445 WARNING! sizes is a development command.
2446 As of 1.4 it is still the only command which will lock your memcached instance for some time.
2447@@ -60,65 +65,74 @@ or at least speed it up.
2448 " unless $addr && $mode;
2449
2450
2451-my $sock;
2452-if ($addr =~ m:/:) {
2453- $sock = IO::Socket::UNIX->new(
2454- Peer => $addr,
2455- );
2456-}
2457-else {
2458- $addr .= ':11211' unless $addr =~ /:\d+$/;
2459+sub server_connect {
2460+ my $sock;
2461+ if ($addr =~ m:/:) {
2462+ $sock = IO::Socket::UNIX->new(
2463+ Peer => $addr,
2464+ );
2465+ }
2466+ else {
2467+ $addr .= ':11211' unless $addr =~ /:\d+$/;
2468
2469- $sock = IO::Socket::INET->new(
2470- PeerAddr => $addr,
2471- Proto => 'tcp',
2472- );
2473+ $sock = IO::Socket::INET->new(
2474+ PeerAddr => $addr,
2475+ Proto => 'tcp',
2476+ );
2477+ }
2478+ die "Couldn't connect to $addr\n" unless $sock;
2479+ return $sock;
2480 }
2481-die "Couldn't connect to $addr\n" unless $sock;
2482
2483-if ($mode eq 'dump') {
2484- my %items;
2485- my $totalitems;
2486-
2487- print $sock "stats items\r\n";
2488+my $sock = server_connect();
2489
2490+if ($mode eq 'dump') {
2491+ print STDERR "Dumping memcache contents";
2492+ print STDERR " (limiting to $limit keys)" unless !$limit;
2493+ print STDERR "\n";
2494+ print $sock "lru_crawler metadump all\r\n";
2495+ my %keyexp;
2496+ my $keycount = 0;
2497 while (<$sock>) {
2498- last if /^END/;
2499- if (/^STAT items:(\d*):number (\d*)/) {
2500- $items{$1} = $2;
2501- $totalitems += $2;
2502- }
2503- }
2504- print STDERR "Dumping memcache contents\n";
2505- print STDERR " Number of buckets: " . scalar(keys(%items)) . "\n";
2506- print STDERR " Number of items : $totalitems\n";
2507-
2508- foreach my $bucket (sort(keys(%items))) {
2509- print STDERR "Dumping bucket $bucket - " . $items{$bucket} . " total items\n";
2510- print $sock "stats cachedump $bucket $items{$bucket}\r\n";
2511- my %keyexp;
2512- while (<$sock>) {
2513- last if /^END/;
2514- # return format looks like this
2515- # ITEM foo [6 b; 1176415152 s]
2516- if (/^ITEM (\S+) \[.* (\d+) s\]/) {
2517- $keyexp{$1} = $2;
2518+ last if /^END/ or ($limit and $keycount == $limit);
2519+ # return format looks like this
2520+ # key=foo exp=2147483647 la=1521046038 cas=717111 fetch=no cls=13 size=1232
2521+ if (/^key=(\S+) exp=(-?\d+) .*/) {
2522+ my $k = $1;
2523+ $k =~ s/%(.{2})/chr hex $1/eg;
2524+
2525+ if ($2 == -1) {
2526+ $keyexp{$k} = 0;
2527+ } else {
2528+ $keyexp{$k} = $2;
2529 }
2530 }
2531+ $keycount++;
2532+ }
2533
2534- foreach my $k (keys(%keyexp)) {
2535- print $sock "get $k\r\n";
2536- my $response = <$sock>;
2537- if ($response =~ /VALUE (\S+) (\d+) (\d+)/) {
2538- my $flags = $2;
2539- my $len = $3;
2540- my $val;
2541- read $sock, $val, $len;
2542- print "add $k $flags $keyexp{$k} $len\r\n$val\r\n";
2543- # get the END
2544- $_ = <$sock>;
2545- $_ = <$sock>;
2546- }
2547+ if ($limit) {
2548+ # Need to reopen the connection here to stop the metadump in
2549+ # case the key limit was reached.
2550+ #
2551+ # XXX: Once a limit on # of keys returned is introduced in
2552+ # `lru_crawler metadump`, this should be removed and the proper
2553+ # parameter passed in the query above.
2554+ close($sock);
2555+ $sock = server_connect();
2556+ }
2557+
2558+ foreach my $k (keys(%keyexp)) {
2559+ print $sock "get $k\r\n";
2560+ my $response = <$sock>;
2561+ if ($response =~ /VALUE (\S+) (\d+) (\d+)/) {
2562+ my $flags = $2;
2563+ my $len = $3;
2564+ my $val;
2565+ read $sock, $val, $len;
2566+ print "add $k $flags $keyexp{$k} $len\r\n$val\r\n";
2567+ # get the END
2568+ $_ = <$sock>;
2569+ $_ = <$sock>;
2570 }
2571 }
2572 exit;
2573diff --git a/scripts/memcached-tool.1 b/scripts/memcached-tool.1
2574index 6bb021b..a863bd4 100644
2575--- a/scripts/memcached-tool.1
2576+++ b/scripts/memcached-tool.1
2577@@ -57,9 +57,12 @@ Number of times the underlying slab class was unable to store a new item.
2578 Print general-purpose statistics of the daemon. Each line contains the name of
2579 the statistic and its value.
2580 .TP
2581-.B dump
2582+.B dump [limit]
2583 Make a partial dump of the cache written in the add statements of the
2584-memcached protocol.
2585+memcached protocol. If
2586+.B limit
2587+is given and is a strictly positive
2588+integer, then the dump is limited to that number of items.
2589
2590 .SH SEE ALSO
2591 .BR memcached (1),
2592diff --git a/slabs.c b/slabs.c
2593index 200d575..2bcc350 100644
2594--- a/slabs.c
2595+++ b/slabs.c
2596@@ -8,6 +8,7 @@
2597 * memcached protocol.
2598 */
2599 #include "memcached.h"
2600+#include <sys/mman.h>
2601 #include <sys/stat.h>
2602 #include <sys/socket.h>
2603 #include <sys/resource.h>
2604@@ -98,6 +99,57 @@ unsigned int slabs_clsid(const size_t size) {
2605 return res;
2606 }
2607
2608+#if defined(__linux__) && defined(MADV_HUGEPAGE)
2609+/* Function split out for better error path handling */
2610+static void * alloc_large_chunk_linux(const size_t limit)
2611+{
2612+ size_t pagesize = 0;
2613+ void *ptr = NULL;
2614+ FILE *fp;
2615+ int ret;
2616+
2617+ /* Get the size of huge pages */
2618+ fp = fopen("/proc/meminfo", "r");
2619+ if (fp != NULL) {
2620+ char buf[64];
2621+
2622+ while ((fgets(buf, sizeof(buf), fp)))
2623+ if (!strncmp(buf, "Hugepagesize:", 13)) {
2624+ ret = sscanf(buf + 13, "%zu\n", &pagesize);
2625+
2626+ /* meminfo huge page size is in KiBs */
2627+ pagesize <<= 10;
2628+ }
2629+ fclose(fp);
2630+ }
2631+
2632+ if (!pagesize) {
2633+ fprintf(stderr, "Failed to get supported huge page size\n");
2634+ return NULL;
2635+ }
2636+
2637+ if (settings.verbose > 1)
2638+ fprintf(stderr, "huge page size: %zu\n", pagesize);
2639+
2640+ /* This works because glibc simply uses mmap when the alignment is
2641+ * above a certain limit. */
2642+ ret = posix_memalign(&ptr, pagesize, limit);
2643+ if (ret != 0) {
2644+ fprintf(stderr, "Failed to get aligned memory chunk: %d\n", ret);
2645+ return NULL;
2646+ }
2647+
2648+ ret = madvise(ptr, limit, MADV_HUGEPAGE);
2649+ if (ret < 0) {
2650+ fprintf(stderr, "Failed to set transparent hugepage hint: %d\n", ret);
2651+ free(ptr);
2652+ ptr = NULL;
2653+ }
2654+
2655+ return ptr;
2656+}
2657+#endif
2658+
2659 /**
2660 * Determines the chunk sizes and initializes the slab class descriptors
2661 * accordingly.
2662@@ -106,11 +158,24 @@ void slabs_init(const size_t limit, const double factor, const bool prealloc, co
2663 int i = POWER_SMALLEST - 1;
2664 unsigned int size = sizeof(item) + settings.chunk_size;
2665
2666+ /* Some platforms use runtime transparent hugepages. If for any reason
2667+ * the initial allocation fails, the required settings do not persist
2668+ * for remaining allocations. As such it makes little sense to do slab
2669+ * preallocation. */
2670+ bool __attribute__ ((unused)) do_slab_prealloc = false;
2671+
2672 mem_limit = limit;
2673
2674 if (prealloc) {
2675+#if defined(__linux__) && defined(MADV_HUGEPAGE)
2676+ mem_base = alloc_large_chunk_linux(mem_limit);
2677+ if (mem_base)
2678+ do_slab_prealloc = true;
2679+#else
2680 /* Allocate everything in a big chunk with malloc */
2681 mem_base = malloc(mem_limit);
2682+ do_slab_prealloc = true;
2683+#endif
2684 if (mem_base != NULL) {
2685 mem_current = mem_base;
2686 mem_avail = mem_limit;
2687@@ -161,7 +226,7 @@ void slabs_init(const size_t limit, const double factor, const bool prealloc, co
2688
2689 }
2690
2691- if (prealloc) {
2692+ if (prealloc && do_slab_prealloc) {
2693 slabs_preallocate(power_largest);
2694 }
2695 }
2696@@ -315,7 +380,7 @@ static void *do_slabs_alloc(const size_t size, unsigned int id, uint64_t *total_
2697 }
2698
2699 static void do_slabs_free_chunked(item *it, const size_t size) {
2700- item_chunk *chunk = (item_chunk *) ITEM_data(it);
2701+ item_chunk *chunk = (item_chunk *) ITEM_schunk(it);
2702 slabclass_t *p;
2703
2704 it->it_flags = ITEM_SLABBED;
2705@@ -339,7 +404,15 @@ static void do_slabs_free_chunked(item *it, const size_t size) {
2706 p->slots = it;
2707 p->sl_curr++;
2708 // TODO: macro
2709+#ifdef NEED_ALIGN
2710+ int total = it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk);
2711+ if (total % 8 != 0) {
2712+ total += 8 - (total % 8);
2713+ }
2714+ p->requested -= total;
2715+#else
2716 p->requested -= it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk);
2717+#endif
2718 if (settings.use_cas) {
2719 p->requested -= sizeof(uint64_t);
2720 }
2721@@ -948,7 +1021,7 @@ static int slab_rebalance_move(void) {
2722 do_item_replace(it, new_it, hv);
2723 /* Need to walk the chunks and repoint head */
2724 if (new_it->it_flags & ITEM_CHUNKED) {
2725- item_chunk *fch = (item_chunk *) ITEM_data(new_it);
2726+ item_chunk *fch = (item_chunk *) ITEM_schunk(new_it);
2727 fch->next->prev = fch;
2728 while (fch) {
2729 fch->head = new_it;
2730diff --git a/storage.c b/storage.c
2731index 6e61cd8..45554cf 100644
2732--- a/storage.c
2733+++ b/storage.c
2734@@ -6,27 +6,18 @@
2735 #include <stdlib.h>
2736 #include <string.h>
2737 #include <limits.h>
2738+#include <ctype.h>
2739
2740 #define PAGE_BUCKET_DEFAULT 0
2741 #define PAGE_BUCKET_COMPACT 1
2742 #define PAGE_BUCKET_CHUNKED 2
2743 #define PAGE_BUCKET_LOWTTL 3
2744
2745-int lru_maintainer_store(void *storage, const int clsid) {
2746- //int i;
2747+/*** WRITE FLUSH THREAD ***/
2748+
2749+static int storage_write(void *storage, const int clsid, const int item_age) {
2750 int did_moves = 0;
2751- int item_age = settings.ext_item_age;
2752- bool mem_limit_reached = false;
2753- unsigned int chunks_free;
2754 struct lru_pull_tail_return it_info;
2755- // FIXME: need to directly ask the slabber how big a class is
2756- if (slabs_clsid(settings.ext_item_size) > clsid)
2757- return 0;
2758- chunks_free = slabs_available_chunks(clsid, &mem_limit_reached,
2759- NULL, NULL);
2760- // if we are low on chunks and no spare, push out early.
2761- if (chunks_free < settings.ext_free_memchunks[clsid] && mem_limit_reached)
2762- item_age = 0;
2763
2764 it_info.it = NULL;
2765 lru_pull_tail(clsid, COLD_LRU, 0, LRU_PULL_RETURN_ITEM, 0, &it_info);
2766@@ -42,14 +33,7 @@ int lru_maintainer_store(void *storage, const int clsid) {
2767 uint32_t flags;
2768 if ((it->it_flags & ITEM_HDR) == 0 &&
2769 (item_age == 0 || current_time - it->time > item_age)) {
2770- // FIXME: flag conversion again
2771- if (settings.inline_ascii_response) {
2772- flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10);
2773- } else if (it->nsuffix > 0) {
2774- flags = *((uint32_t *)ITEM_suffix(it));
2775- } else {
2776- flags = 0;
2777- }
2778+ FLAGS_CONV(settings.inline_ascii_response, it, flags);
2779 item *hdr_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, sizeof(item_hdr));
2780 /* Run the storage write understanding the start of the item is dirty.
2781 * We will fill it (time/exptime/etc) from the header item on read.
2782@@ -57,7 +41,7 @@ int lru_maintainer_store(void *storage, const int clsid) {
2783 if (hdr_it != NULL) {
2784 int bucket = (it->it_flags & ITEM_CHUNKED) ?
2785 PAGE_BUCKET_CHUNKED : PAGE_BUCKET_DEFAULT;
2786- // Compres soon to expire items into similar pages.
2787+ // Compress soon to expire items into similar pages.
2788 if (it->exptime - current_time < settings.ext_low_ttl) {
2789 bucket = PAGE_BUCKET_LOWTTL;
2790 }
2791@@ -67,7 +51,9 @@ int lru_maintainer_store(void *storage, const int clsid) {
2792 // NOTE: when the item is read back in, the slab mover
2793 // may see it. Important to have refcount>=2 or ~ITEM_LINKED
2794 assert(it->refcount >= 2);
2795- if (extstore_write_request(storage, bucket, &io) == 0) {
2796+ // NOTE: write bucket vs free page bucket will disambiguate once
2797+ // lowttl feature is better understood.
2798+ if (extstore_write_request(storage, bucket, bucket, &io) == 0) {
2799 // cuddle the hash value into the time field so we don't have
2800 // to recalculate it.
2801 item *buf_it = (item *) io.buf;
2802@@ -76,12 +62,12 @@ int lru_maintainer_store(void *storage, const int clsid) {
2803 // TODO: should be in items.c
2804 if (it->it_flags & ITEM_CHUNKED) {
2805 // Need to loop through the item and copy
2806- item_chunk *sch = (item_chunk *) ITEM_data(it);
2807+ item_chunk *sch = (item_chunk *) ITEM_schunk(it);
2808 int remain = orig_ntotal;
2809 int copied = 0;
2810 // copy original header
2811 int hdrtotal = ITEM_ntotal(it) - it->nbytes;
2812- memcpy((char *)io.buf+32, (char *)it+32, hdrtotal - 32);
2813+ memcpy((char *)io.buf+STORE_OFFSET, (char *)it+STORE_OFFSET, hdrtotal - STORE_OFFSET);
2814 copied = hdrtotal;
2815 // copy data in like it were one large object.
2816 while (sch && remain) {
2817@@ -93,11 +79,11 @@ int lru_maintainer_store(void *storage, const int clsid) {
2818 sch = sch->next;
2819 }
2820 } else {
2821- memcpy((char *)io.buf+32, (char *)it+32, io.len-32);
2822+ memcpy((char *)io.buf+STORE_OFFSET, (char *)it+STORE_OFFSET, io.len-STORE_OFFSET);
2823 }
2824 // crc what we copied so we can do it sequentially.
2825 buf_it->it_flags &= ~ITEM_LINKED;
2826- buf_it->exptime = crc32c(0, (char*)io.buf+32, orig_ntotal-32);
2827+ buf_it->exptime = crc32c(0, (char*)io.buf+STORE_OFFSET, orig_ntotal-STORE_OFFSET);
2828 extstore_write(storage, &io);
2829 item_hdr *hdr = (item_hdr *) ITEM_data(hdr_it);
2830 hdr->page_version = io.page_version;
2831@@ -125,6 +111,128 @@ int lru_maintainer_store(void *storage, const int clsid) {
2832 return did_moves;
2833 }
2834
2835+static pthread_t storage_write_tid;
2836+static pthread_mutex_t storage_write_plock;
2837+#define WRITE_SLEEP_MAX 1000000
2838+#define WRITE_SLEEP_MIN 500
2839+
2840+static void *storage_write_thread(void *arg) {
2841+ void *storage = arg;
2842+ // NOTE: ignoring overflow since that would take years of uptime in a
2843+ // specific load pattern of never going to sleep.
2844+ unsigned int backoff[MAX_NUMBER_OF_SLAB_CLASSES] = {0};
2845+ unsigned int counter = 0;
2846+ useconds_t to_sleep = WRITE_SLEEP_MIN;
2847+ logger *l = logger_create();
2848+ if (l == NULL) {
2849+ fprintf(stderr, "Failed to allocate logger for storage compaction thread\n");
2850+ abort();
2851+ }
2852+
2853+ pthread_mutex_lock(&storage_write_plock);
2854+
2855+ while (1) {
2856+ // cache per-loop to avoid calls to the slabs_clsid() search loop
2857+ int min_class = slabs_clsid(settings.ext_item_size);
2858+ bool do_sleep = true;
2859+ counter++;
2860+ if (to_sleep > WRITE_SLEEP_MAX)
2861+ to_sleep = WRITE_SLEEP_MAX;
2862+
2863+ for (int x = 0; x < MAX_NUMBER_OF_SLAB_CLASSES; x++) {
2864+ bool did_move = false;
2865+ bool mem_limit_reached = false;
2866+ unsigned int chunks_free;
2867+ int item_age;
2868+ int target = settings.ext_free_memchunks[x];
2869+ if (min_class > x || (backoff[x] && (counter % backoff[x] != 0))) {
2870+ // Long sleeps means we should retry classes sooner.
2871+ if (to_sleep > WRITE_SLEEP_MIN * 10)
2872+ backoff[x] /= 2;
2873+ continue;
2874+ }
2875+
2876+ // Avoid extra slab lock calls during heavy writing.
2877+ chunks_free = slabs_available_chunks(x, &mem_limit_reached,
2878+ NULL, NULL);
2879+
2880+ // storage_write() will fail and cut loop after filling write buffer.
2881+ while (1) {
2882+ // if we are low on chunks and no spare, push out early.
2883+ if (chunks_free < target && mem_limit_reached) {
2884+ item_age = 0;
2885+ } else {
2886+ item_age = settings.ext_item_age;
2887+ }
2888+ if (storage_write(storage, x, item_age)) {
2889+ chunks_free++; // Allow stopping if we've done enough this loop
2890+ did_move = true;
2891+ do_sleep = false;
2892+ if (to_sleep > WRITE_SLEEP_MIN)
2893+ to_sleep /= 2;
2894+ } else {
2895+ break;
2896+ }
2897+ }
2898+
2899+ if (!did_move) {
2900+ backoff[x]++;
2901+ } else if (backoff[x]) {
2902+ backoff[x] /= 2;
2903+ }
2904+ }
2905+
2906+ // flip lock so we can be paused or stopped
2907+ pthread_mutex_unlock(&storage_write_plock);
2908+ if (do_sleep) {
2909+ usleep(to_sleep);
2910+ to_sleep *= 2;
2911+ }
2912+ pthread_mutex_lock(&storage_write_plock);
2913+ }
2914+ return NULL;
2915+}
2916+
2917+// TODO
2918+// logger needs logger_destroy() to exist/work before this is safe.
2919+/*int stop_storage_write_thread(void) {
2920+ int ret;
2921+ pthread_mutex_lock(&lru_maintainer_lock);
2922+ do_run_lru_maintainer_thread = 0;
2923+ pthread_mutex_unlock(&lru_maintainer_lock);
2924+ // WAKEUP SIGNAL
2925+ if ((ret = pthread_join(lru_maintainer_tid, NULL)) != 0) {
2926+ fprintf(stderr, "Failed to stop LRU maintainer thread: %s\n", strerror(ret));
2927+ return -1;
2928+ }
2929+ settings.lru_maintainer_thread = false;
2930+ return 0;
2931+}*/
2932+
2933+void storage_write_pause(void) {
2934+ pthread_mutex_lock(&storage_write_plock);
2935+}
2936+
2937+void storage_write_resume(void) {
2938+ pthread_mutex_unlock(&storage_write_plock);
2939+}
2940+
2941+int start_storage_write_thread(void *arg) {
2942+ int ret;
2943+
2944+ pthread_mutex_init(&storage_write_plock, NULL);
2945+ if ((ret = pthread_create(&storage_write_tid, NULL,
2946+ storage_write_thread, arg)) != 0) {
2947+ fprintf(stderr, "Can't create storage_write thread: %s\n",
2948+ strerror(ret));
2949+ return -1;
2950+ }
2951+
2952+ return 0;
2953+}
2954+
2955+/*** COMPACTOR ***/
2956+
2957 /* Fetch stats from the external storage system and decide to compact.
2958 * If we're more than half full, start skewing how aggressively to run
2959 * compaction, up to a desired target when all pages are full.
2960@@ -260,7 +368,7 @@ static void storage_compact_readback(void *storage, logger *l,
2961 io.len = ntotal;
2962 io.mode = OBJ_IO_WRITE;
2963 for (tries = 10; tries > 0; tries--) {
2964- if (extstore_write_request(storage, PAGE_BUCKET_COMPACT, &io) == 0) {
2965+ if (extstore_write_request(storage, PAGE_BUCKET_COMPACT, PAGE_BUCKET_COMPACT, &io) == 0) {
2966 memcpy(io.buf, it, io.len);
2967 extstore_write(storage, &io);
2968 do_update = true;
2969@@ -456,4 +564,88 @@ int start_storage_compact_thread(void *arg) {
2970 return 0;
2971 }
2972
2973+/*** UTILITY ***/
2974+// /path/to/file:100G:bucket1
2975+// FIXME: Modifies argument. copy instead?
2976+struct extstore_conf_file *storage_conf_parse(char *arg, unsigned int page_size) {
2977+ struct extstore_conf_file *cf = NULL;
2978+ char *b = NULL;
2979+ char *p = strtok_r(arg, ":", &b);
2980+ char unit = 0;
2981+ uint64_t multiplier = 0;
2982+ int base_size = 0;
2983+ if (p == NULL)
2984+ goto error;
2985+ // First arg is the filepath.
2986+ cf = calloc(1, sizeof(struct extstore_conf_file));
2987+ cf->file = strdup(p);
2988+
2989+ p = strtok_r(NULL, ":", &b);
2990+ if (p == NULL) {
2991+ fprintf(stderr, "must supply size to ext_path, ie: ext_path=/f/e:64m (M|G|T|P supported)\n");
2992+ goto error;
2993+ }
2994+ unit = tolower(p[strlen(p)-1]);
2995+ p[strlen(p)-1] = '\0';
2996+ // sigh.
2997+ switch (unit) {
2998+ case 'm':
2999+ multiplier = 1024 * 1024;
3000+ break;
3001+ case 'g':
3002+ multiplier = 1024 * 1024 * 1024;
3003+ break;
3004+ case 't':
3005+ multiplier = 1024 * 1024;
3006+ multiplier *= 1024 * 1024;
3007+ break;
3008+ case 'p':
3009+ multiplier = 1024 * 1024;
3010+ multiplier *= 1024 * 1024 * 1024;
3011+ break;
3012+ }
3013+ base_size = atoi(p);
3014+ multiplier *= base_size;
3015+ // page_count is nearest-but-not-larger-than pages * psize
3016+ cf->page_count = multiplier / page_size;
3017+ assert(page_size * cf->page_count <= multiplier);
3018+
3019+ // final token would be a default free bucket
3020+ p = strtok_r(NULL, ",", &b);
3021+ // TODO: We reuse the original DEFINES for now,
3022+ // but if lowttl gets split up this needs to be its own set.
3023+ if (p != NULL) {
3024+ if (strcmp(p, "compact") == 0) {
3025+ cf->free_bucket = PAGE_BUCKET_COMPACT;
3026+ } else if (strcmp(p, "lowttl") == 0) {
3027+ cf->free_bucket = PAGE_BUCKET_LOWTTL;
3028+ } else if (strcmp(p, "chunked") == 0) {
3029+ cf->free_bucket = PAGE_BUCKET_CHUNKED;
3030+ } else if (strcmp(p, "default") == 0) {
3031+ cf->free_bucket = PAGE_BUCKET_DEFAULT;
3032+ } else {
3033+ fprintf(stderr, "Unknown extstore bucket: %s\n", p);
3034+ goto error;
3035+ }
3036+ } else {
3037+ // TODO: is this necessary?
3038+ cf->free_bucket = PAGE_BUCKET_DEFAULT;
3039+ }
3040+
3041+ // TODO: disabling until compact algorithm is improved.
3042+ if (cf->free_bucket != PAGE_BUCKET_DEFAULT) {
3043+ fprintf(stderr, "ext_path only presently supports the default bucket\n");
3044+ goto error;
3045+ }
3046+
3047+ return cf;
3048+error:
3049+ if (cf) {
3050+ if (cf->file)
3051+ free(cf->file);
3052+ free(cf);
3053+ }
3054+ return NULL;
3055+}
3056+
3057 #endif
3058diff --git a/storage.h b/storage.h
3059index 7af6cc0..60f499b 100644
3060--- a/storage.h
3061+++ b/storage.h
3062@@ -1,9 +1,15 @@
3063 #ifndef STORAGE_H
3064 #define STORAGE_H
3065
3066-int lru_maintainer_store(void *storage, const int clsid);
3067+int start_storage_write_thread(void *arg);
3068+void storage_write_pause(void);
3069+void storage_write_resume(void);
3070 int start_storage_compact_thread(void *arg);
3071 void storage_compact_pause(void);
3072 void storage_compact_resume(void);
3073+struct extstore_conf_file *storage_conf_parse(char *arg, unsigned int page_size);
3074+
3075+// Ignore pointers and header bits from the CRC
3076+#define STORE_OFFSET offsetof(item, nbytes)
3077
3078 #endif
3079diff --git a/t/binary-extstore.t b/t/binary-extstore.t
3080index 390e26a..676d13b 100755
3081--- a/t/binary-extstore.t
3082+++ b/t/binary-extstore.t
3083@@ -17,7 +17,7 @@ if (!supports_extstore()) {
3084
3085 $ext_path = "/tmp/extstore.$$";
3086
3087-my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,no_lru_crawler,slab_automove=0");
3088+my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,no_lru_crawler,slab_automove=0");
3089 ok($server, "started the server");
3090
3091 # Based almost 100% off testClient.py which is:
3092@@ -176,7 +176,15 @@ $set->('x', 10, 19, "somevalue");
3093 $set->("mfoo$_", 0, 19, $value);
3094 }
3095 sleep 4;
3096- $empty->('mfoo1');
3097+ # FIXME: Need to sample through a few values, or fix eviction to be
3098+ # more accurate. On 32bit systems some pages unused to this point get
3099+ # filled after the first few items, then the eviction algo pulls those
3100+ # pages since they have the lowest version number, leaving older objects
3101+ # in memory and evicting newer ones.
3102+ for (1 .. ($keycount*3)) {
3103+ next unless $_ % 100 == 0;
3104+ eval { $mc->get("mfoo$_"); };
3105+ }
3106
3107 my %s = $mc->stats('');
3108 cmp_ok($s{extstore_objects_evicted}, '>', 0);
3109diff --git a/t/binary-sasl.t b/t/binary-sasl.t
3110index 85ef069..e923584 100755
3111--- a/t/binary-sasl.t
3112+++ b/t/binary-sasl.t
3113@@ -13,7 +13,7 @@ use Test::More;
3114
3115 if (supports_sasl()) {
3116 if ($ENV{'RUN_SASL_TESTS'}) {
3117- plan tests => 33;
3118+ plan tests => 34;
3119 } else {
3120 plan skip_all => 'Skipping SASL tests';
3121 exit 0;
3122@@ -92,7 +92,7 @@ use constant RES_MAGIC => 0x81;
3123 my $pwd=getcwd;
3124 $ENV{'SASL_CONF_PATH'} = "$pwd/t/sasl";
3125
3126-my $server = new_memcached('-B binary -S ');
3127+my $server = new_memcached('-B binary -S -l 127.0.0.1 ');
3128
3129 my $mc = MC::Client->new;
3130
3131@@ -192,7 +192,9 @@ for my $dir (split(/:/, $ENV{PATH}),
3132 }
3133 }
3134
3135-system("echo testpass | $saslpasswd_path -a memcached -c -p testuser");
3136+my $sasl_realm = 'memcached.realm';
3137+
3138+system("echo testpass | $saslpasswd_path -a memcached -u $sasl_realm -c -p testuser");
3139
3140 $mc = MC::Client->new;
3141
3142@@ -261,6 +263,11 @@ $empty->('x', 'somevalue');
3143 cmp_ok($status,'==',ERR_AUTH_ERROR, "error code matches");
3144 }
3145
3146+{
3147+ my $mc = MC::Client->new;
3148+ is ($mc->sasl_step('testuser', 'testpass'), 0x20, "sasl_step_fails_no_segfault");
3149+}
3150+
3151 # check the SASL stats, make sure they track things correctly
3152 # note: the enabled or not is presence checked in stats.t
3153
3154@@ -273,8 +280,8 @@ $empty->('x', 'somevalue');
3155
3156 {
3157 my %stats = $mc->stats('');
3158- is ($stats{'auth_cmds'}, 5, "auth commands counted");
3159- is ($stats{'auth_errors'}, 3, "auth errors correct");
3160+ is ($stats{'auth_cmds'}, 6, "auth commands counted");
3161+ is ($stats{'auth_errors'}, 4, "auth errors correct");
3162 }
3163
3164
3165@@ -309,10 +316,17 @@ sub new {
3166 sub authenticate {
3167 my ($self, $user, $pass, $mech)= @_;
3168 $mech ||= 'PLAIN';
3169- my $buf = sprintf("%c%s%c%s", 0, $user, 0, $pass);
3170+ my $buf = sprintf("%c%s@%s%c%s", 0, $user, $sasl_realm, 0, $pass);
3171 my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_AUTH, $mech, $buf, '');
3172 return $status;
3173 }
3174+sub sasl_step {
3175+ my ($self, $user, $pass, $mech)= @_;
3176+ $mech ||= 'PLAIN';
3177+ my $buf = sprintf("%c%s@%s%c%s", 0, $user, $sasl_realm, 0, $pass);
3178+ my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_STEP, $mech, $buf, '');
3179+ return $status;
3180+}
3181 sub list_mechs {
3182 my ($self)= @_;
3183 my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_LIST_MECHS, '', '', '');
3184@@ -661,4 +675,3 @@ sub auth_error {
3185 unlink $sasldb;
3186
3187 # vim: filetype=perl
3188-
3189diff --git a/t/chunked-extstore.t b/t/chunked-extstore.t
3190index ebd0f0b..3515f21 100644
3191--- a/t/chunked-extstore.t
3192+++ b/t/chunked-extstore.t
3193@@ -18,9 +18,26 @@ if (!supports_extstore()) {
3194
3195 $ext_path = "/tmp/extstore.$$";
3196
3197-my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,slab_chunk_max=16384,slab_automove=0");
3198+my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_chunk_max=16384,slab_automove=0,ext_compact_under=1");
3199 my $sock = $server->sock;
3200
3201+# Wait until all items have flushed
3202+sub wait_for_ext {
3203+ my $sum = 1;
3204+ while ($sum != 0) {
3205+ my $s = mem_stats($sock, "items");
3206+ $sum = 0;
3207+ for my $key (keys %$s) {
3208+ if ($key =~ m/items:(\d+):number/) {
3209+ # Ignore classes which can contain extstore items
3210+ next if $1 < 3;
3211+ $sum += $s->{$key};
3212+ }
3213+ }
3214+ sleep 1 if $sum != 0;
3215+ }
3216+}
3217+
3218 # We're testing to ensure item chaining doesn't corrupt or poorly overlap
3219 # data, so create a non-repeating pattern.
3220 my @parts = ();
3221@@ -58,7 +75,7 @@ for (1..5) {
3222 print $sock "set toast$_ 0 0 $biglen\r\n$big\r\n";
3223 is(scalar <$sock>, "STORED\r\n", "stored big");
3224 }
3225- sleep 4;
3226+ wait_for_ext();
3227
3228 for (1..40) {
3229 mem_get_is($sock, "toast$_", $big);
3230@@ -76,17 +93,19 @@ for (1..5) {
3231
3232 # fill to eviction
3233 {
3234- my $keycount = 1000;
3235+ my $keycount = 1250;
3236 for (1 .. $keycount) {
3237 print $sock "set mfoo$_ 0 0 $plen noreply\r\n$pattern\r\n";
3238+ wait_for_ext() if $_ % 500 == 0;
3239 }
3240- sleep 6;
3241+ # because item_age is set to 2s.
3242+ wait_for_ext();
3243
3244 my $stats = mem_stats($sock);
3245 cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted');
3246 cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted');
3247 cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted');
3248- is($stats->{extstore_pages_free}, 1, '1 page is free');
3249+ cmp_ok($stats->{extstore_pages_free}, '<', 2, 'most pages are used');
3250 is($stats->{miss_from_extstore}, 0, 'no misses');
3251
3252 # original "pattern" key should be gone.
3253@@ -106,14 +125,18 @@ for (1..5) {
3254 print $sock "delete toast$_ noreply\r\n" if $_ % 2 == 0;
3255 }
3256
3257- for (1..1000) {
3258+ for (1..1250) {
3259 # Force a read so objects don't get skipped.
3260- mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;
3261+ print $sock "add mfoo$_ 0 0 1 noreply\r\n1\r\n" if $_ % 2 == 1;
3262 }
3263- for (1..1000) {
3264- # Force a read so objects don't get skipped.
3265+ for (1..1250) {
3266+ # Delete lots of objects to trigger compaction.
3267 print $sock "delete mfoo$_ noreply\r\n" if $_ % 2 == 0;
3268 }
3269+ print $sock "extstore compact_under 4\r\n";
3270+ my $res = <$sock>;
3271+ print $sock "extstore drop_under 3\r\n";
3272+ $res = <$sock>;
3273
3274 sleep 4;
3275
3276@@ -122,7 +145,7 @@ for (1..5) {
3277 cmp_ok($stats->{extstore_compact_rescues}, '>', 0, 'some compaction rescues happened');
3278
3279 # Some of the early items got evicted
3280- for (100..1000) {
3281+ for (750..1250) {
3282 # everything should validate properly.
3283 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;
3284 }
3285@@ -133,20 +156,12 @@ for (1..5) {
3286 print $sock "extstore recache_rate 1\r\n";
3287 is(scalar <$sock>, "OK\r\n", "upped recache rate");
3288
3289- for (800..1000) {
3290+ for (1150..1250) {
3291 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;
3292 }
3293
3294 my $stats = mem_stats($sock);
3295- cmp_ok($stats->{recache_from_extstore}, '>', 100, 'recaching happening');
3296-
3297- for (800..1000) {
3298- mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;
3299- }
3300-
3301- my $stats2 = mem_stats($sock);
3302- is($stats->{recache_from_extstore}, $stats2->{recache_from_extstore},
3303- 'values already recached');
3304+ cmp_ok($stats->{recache_from_extstore}, '>', 25, 'recaching happening');
3305 }
3306
3307 done_testing();
3308diff --git a/t/error-extstore.t b/t/error-extstore.t
3309new file mode 100644
3310index 0000000..6df1528
3311--- /dev/null
3312+++ b/t/error-extstore.t
3313@@ -0,0 +1,130 @@
3314+#!/usr/bin/perl
3315+# Test the "Error on get" path for extstore.
3316+# the entire error handling code for process_get_command() never worked, and
3317+# would infinite loop. get_extstore() can hit it sometimes.
3318+
3319+use strict;
3320+use warnings;
3321+
3322+use Test::More;
3323+use FindBin qw($Bin);
3324+use lib "$Bin/lib";
3325+use MemcachedTest;
3326+
3327+my $ext_path;
3328+
3329+if (!supports_extstore()) {
3330+ plan skip_all => 'extstore not enabled';
3331+ exit 0;
3332+}
3333+
3334+$ext_path = "/tmp/extstore.$$";
3335+
3336+my $server = new_memcached("-m 64 -I 4m -U 0 -o ext_page_size=8,ext_wbuf_size=8,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_automove=0,ext_compact_under=1");
3337+my $sock = $server->sock;
3338+
3339+# Wait until all items have flushed
3340+sub wait_for_ext {
3341+ my $sum = 1;
3342+ while ($sum != 0) {
3343+ my $s = mem_stats($sock, "items");
3344+ $sum = 0;
3345+ for my $key (keys %$s) {
3346+ if ($key =~ m/items:(\d+):number/) {
3347+ # Ignore classes which can contain extstore items
3348+ next if $1 < 3;
3349+ $sum += $s->{$key};
3350+ }
3351+ }
3352+ sleep 1 if $sum != 0;
3353+ }
3354+}
3355+
3356+# We're testing to ensure item chaining doesn't corrupt or poorly overlap
3357+# data, so create a non-repeating pattern.
3358+my @parts = ();
3359+for (1 .. 8000) {
3360+ push(@parts, $_);
3361+}
3362+my $pattern = join(':', @parts);
3363+my $plen = length($pattern);
3364+
3365+# Set some large items and let them flush to extstore.
3366+for (1..5) {
3367+ my $size = 3000 * 1024;
3368+ my $data = "x" x $size;
3369+ print $sock "set foo$_ 0 0 $size\r\n$data\r\n";
3370+ my $res = <$sock>;
3371+ is($res, "STORED\r\n", "stored some big items");
3372+}
3373+
3374+wait_for_ext();
3375+
3376+{
3377+ my $long_key = "f" x 512;
3378+ print $sock "get foo1 foo2 foo3 $long_key\r\n";
3379+ ok(scalar <$sock> =~ /CLIENT_ERROR bad command line format/, 'long key fails');
3380+ my $stats = mem_stats($sock);
3381+ cmp_ok($stats->{get_aborted_extstore}, '>', 1, 'some extstore queries aborted');
3382+}
3383+
3384+# Disable automatic page balancing, then move enough pages that the large
3385+# items can no longer be loaded from extstore
3386+{
3387+ print $sock "slabs automove 0\r\n";
3388+ my $res = <$sock>;
3389+ my $source = 0;
3390+ while (1) {
3391+ print $sock "slabs reassign $source 1\r\n";
3392+ $res = <$sock>;
3393+ if ($res =~ m/NOSPARE/) {
3394+ $source = -1;
3395+ my $stats = mem_stats($sock, 'slabs');
3396+ for my $key (grep { /total_pages/ } keys %$stats) {
3397+ if ($key =~ m/(\d+):total_pages/) {
3398+ next if $1 < 3;
3399+ $source = $1 if $stats->{$key} > 1;
3400+ }
3401+ }
3402+ last if $source == -1;
3403+ }
3404+ select undef, undef, undef, 0.10;
3405+ }
3406+}
3407+
3408+# fetching the large keys should now fail.
3409+{
3410+ print $sock "get foo1\r\n";
3411+ my $res = <$sock>;
3412+ $res =~ s/[\r\n]//g;
3413+ is($res, 'SERVER_ERROR out of memory writing get response', 'can no longer read back item');
3414+ my $stats = mem_stats($sock);
3415+ is($stats->{get_oom_extstore}, 1, 'check extstore oom counter');
3416+}
3417+
3418+# Leaving this for future generations.
3419+# The process_get_command() function had several memory leaks.
3420+my $LEAK_TEST = 0;
3421+if ($LEAK_TEST) {
3422+ my $tries = 0;
3423+ while ($tries) {
3424+ print $sock "slabs reassign 1 39\r\n";
3425+ my $res = <$sock>;
3426+ if ($res =~ m/BUSY/) {
3427+ select undef, undef, undef, 0.10;
3428+ } else {
3429+ $tries--;
3430+ }
3431+ }
3432+ my $long_key = "f" x 512;
3433+ while (1) {
3434+ print $sock "get foo1 foo2 foo3 $long_key\r\n";
3435+ my $res = <$sock>;
3436+ }
3437+}
3438+
3439+done_testing();
3440+
3441+END {
3442+ unlink $ext_path if $ext_path;
3443+}
3444diff --git a/t/extstore-buckets.t b/t/extstore-buckets.t
3445index a2c1c90..e3027ad 100644
3446--- a/t/extstore-buckets.t
3447+++ b/t/extstore-buckets.t
3448@@ -17,7 +17,7 @@ if (!supports_extstore()) {
3449
3450 $ext_path = "/tmp/extstore.$$";
3451
3452-my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0,ext_path=$ext_path,ext_low_ttl=60,slab_automove=1");
3453+my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0,ext_path=$ext_path:64m,ext_low_ttl=60,slab_automove=1");
3454 my $sock = $server->sock;
3455
3456 my $value;
3457diff --git a/t/extstore-jbod.t b/t/extstore-jbod.t
3458new file mode 100644
3459index 0000000..1618803
3460--- /dev/null
3461+++ b/t/extstore-jbod.t
3462@@ -0,0 +1,69 @@
3463+#!/usr/bin/perl
3464+
3465+use strict;
3466+use warnings;
3467+use Test::More;
3468+use FindBin qw($Bin);
3469+use lib "$Bin/lib";
3470+use MemcachedTest;
3471+use Data::Dumper qw/Dumper/;
3472+
3473+my $ext_path;
3474+my $ext_path2;
3475+
3476+if (!supports_extstore()) {
3477+ plan skip_all => 'extstore not enabled';
3478+ exit 0;
3479+}
3480+
3481+$ext_path = "/tmp/extstore1.$$";
3482+$ext_path2 = "/tmp/extstore2.$$";
3483+
3484+my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,ext_path=$ext_path2:96m,slab_automove=1");
3485+my $sock = $server->sock;
3486+
3487+my $value;
3488+{
3489+ my @chars = ("C".."Z");
3490+ for (1 .. 20000) {
3491+ $value .= $chars[rand @chars];
3492+ }
3493+}
3494+
3495+# fill some larger objects
3496+{
3497+ # interleave sets with 0 ttl vs long ttl's.
3498+ my $keycount = 3700;
3499+ for (1 .. $keycount) {
3500+ print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n";
3501+ print $sock "set lfoo$_ 0 0 20000 noreply\r\n$value\r\n";
3502+ }
3503+ # wait for a flush
3504+ wait_ext_flush($sock);
3505+ # delete half
3506+ mem_get_is($sock, "nfoo1", $value);
3507+ for (1 .. $keycount) {
3508+ print $sock "delete lfoo$_ noreply\r\n";
3509+ }
3510+ print $sock "lru_crawler crawl all\r\n";
3511+ <$sock>;
3512+ sleep 10;
3513+ # fetch
3514+ # check extstore counters
3515+ my $stats = mem_stats($sock);
3516+ is($stats->{evictions}, 0, 'no RAM evictions');
3517+ cmp_ok($stats->{extstore_page_allocs}, '>', 0, 'at least one page allocated');
3518+ cmp_ok($stats->{extstore_objects_written}, '>', $keycount / 2, 'some objects written');
3519+ cmp_ok($stats->{extstore_bytes_written}, '>', length($value) * 2, 'some bytes written');
3520+ cmp_ok($stats->{get_extstore}, '>', 0, 'one object was fetched');
3521+ cmp_ok($stats->{extstore_objects_read}, '>', 0, 'one object read');
3522+ cmp_ok($stats->{extstore_bytes_read}, '>', length($value), 'some bytes read');
3523+ cmp_ok($stats->{extstore_page_reclaims}, '>', 1, 'at least two pages reclaimed');
3524+}
3525+
3526+done_testing();
3527+
3528+END {
3529+ unlink $ext_path if $ext_path;
3530+ unlink $ext_path2 if $ext_path2;
3531+}
3532diff --git a/t/extstore.t b/t/extstore.t
3533index df4ca05..1790a54 100644
3534--- a/t/extstore.t
3535+++ b/t/extstore.t
3536@@ -17,9 +17,27 @@ if (!supports_extstore()) {
3537
3538 $ext_path = "/tmp/extstore.$$";
3539
3540-my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,slab_automove=0");
3541+my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_automove=0,ext_compact_under=1");
3542 my $sock = $server->sock;
3543
3544+# Wait until all items have flushed
3545+sub wait_for_ext {
3546+ my $target = shift || 0;
3547+ my $sum = $target + 1;
3548+ while ($sum > $target) {
3549+ my $s = mem_stats($sock, "items");
3550+ $sum = 0;
3551+ for my $key (keys %$s) {
3552+ if ($key =~ m/items:(\d+):number/) {
3553+ # Ignore classes which can contain extstore items
3554+ next if $1 < 3;
3555+ $sum += $s->{$key};
3556+ }
3557+ }
3558+ sleep 1 if $sum > $target;
3559+ }
3560+}
3561+
3562 my $value;
3563 {
3564 my @chars = ("C".."Z");
3565@@ -47,7 +65,7 @@ mem_get_is($sock, "foo", "hi");
3566 print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n";
3567 }
3568 # wait for a flush
3569- sleep 4;
3570+ wait_for_ext();
3571 # fetch
3572 # TODO: Fetch back all values
3573 mem_get_is($sock, "nfoo1", $value);
3574@@ -83,22 +101,28 @@ mem_get_is($sock, "foo", "hi");
3575
3576 # fill to eviction
3577 {
3578- my $keycount = 3000;
3579+ my $keycount = 4000;
3580 for (1 .. $keycount) {
3581 print $sock "set mfoo$_ 0 0 20000 noreply\r\n$value\r\n";
3582+ # wait to avoid evictions
3583+ wait_for_ext(500) if ($_ % 2000 == 0);
3584 }
3585- sleep 4;
3586+ # because item_age is set to 2s
3587+ wait_for_ext();
3588 my $stats = mem_stats($sock);
3589+ is($stats->{evictions}, 0, 'no evictions');
3590 is($stats->{miss_from_extstore}, 0, 'no misses');
3591- mem_get_is($sock, "canary", undef);
3592+ # FIXME: test is flaky; something can rescue the canary because of a race
3593+ # condition. might need to roundtrip twice or disable compaction?
3594+ #mem_get_is($sock, "canary", undef);
3595
3596 # check counters
3597 $stats = mem_stats($sock);
3598 cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted');
3599 cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted');
3600 cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted');
3601- is($stats->{extstore_pages_free}, 0, '0 pages are free');
3602- is($stats->{miss_from_extstore}, 1, 'exactly one miss');
3603+ cmp_ok($stats->{extstore_pages_free}, '<', 2, 'few pages are free');
3604+ #is($stats->{miss_from_extstore}, 1, 'exactly one miss');
3605
3606 # refresh some keys so rescues happen while drop_unread == 1.
3607 for (1 .. $keycount / 2) {
3608@@ -111,6 +135,10 @@ mem_get_is($sock, "foo", "hi");
3609 my $res = <$sock>;
3610 print $sock "extstore max_frag 0\r\n";
3611 $res = <$sock>;
3612+ print $sock "extstore compact_under 4\r\n";
3613+ $res = <$sock>;
3614+ print $sock "extstore drop_under 3\r\n";
3615+ $res = <$sock>;
3616 for (1 .. $keycount) {
3617 next unless $_ % 2 == 0;
3618 print $sock "delete mfoo$_ noreply\r\n";
3619@@ -131,7 +159,7 @@ mem_get_is($sock, "foo", "hi");
3620 for (1 .. $keycount) {
3621 print $sock "set bfoo$_ 0 0 20000 noreply\r\n$value\r\n";
3622 }
3623- sleep 4;
3624+ wait_for_ext();
3625
3626 # incr should be blocked.
3627 print $sock "incr bfoo1 1\r\n";
3628@@ -139,7 +167,7 @@ mem_get_is($sock, "foo", "hi");
3629
3630 # append/prepend *could* work, but it would require pulling the item back in.
3631 print $sock "append bfoo1 0 0 2\r\nhi\r\n";
3632- is(scalar <$sock>, "NOT_STORED\r\n", 'append falis');
3633+ is(scalar <$sock>, "NOT_STORED\r\n", 'append fails');
3634 print $sock "prepend bfoo1 0 0 2\r\nhi\r\n";
3635 is(scalar <$sock>, "NOT_STORED\r\n", 'prepend fails');
3636 }
3637diff --git a/t/flush-all.t b/t/flush-all.t
3638index e30d819..d9c536a 100755
3639--- a/t/flush-all.t
3640+++ b/t/flush-all.t
3641@@ -40,7 +40,7 @@ is(scalar <$sock>, "OK\r\n", "did flush_all in future");
3642 print $sock "set foo 0 0 4\r\n1234\r\n";
3643 is(scalar <$sock>, "STORED\r\n", "stored foo = '1234'");
3644 mem_get_is($sock, "foo", '1234');
3645-sleep(3);
3646+sleep(5);
3647 mem_get_is($sock, "foo", undef);
3648
3649 print $sock "set foo 0 0 5\r\n12345\r\n";
3650diff --git a/t/issue_67.t b/t/issue_67.t
3651index b2d374f..1dbaba2 100644
3652--- a/t/issue_67.t
3653+++ b/t/issue_67.t
3654@@ -1,11 +1,12 @@
3655 #!/usr/bin/perl
3656
3657 use strict;
3658-use Test::More tests => 22;
3659+use Test::More tests => 24;
3660 use FindBin qw($Bin);
3661 use lib "$Bin/lib";
3662 use MemcachedTest;
3663 use Carp qw(croak);
3664+use Socket qw(sockaddr_in INADDR_ANY PF_INET SOCK_STREAM);
3665
3666 use Cwd;
3667 my $builddir = getcwd;
3668@@ -29,12 +30,25 @@ sub validate_port {
3669 if ($expected == -1) {
3670 ok(!defined($got), "$name expected no port, got $got");
3671 } elsif ($expected == 0) {
3672- ok($got != 11211, "$name expected random port (got $got)");
3673+ ok(defined($got) && $got != 11211, "$name expected random port (got $got)");
3674 } else {
3675 is($got, $expected, "$name");
3676 }
3677 }
3678
3679+sub skip_if_default_addr_in_use(&) {
3680+ my ($block) = @_;
3681+
3682+ socket(my $socket, PF_INET, SOCK_STREAM, 0) or die $!;
3683+ my $addr_in_use = !bind($socket, sockaddr_in(11211, INADDR_ANY));
3684+ close($socket);
3685+
3686+ SKIP: {
3687+ skip 'Default address is in use. Do you have a running instance?', 2 if $addr_in_use;
3688+ return $block->();
3689+ }
3690+}
3691+
3692 sub run_server {
3693 my ($args) = @_;
3694
3695@@ -75,14 +89,13 @@ sub when {
3696 validate_port($name, $ports{'UDP INET'}, $expected_udp);
3697 }
3698
3699-# Disabling the defaults since it conflicts with a running instance.
3700-# when('no arguments', '', 11211, 11211);
3701+skip_if_default_addr_in_use { when('no arguments', '', 11211, -1) };
3702 when('specifying tcp port', '-p 11212', 11212, -1);
3703 when('specifying udp port', '-U 11222', 11222, 11222);
3704-when('specifying tcp ephemeral port', '-p -1', 0, 0);
3705+when('specifying tcp ephemeral port', '-p -1', 0, -1);
3706 when('specifying udp ephemeral port', '-U -1', 0, 0);
3707 when('tcp port disabled', '-p 0', -1, -1);
3708-when('udp port disabled', '-U 0', 11211, -1);
3709+skip_if_default_addr_in_use { when('udp port disabled', '-U 0', 11211, -1) };
3710 when('specifying tcp and udp ports', '-p 11232 -U 11233', 11232, 11233);
3711 when('specifying tcp and disabling udp', '-p 11242 -U 0', 11242, -1);
3712 when('specifying udp and disabling tcp', '-p -1 -U 11252', 0, 11252);
3713diff --git a/t/lib/MemcachedTest.pm b/t/lib/MemcachedTest.pm
3714index 4e1da66..416daaf 100644
3715--- a/t/lib/MemcachedTest.pm
3716+++ b/t/lib/MemcachedTest.pm
3717@@ -14,13 +14,33 @@ my $builddir = getcwd;
3718 my @unixsockets = ();
3719
3720 @EXPORT = qw(new_memcached sleep mem_get_is mem_gets mem_gets_is mem_stats
3721- supports_sasl free_port supports_drop_priv supports_extstore);
3722+ supports_sasl free_port supports_drop_priv supports_extstore
3723+ wait_ext_flush);
3724
3725 sub sleep {
3726 my $n = shift;
3727 select undef, undef, undef, $n;
3728 }
3729
3730+# Wait until all items have flushed
3731+sub wait_ext_flush {
3732+ my $sock = shift;
3733+ my $target = shift || 0;
3734+ my $sum = $target + 1;
3735+ while ($sum > $target) {
3736+ my $s = mem_stats($sock, "items");
3737+ $sum = 0;
3738+ for my $key (keys %$s) {
3739+ if ($key =~ m/items:(\d+):number/) {
3740+ # Ignore classes which can contain extstore items
3741+ next if $1 < 3;
3742+ $sum += $s->{$key};
3743+ }
3744+ }
3745+ sleep 1 if $sum > $target;
3746+ }
3747+}
3748+
3749 sub mem_stats {
3750 my ($sock, $type) = @_;
3751 $type = $type ? " $type" : "";
3752diff --git a/t/lru-crawler.t b/t/lru-crawler.t
3753index 18375c2..4e94fb3 100644
3754--- a/t/lru-crawler.t
3755+++ b/t/lru-crawler.t
3756@@ -2,7 +2,7 @@
3757
3758 use strict;
3759 use warnings;
3760-use Test::More tests => 221;
3761+use Test::More tests => 222;
3762 use FindBin qw($Bin);
3763 use lib "$Bin/lib";
3764 use MemcachedTest;
3765@@ -60,6 +60,18 @@ while (1) {
3766 is($items->{"items:1:crawler_reclaimed"}, 30, "slab1 has 30 reclaims");
3767 }
3768
3769+# Check that crawler metadump works correctly.
3770+{
3771+ print $sock "lru_crawler metadump all\r\n";
3772+ my $count = 0;
3773+ while (<$sock>) {
3774+ last if /^(\.|END)/;
3775+ /^(key=) (\S+).*([^\r\n]+)/;
3776+ $count++;
3777+ }
3778+ is ($count, 60);
3779+}
3780+
3781 for (1 .. 30) {
3782 mem_get_is($sock, "ifoo$_", "ok");
3783 mem_get_is($sock, "lfoo$_", "ok");
3784diff --git a/t/lru-maintainer.t b/t/lru-maintainer.t
3785index ac2ba7b..04fa76a 100644
3786--- a/t/lru-maintainer.t
3787+++ b/t/lru-maintainer.t
3788@@ -56,6 +56,18 @@ for (my $key = 0; $key < 100; $key++) {
3789 # Items need two fetches to become active
3790 mem_get_is($sock, "canary", $value);
3791 mem_get_is($sock, "canary", $value);
3792+ $stats = mem_stats($sock);
3793+ # The maintainer thread needs to juggle a bit to actually rescue an
3794+ # item. If it's slow we could evict after resuming setting.
3795+ sleep 1;
3796+ for (0..4) {
3797+ my $s2 = mem_stats($sock);
3798+ if ($s2->{lru_maintainer_juggles} - $stats->{lru_maintainer_juggles} < 5) {
3799+ sleep 1;
3800+ next;
3801+ }
3802+ last;
3803+ }
3804 }
3805 print $sock "set key$key 0 0 66560\r\n$value\r\n";
3806 is(scalar <$sock>, "STORED\r\n", "stored key$key");
3807diff --git a/t/misbehave.t b/t/misbehave.t
3808old mode 100644
3809new mode 100755
3810index ccf88f5..13cb7f3
3811--- a/t/misbehave.t
3812+++ b/t/misbehave.t
3813@@ -3,6 +3,7 @@
3814 use strict;
3815 use Test::More;
3816 use FindBin qw($Bin);
3817+use Socket qw(MSG_PEEK MSG_DONTWAIT);
3818 use lib "$Bin/lib";
3819 use MemcachedTest;
3820
3821@@ -13,8 +14,15 @@ if (supports_drop_priv()) {
3822 exit 0;
3823 }
3824
3825-my $server = new_memcached();
3826+my $server = new_memcached('-o drop_privileges');
3827 my $sock = $server->sock;
3828
3829 print $sock "misbehave\r\n";
3830-is(scalar <$sock>, "OK\r\n", "did not allow misbehaving");
3831+sleep(1);
3832+
3833+# check if the socket is dead now
3834+my $buff;
3835+my $ret = recv($sock, $buff, 1, MSG_PEEK | MSG_DONTWAIT);
3836+is($ret, undef, "did not allow misbehaving");
3837+
3838+$server->DESTROY();
3839diff --git a/thread.c b/thread.c
3840index d17a387..618ffac 100644
3841--- a/thread.c
3842+++ b/thread.c
3843@@ -139,22 +139,24 @@ void pause_threads(enum pause_thread_types type) {
3844 buf[0] = 0;
3845 switch (type) {
3846 case PAUSE_ALL_THREADS:
3847- lru_maintainer_pause();
3848 slabs_rebalancer_pause();
3849+ lru_maintainer_pause();
3850 lru_crawler_pause();
3851 #ifdef EXTSTORE
3852 storage_compact_pause();
3853+ storage_write_pause();
3854 #endif
3855 case PAUSE_WORKER_THREADS:
3856 buf[0] = 'p';
3857 pthread_mutex_lock(&worker_hang_lock);
3858 break;
3859 case RESUME_ALL_THREADS:
3860- lru_maintainer_resume();
3861 slabs_rebalancer_resume();
3862+ lru_maintainer_resume();
3863 lru_crawler_resume();
3864 #ifdef EXTSTORE
3865 storage_compact_resume();
3866+ storage_write_resume();
3867 #endif
3868 case RESUME_WORKER_THREADS:
3869 pthread_mutex_unlock(&worker_hang_lock);
3870@@ -625,7 +627,7 @@ void item_unlink(item *item) {
3871 * Does arithmetic on a numeric item value.
3872 */
3873 enum delta_result_type add_delta(conn *c, const char *key,
3874- const size_t nkey, int incr,
3875+ const size_t nkey, bool incr,
3876 const int64_t delta, char *buf,
3877 uint64_t *cas) {
3878 enum delta_result_type ret;
3879diff --git a/timedrun.c b/timedrun.c
3880index 6888453..a869004 100644
3881--- a/timedrun.c
3882+++ b/timedrun.c
3883@@ -7,11 +7,11 @@
3884
3885 #include <assert.h>
3886
3887-static int caught = 0;
3888+volatile sig_atomic_t caught_sig = 0;
3889
3890-static void caught_signal(int which)
3891+static void signal_handler(int which)
3892 {
3893- caught = which;
3894+ caught_sig = which;
3895 }
3896
3897 static int wait_for_process(pid_t pid)
3898@@ -21,7 +21,7 @@ static int wait_for_process(pid_t pid)
3899 int i = 0;
3900 struct sigaction sig_handler;
3901
3902- sig_handler.sa_handler = caught_signal;
3903+ sig_handler.sa_handler = signal_handler;
3904 sig_handler.sa_flags = 0;
3905
3906 sigaction(SIGALRM, &sig_handler, NULL);
3907@@ -44,8 +44,8 @@ static int wait_for_process(pid_t pid)
3908 switch (i) {
3909 case 0:
3910 /* On the first iteration, pass the signal through */
3911- sig = caught > 0 ? caught : SIGTERM;
3912- if (caught == SIGALRM) {
3913+ sig = caught_sig > 0 ? caught_sig : SIGTERM;
3914+ if (caught_sig == SIGALRM) {
3915 fprintf(stderr, "Timeout.. killing the process\n");
3916 }
3917 break;
3918diff --git a/version.m4 b/version.m4
3919index 4a02156..ff2636d 100644
3920--- a/version.m4
3921+++ b/version.m4
3922@@ -1 +1 @@
3923-m4_define([VERSION_NUMBER], [1.5.6])
3924+m4_define([VERSION_NUMBER], [1.5.10])

Subscribers

People subscribed via source and target branches