Merge ~racb/ubuntu/+source/memcached:upstream-bump-1.5.10 into ubuntu/+source/memcached:ubuntu/devel

Proposed by Robie Basak
Status: Merged
Merged at revision: 05743ddcb21ad054c5102c4cd3660bea10c6f17e
Proposed branch: ~racb/ubuntu/+source/memcached:upstream-bump-1.5.10
Merge into: ubuntu/+source/memcached:ubuntu/devel
Diff against target: 3924 lines (+1746/-336)
46 files modified
INSTALL (+370/-0)
Makefile.in (+2/-2)
README.md (+2/-2)
config.h.in (+7/-1)
configure (+80/-11)
configure.ac (+39/-3)
crawler.c (+8/-1)
crc32c.c (+72/-2)
debian/changelog (+8/-0)
debian/control (+2/-1)
doc/Makefile (+3/-3)
doc/memcached.1 (+2/-2)
doc/protocol.txt (+2/-0)
extstore.c (+97/-33)
extstore.h (+14/-2)
items.c (+8/-16)
linux_priv.c (+46/-4)
logger.c (+12/-3)
logger.h (+1/-0)
memcached.c (+155/-87)
memcached.h (+34/-2)
memcached.spec (+4/-4)
sasl_defs.c (+6/-2)
scripts/memcached-automove (+1/-1)
scripts/memcached-automove-extstore (+1/-2)
scripts/memcached-tool (+72/-58)
scripts/memcached-tool.1 (+5/-2)
slabs.c (+76/-3)
storage.c (+220/-28)
storage.h (+7/-1)
t/binary-extstore.t (+10/-2)
t/binary-sasl.t (+20/-7)
t/chunked-extstore.t (+35/-20)
t/error-extstore.t (+130/-0)
t/extstore-buckets.t (+1/-1)
t/extstore-jbod.t (+69/-0)
t/extstore.t (+37/-9)
t/flush-all.t (+1/-1)
t/issue_67.t (+19/-6)
t/lib/MemcachedTest.pm (+21/-1)
t/lru-crawler.t (+13/-1)
t/lru-maintainer.t (+12/-0)
t/misbehave.t (+10/-2)
thread.c (+5/-3)
timedrun.c (+6/-6)
version.m4 (+1/-1)
Reviewer Review Type Date Requested Status
Christian Ehrhardt  (community) Approve
Canonical Server Pending
Review via email: mp+352972@code.launchpad.net

Description of the change

This includes fixes for some issues which were causing build time test failures, preventing memcached from migrating to the release pocket. Thanks to upstream working closely with us, all the fixes are released upstream, so we just pull in the latest and it works.

After Debian takes this we should be able to sync.

To post a comment you must log in.
Revision history for this message
Robie Basak (racb) wrote :

A PPA build should be available in ppa:racb/experimental shortly.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi,
a few things inline - mostly suggestions for the changelog.

The bump itself seems correct to me - content change wise.
The md5sum of the upstream download matches:
$ md5sum memcached-1.5.10.tar.gz memcached_1.5.10.orig.tar.gz
8462616b554183a75845b03c56837cca memcached-1.5.10.tar.gz
8462616b554183a75845b03c56837cca memcached_1.5.10.orig.tar.gz

From:
http://www.memcached.org/files/memcached-1.5.10.tar.gz
https://launchpad.net/~racb/+archive/ubuntu/experimental/+sourcefiles/memcached/1.5.10-0ubuntu1~ppa1/memcached_1.5.10.orig.tar.gz

No other functional changes required.

The inline comments are all style-questions which you can decide either way.
I'll do some testing before a full +1

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I copied from your ppa that was identical to the proposed changes.
Testing from that ...

qa tests:
- I ran against the old version in cosmic 1.5.6-0ubuntu1 to know which might be known issues and if the test is repeatable
- of these memcdump had a fail, but all others worked
- after upgrading to the ppa version (upgrade worked) I reran the tests
- all tests that worked still work (no change on memcdump)

Running test: './test-memcached.py' distro: 'Ubuntu 18.10' kernel: '4.17.0-6.7 (Ubuntu 4.17.0-6.7-generic 4.17.9)' arch: 'amd64' uid: 0/0 SUDO_USER: 'ubuntu')
Skipping private tests
test_add (__main__.MemcachedTest)
Test adding data ... ok
test_cve_2011_4971 (__main__.MemcachedTest)
Test CVE-2011-4971 ... ok
test_default_tcp (__main__.MemcachedTest)
Test that memcached is listening on tcp port by default ... ok
test_default_udp (__main__.MemcachedTest)
Test that memcached is listening on udp port by default ... ok
test_memccapable (__main__.MemcachedTest)
Test memccapable ... ok
test_memccat (__main__.MemcachedTest)
Test memccat ... ok
test_memcdump (__main__.MemcachedTest)
Test memcdump ... FAIL
test_memcrm (__main__.MemcachedTest)
Test memcrm ... ok
test_memcstat (__main__.MemcachedTest)
Test memcstat ... ok
test_sample_data (__main__.MemcachedTest)
Test sample data ... ok
test_set (__main__.MemcachedTest)
Test setting data ... ok

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Autopkgtests are enqueued, will check results after lunch.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Autopkgtests are also good:
flask-limiter/1.0.1-1
gnocchi/4.2.4-0ubuntu3
memcached/1.5.10-0ubuntu1~ppa1
pgmemcache/2.3.0-4
python-limits/1.3-1

That said, code good, changelog ok (up to you to adapt my suggestions) and tests good - +1 on the MP

review: Approve
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
diff --git a/INSTALL b/INSTALL
0new file mode 1006440new file mode 100644
index 0000000..2099840
--- /dev/null
+++ b/INSTALL
@@ -0,0 +1,370 @@
1Installation Instructions
2*************************
3
4Copyright (C) 1994-1996, 1999-2002, 2004-2013 Free Software Foundation,
5Inc.
6
7 Copying and distribution of this file, with or without modification,
8are permitted in any medium without royalty provided the copyright
9notice and this notice are preserved. This file is offered as-is,
10without warranty of any kind.
11
12Basic Installation
13==================
14
15 Briefly, the shell command `./configure && make && make install'
16should configure, build, and install this package. The following
17more-detailed instructions are generic; see the `README' file for
18instructions specific to this package. Some packages provide this
19`INSTALL' file but do not implement all of the features documented
20below. The lack of an optional feature in a given package is not
21necessarily a bug. More recommendations for GNU packages can be found
22in *note Makefile Conventions: (standards)Makefile Conventions.
23
24 The `configure' shell script attempts to guess correct values for
25various system-dependent variables used during compilation. It uses
26those values to create a `Makefile' in each directory of the package.
27It may also create one or more `.h' files containing system-dependent
28definitions. Finally, it creates a shell script `config.status' that
29you can run in the future to recreate the current configuration, and a
30file `config.log' containing compiler output (useful mainly for
31debugging `configure').
32
33 It can also use an optional file (typically called `config.cache'
34and enabled with `--cache-file=config.cache' or simply `-C') that saves
35the results of its tests to speed up reconfiguring. Caching is
36disabled by default to prevent problems with accidental use of stale
37cache files.
38
39 If you need to do unusual things to compile the package, please try
40to figure out how `configure' could check whether to do them, and mail
41diffs or instructions to the address given in the `README' so they can
42be considered for the next release. If you are using the cache, and at
43some point `config.cache' contains results you don't want to keep, you
44may remove or edit it.
45
46 The file `configure.ac' (or `configure.in') is used to create
47`configure' by a program called `autoconf'. You need `configure.ac' if
48you want to change it or regenerate `configure' using a newer version
49of `autoconf'.
50
51 The simplest way to compile this package is:
52
53 1. `cd' to the directory containing the package's source code and type
54 `./configure' to configure the package for your system.
55
56 Running `configure' might take a while. While running, it prints
57 some messages telling which features it is checking for.
58
59 2. Type `make' to compile the package.
60
61 3. Optionally, type `make check' to run any self-tests that come with
62 the package, generally using the just-built uninstalled binaries.
63
64 4. Type `make install' to install the programs and any data files and
65 documentation. When installing into a prefix owned by root, it is
66 recommended that the package be configured and built as a regular
67 user, and only the `make install' phase executed with root
68 privileges.
69
70 5. Optionally, type `make installcheck' to repeat any self-tests, but
71 this time using the binaries in their final installed location.
72 This target does not install anything. Running this target as a
73 regular user, particularly if the prior `make install' required
74 root privileges, verifies that the installation completed
75 correctly.
76
77 6. You can remove the program binaries and object files from the
78 source code directory by typing `make clean'. To also remove the
79 files that `configure' created (so you can compile the package for
80 a different kind of computer), type `make distclean'. There is
81 also a `make maintainer-clean' target, but that is intended mainly
82 for the package's developers. If you use it, you may have to get
83 all sorts of other programs in order to regenerate files that came
84 with the distribution.
85
86 7. Often, you can also type `make uninstall' to remove the installed
87 files again. In practice, not all packages have tested that
88 uninstallation works correctly, even though it is required by the
89 GNU Coding Standards.
90
91 8. Some packages, particularly those that use Automake, provide `make
92 distcheck', which can by used by developers to test that all other
93 targets like `make install' and `make uninstall' work correctly.
94 This target is generally not run by end users.
95
96Compilers and Options
97=====================
98
99 Some systems require unusual options for compilation or linking that
100the `configure' script does not know about. Run `./configure --help'
101for details on some of the pertinent environment variables.
102
103 You can give `configure' initial values for configuration parameters
104by setting variables in the command line or in the environment. Here
105is an example:
106
107 ./configure CC=c99 CFLAGS=-g LIBS=-lposix
108
109 *Note Defining Variables::, for more details.
110
111Compiling For Multiple Architectures
112====================================
113
114 You can compile the package for more than one kind of computer at the
115same time, by placing the object files for each architecture in their
116own directory. To do this, you can use GNU `make'. `cd' to the
117directory where you want the object files and executables to go and run
118the `configure' script. `configure' automatically checks for the
119source code in the directory that `configure' is in and in `..'. This
120is known as a "VPATH" build.
121
122 With a non-GNU `make', it is safer to compile the package for one
123architecture at a time in the source code directory. After you have
124installed the package for one architecture, use `make distclean' before
125reconfiguring for another architecture.
126
127 On MacOS X 10.5 and later systems, you can create libraries and
128executables that work on multiple system types--known as "fat" or
129"universal" binaries--by specifying multiple `-arch' options to the
130compiler but only a single `-arch' option to the preprocessor. Like
131this:
132
133 ./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
134 CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \
135 CPP="gcc -E" CXXCPP="g++ -E"
136
137 This is not guaranteed to produce working output in all cases, you
138may have to build one architecture at a time and combine the results
139using the `lipo' tool if you have problems.
140
141Installation Names
142==================
143
144 By default, `make install' installs the package's commands under
145`/usr/local/bin', include files under `/usr/local/include', etc. You
146can specify an installation prefix other than `/usr/local' by giving
147`configure' the option `--prefix=PREFIX', where PREFIX must be an
148absolute file name.
149
150 You can specify separate installation prefixes for
151architecture-specific files and architecture-independent files. If you
152pass the option `--exec-prefix=PREFIX' to `configure', the package uses
153PREFIX as the prefix for installing programs and libraries.
154Documentation and other data files still use the regular prefix.
155
156 In addition, if you use an unusual directory layout you can give
157options like `--bindir=DIR' to specify different values for particular
158kinds of files. Run `configure --help' for a list of the directories
159you can set and what kinds of files go in them. In general, the
160default for these options is expressed in terms of `${prefix}', so that
161specifying just `--prefix' will affect all of the other directory
162specifications that were not explicitly provided.
163
164 The most portable way to affect installation locations is to pass the
165correct locations to `configure'; however, many packages provide one or
166both of the following shortcuts of passing variable assignments to the
167`make install' command line to change installation locations without
168having to reconfigure or recompile.
169
170 The first method involves providing an override variable for each
171affected directory. For example, `make install
172prefix=/alternate/directory' will choose an alternate location for all
173directory configuration variables that were expressed in terms of
174`${prefix}'. Any directories that were specified during `configure',
175but not in terms of `${prefix}', must each be overridden at install
176time for the entire installation to be relocated. The approach of
177makefile variable overrides for each directory variable is required by
178the GNU Coding Standards, and ideally causes no recompilation.
179However, some platforms have known limitations with the semantics of
180shared libraries that end up requiring recompilation when using this
181method, particularly noticeable in packages that use GNU Libtool.
182
183 The second method involves providing the `DESTDIR' variable. For
184example, `make install DESTDIR=/alternate/directory' will prepend
185`/alternate/directory' before all installation names. The approach of
186`DESTDIR' overrides is not required by the GNU Coding Standards, and
187does not work on platforms that have drive letters. On the other hand,
188it does better at avoiding recompilation issues, and works well even
189when some directory options were not specified in terms of `${prefix}'
190at `configure' time.
191
192Optional Features
193=================
194
195 If the package supports it, you can cause programs to be installed
196with an extra prefix or suffix on their names by giving `configure' the
197option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'.
198
199 Some packages pay attention to `--enable-FEATURE' options to
200`configure', where FEATURE indicates an optional part of the package.
201They may also pay attention to `--with-PACKAGE' options, where PACKAGE
202is something like `gnu-as' or `x' (for the X Window System). The
203`README' should mention any `--enable-' and `--with-' options that the
204package recognizes.
205
206 For packages that use the X Window System, `configure' can usually
207find the X include and library files automatically, but if it doesn't,
208you can use the `configure' options `--x-includes=DIR' and
209`--x-libraries=DIR' to specify their locations.
210
211 Some packages offer the ability to configure how verbose the
212execution of `make' will be. For these packages, running `./configure
213--enable-silent-rules' sets the default to minimal output, which can be
214overridden with `make V=1'; while running `./configure
215--disable-silent-rules' sets the default to verbose, which can be
216overridden with `make V=0'.
217
218Particular systems
219==================
220
221 On HP-UX, the default C compiler is not ANSI C compatible. If GNU
222CC is not installed, it is recommended to use the following options in
223order to use an ANSI C compiler:
224
225 ./configure CC="cc -Ae -D_XOPEN_SOURCE=500"
226
227and if that doesn't work, install pre-built binaries of GCC for HP-UX.
228
229 HP-UX `make' updates targets which have the same time stamps as
230their prerequisites, which makes it generally unusable when shipped
231generated files such as `configure' are involved. Use GNU `make'
232instead.
233
234 On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot
235parse its `<wchar.h>' header file. The option `-nodtk' can be used as
236a workaround. If GNU CC is not installed, it is therefore recommended
237to try
238
239 ./configure CC="cc"
240
241and if that doesn't work, try
242
243 ./configure CC="cc -nodtk"
244
245 On Solaris, don't put `/usr/ucb' early in your `PATH'. This
246directory contains several dysfunctional programs; working variants of
247these programs are available in `/usr/bin'. So, if you need `/usr/ucb'
248in your `PATH', put it _after_ `/usr/bin'.
249
250 On Haiku, software installed for all users goes in `/boot/common',
251not `/usr/local'. It is recommended to use the following options:
252
253 ./configure --prefix=/boot/common
254
255Specifying the System Type
256==========================
257
258 There may be some features `configure' cannot figure out
259automatically, but needs to determine by the type of machine the package
260will run on. Usually, assuming the package is built to be run on the
261_same_ architectures, `configure' can figure that out, but if it prints
262a message saying it cannot guess the machine type, give it the
263`--build=TYPE' option. TYPE can either be a short name for the system
264type, such as `sun4', or a canonical name which has the form:
265
266 CPU-COMPANY-SYSTEM
267
268where SYSTEM can have one of these forms:
269
270 OS
271 KERNEL-OS
272
273 See the file `config.sub' for the possible values of each field. If
274`config.sub' isn't included in this package, then this package doesn't
275need to know the machine type.
276
277 If you are _building_ compiler tools for cross-compiling, you should
278use the option `--target=TYPE' to select the type of system they will
279produce code for.
280
281 If you want to _use_ a cross compiler, that generates code for a
282platform different from the build platform, you should specify the
283"host" platform (i.e., that on which the generated programs will
284eventually be run) with `--host=TYPE'.
285
286Sharing Defaults
287================
288
289 If you want to set default values for `configure' scripts to share,
290you can create a site shell script called `config.site' that gives
291default values for variables like `CC', `cache_file', and `prefix'.
292`configure' looks for `PREFIX/share/config.site' if it exists, then
293`PREFIX/etc/config.site' if it exists. Or, you can set the
294`CONFIG_SITE' environment variable to the location of the site script.
295A warning: not all `configure' scripts look for a site script.
296
297Defining Variables
298==================
299
300 Variables not defined in a site shell script can be set in the
301environment passed to `configure'. However, some packages may run
302configure again during the build, and the customized values of these
303variables may be lost. In order to avoid this problem, you should set
304them in the `configure' command line, using `VAR=value'. For example:
305
306 ./configure CC=/usr/local2/bin/gcc
307
308causes the specified `gcc' to be used as the C compiler (unless it is
309overridden in the site shell script).
310
311Unfortunately, this technique does not work for `CONFIG_SHELL' due to
312an Autoconf limitation. Until the limitation is lifted, you can use
313this workaround:
314
315 CONFIG_SHELL=/bin/bash ./configure CONFIG_SHELL=/bin/bash
316
317`configure' Invocation
318======================
319
320 `configure' recognizes the following options to control how it
321operates.
322
323`--help'
324`-h'
325 Print a summary of all of the options to `configure', and exit.
326
327`--help=short'
328`--help=recursive'
329 Print a summary of the options unique to this package's
330 `configure', and exit. The `short' variant lists options used
331 only in the top level, while the `recursive' variant lists options
332 also present in any nested packages.
333
334`--version'
335`-V'
336 Print the version of Autoconf used to generate the `configure'
337 script, and exit.
338
339`--cache-file=FILE'
340 Enable the cache: use and save the results of the tests in FILE,
341 traditionally `config.cache'. FILE defaults to `/dev/null' to
342 disable caching.
343
344`--config-cache'
345`-C'
346 Alias for `--cache-file=config.cache'.
347
348`--quiet'
349`--silent'
350`-q'
351 Do not print messages saying which checks are being made. To
352 suppress all normal output, redirect it to `/dev/null' (any error
353 messages will still be shown).
354
355`--srcdir=DIR'
356 Look for the package's source code in directory DIR. Usually
357 `configure' can determine that directory automatically.
358
359`--prefix=DIR'
360 Use DIR as the installation prefix. *note Installation Names::
361 for more details, including other options available for fine-tuning
362 the installation locations.
363
364`--no-create'
365`-n'
366 Run the configure checks, but stop before creating any output
367 files.
368
369`configure' also accepts some other, not widely useful, options. Run
370`configure --help' for more details.
diff --git a/Makefile.in b/Makefile.in
index 966b4d7..c125ace 100644
--- a/Makefile.in
+++ b/Makefile.in
@@ -327,8 +327,8 @@ CTAGS = ctags
327CSCOPE = cscope327CSCOPE = cscope
328DIST_SUBDIRS = $(SUBDIRS)328DIST_SUBDIRS = $(SUBDIRS)
329am__DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/config.h.in AUTHORS \329am__DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/config.h.in AUTHORS \
330 COPYING ChangeLog NEWS compile config.guess config.sub depcomp \330 COPYING ChangeLog INSTALL NEWS compile config.guess config.sub \
331 install-sh missing331 depcomp install-sh missing
332DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)332DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST)
333distdir = $(PACKAGE)-$(VERSION)333distdir = $(PACKAGE)-$(VERSION)
334top_distdir = $(distdir)334top_distdir = $(distdir)
diff --git a/README.md b/README.md
index 935e4cd..e330fbe 100644
--- a/README.md
+++ b/README.md
@@ -18,8 +18,8 @@ list to ask questions, github issues aren't seen by everyone!
18## Dependencies18## Dependencies
1919
20* libevent, http://www.monkey.org/~provos/libevent/ (libevent-dev)20* libevent, http://www.monkey.org/~provos/libevent/ (libevent-dev)
21* libseccomp, (optional, linux) - enables process restrictions for better21* libseccomp, (optional, experimental, linux) - enables process restrictions for
22 security.22 better security. Tested only on x86_64 architectures.
2323
24## Environment24## Environment
2525
diff --git a/config.h.in b/config.h.in
index 9638dfa..3f3d42d 100644
--- a/config.h.in
+++ b/config.h.in
@@ -1,5 +1,8 @@
1/* config.h.in. Generated from configure.ac by autoheader. */1/* config.h.in. Generated from configure.ac by autoheader. */
22
3/* Set to nonzero if you want to enable ARMv8 crc32 */
4#undef ARM_CRC32
5
3/* Set to nonzero if you want to include DTRACE */6/* Set to nonzero if you want to include DTRACE */
4#undef ENABLE_DTRACE7#undef ENABLE_DTRACE
58
@@ -15,7 +18,7 @@
15/* machine is littleendian */18/* machine is littleendian */
16#undef ENDIAN_LITTLE19#undef ENDIAN_LITTLE
1720
18/* Set to nonzero if you want to enable extstorextstore */21/* Set to nonzero if you want to enable extstore */
19#undef EXTSTORE22#undef EXTSTORE
2023
21/* Define to 1 if support accept4 */24/* Define to 1 if support accept4 */
@@ -66,6 +69,9 @@
66/* Set to nonzero if your SASL implementation supports SASL_CB_GETCONF */69/* Set to nonzero if your SASL implementation supports SASL_CB_GETCONF */
67#undef HAVE_SASL_CB_GETCONF70#undef HAVE_SASL_CB_GETCONF
6871
72/* Set to nonzero if your SASL implementation supports SASL_CB_GETCONFPATH */
73#undef HAVE_SASL_CB_GETCONFPATH
74
69/* Define to 1 if you have the <sasl/sasl.h> header file. */75/* Define to 1 if you have the <sasl/sasl.h> header file. */
70#undef HAVE_SASL_SASL_H76#undef HAVE_SASL_SASL_H
7177
diff --git a/configure b/configure
index e479e60..9ccb4dd 100755
--- a/configure
+++ b/configure
@@ -1,6 +1,6 @@
1#! /bin/sh1#! /bin/sh
2# Guess values for system-dependent variables and create Makefiles.2# Guess values for system-dependent variables and create Makefiles.
3# Generated by GNU Autoconf 2.69 for memcached 1.5.6.3# Generated by GNU Autoconf 2.69 for memcached 1.5.10.
4#4#
5# Report bugs to <memcached@googlegroups.com>.5# Report bugs to <memcached@googlegroups.com>.
6#6#
@@ -580,8 +580,8 @@ MAKEFLAGS=
580# Identity of this package.580# Identity of this package.
581PACKAGE_NAME='memcached'581PACKAGE_NAME='memcached'
582PACKAGE_TARNAME='memcached'582PACKAGE_TARNAME='memcached'
583PACKAGE_VERSION='1.5.6'583PACKAGE_VERSION='1.5.10'
584PACKAGE_STRING='memcached 1.5.6'584PACKAGE_STRING='memcached 1.5.10'
585PACKAGE_BUGREPORT='memcached@googlegroups.com'585PACKAGE_BUGREPORT='memcached@googlegroups.com'
586PACKAGE_URL=''586PACKAGE_URL=''
587587
@@ -643,6 +643,8 @@ PROFILER
643PROFILER_LDFLAGS643PROFILER_LDFLAGS
644ENABLE_SASL644ENABLE_SASL
645DTRACEFLAGS645DTRACEFLAGS
646ENABLE_ARM_CRC32_FALSE
647ENABLE_ARM_CRC32_TRUE
646ENABLE_EXTSTORE_FALSE648ENABLE_EXTSTORE_FALSE
647ENABLE_EXTSTORE_TRUE649ENABLE_EXTSTORE_TRUE
648ENABLE_SASL_FALSE650ENABLE_SASL_FALSE
@@ -751,6 +753,7 @@ ac_user_opts='
751enable_option_checking753enable_option_checking
752enable_silent_rules754enable_silent_rules
753enable_dependency_tracking755enable_dependency_tracking
756enable_arm_crc32
754enable_extstore757enable_extstore
755enable_seccomp758enable_seccomp
756enable_sasl759enable_sasl
@@ -1320,7 +1323,7 @@ if test "$ac_init_help" = "long"; then
1320 # Omit some internal or obsolete options to make the list less imposing.1323 # Omit some internal or obsolete options to make the list less imposing.
1321 # This message is too long to be a string in the A/UX 3.1 sh.1324 # This message is too long to be a string in the A/UX 3.1 sh.
1322 cat <<_ACEOF1325 cat <<_ACEOF
1323\`configure' configures memcached 1.5.6 to adapt to many kinds of systems.1326\`configure' configures memcached 1.5.10 to adapt to many kinds of systems.
13241327
1325Usage: $0 [OPTION]... [VAR=VALUE]...1328Usage: $0 [OPTION]... [VAR=VALUE]...
13261329
@@ -1391,7 +1394,7 @@ fi
13911394
1392if test -n "$ac_init_help"; then1395if test -n "$ac_init_help"; then
1393 case $ac_init_help in1396 case $ac_init_help in
1394 short | recursive ) echo "Configuration of memcached 1.5.6:";;1397 short | recursive ) echo "Configuration of memcached 1.5.10:";;
1395 esac1398 esac
1396 cat <<\_ACEOF1399 cat <<\_ACEOF
13971400
@@ -1405,8 +1408,9 @@ Optional Features:
1405 do not reject slow dependency extractors1408 do not reject slow dependency extractors
1406 --disable-dependency-tracking1409 --disable-dependency-tracking
1407 speeds up one-time build1410 speeds up one-time build
1411 --enable-arm-crc32 Enable ARMv8 CRC32 instructions
1408 --enable-extstore Enable external storage EXPERIMENTAL1412 --enable-extstore Enable external storage EXPERIMENTAL
1409 --enable-seccomp Enable seccomp restrictions1413 --enable-seccomp Enable seccomp restrictions EXPERIMENTAL
1410 --enable-sasl Enable SASL authentication1414 --enable-sasl Enable SASL authentication
1411 --enable-sasl-pwdb Enable plaintext password db1415 --enable-sasl-pwdb Enable plaintext password db
1412 --enable-dtrace Enable dtrace probes1416 --enable-dtrace Enable dtrace probes
@@ -1495,7 +1499,7 @@ fi
1495test -n "$ac_init_help" && exit $ac_status1499test -n "$ac_init_help" && exit $ac_status
1496if $ac_init_version; then1500if $ac_init_version; then
1497 cat <<\_ACEOF1501 cat <<\_ACEOF
1498memcached configure 1.5.61502memcached configure 1.5.10
1499generated by GNU Autoconf 2.691503generated by GNU Autoconf 2.69
15001504
1501Copyright (C) 2012 Free Software Foundation, Inc.1505Copyright (C) 2012 Free Software Foundation, Inc.
@@ -1964,7 +1968,7 @@ cat >config.log <<_ACEOF
1964This file contains any messages produced by compilers while1968This file contains any messages produced by compilers while
1965running configure, to aid debugging if configure makes a mistake.1969running configure, to aid debugging if configure makes a mistake.
19661970
1967It was created by memcached $as_me 1.5.6, which was1971It was created by memcached $as_me 1.5.10, which was
1968generated by GNU Autoconf 2.69. Invocation command line was1972generated by GNU Autoconf 2.69. Invocation command line was
19691973
1970 $ $0 $@1974 $ $0 $@
@@ -2899,7 +2903,7 @@ fi
28992903
2900# Define the identity of the package.2904# Define the identity of the package.
2901 PACKAGE='memcached'2905 PACKAGE='memcached'
2902 VERSION='1.5.6'2906 VERSION='1.5.10'
29032907
29042908
2905cat >>confdefs.h <<_ACEOF2909cat >>confdefs.h <<_ACEOF
@@ -4701,6 +4705,12 @@ fi
47014705
47024706
47034707
4708# Check whether --enable-arm_crc32 was given.
4709if test "${enable_arm_crc32+set}" = set; then :
4710 enableval=$enable_arm_crc32;
4711fi
4712
4713
4704# Check whether --enable-extstore was given.4714# Check whether --enable-extstore was given.
4705if test "${enable_extstore+set}" = set; then :4715if test "${enable_extstore+set}" = set; then :
4706 enableval=$enable_extstore;4716 enableval=$enable_extstore;
@@ -4733,6 +4743,8 @@ fi
47334743
47344744
47354745
4746
4747
4736for ac_header in sasl/sasl.h4748for ac_header in sasl/sasl.h
4737do :4749do :
4738 ac_fn_c_check_header_mongrel "$LINENO" "sasl/sasl.h" "ac_cv_header_sasl_sasl_h" "$ac_includes_default"4750 ac_fn_c_check_header_mongrel "$LINENO" "sasl/sasl.h" "ac_cv_header_sasl_sasl_h" "$ac_includes_default"
@@ -4784,6 +4796,43 @@ $as_echo "#define HAVE_SASL_CB_GETCONF 1" >>confdefs.h
4784fi4796fi
47854797
47864798
4799 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for SASL_CB_GETCONFPATH" >&5
4800$as_echo_n "checking for SASL_CB_GETCONFPATH... " >&6; }
4801if ${ac_cv_c_sasl_cb_getconfpath+:} false; then :
4802 $as_echo_n "(cached) " >&6
4803else
4804 cat confdefs.h - <<_ACEOF >conftest.$ac_ext
4805/* end confdefs.h. */
4806
4807#include <sasl/sasl.h>
4808
4809int
4810main ()
4811{
4812
4813unsigned long val = SASL_CB_GETCONFPATH;
4814
4815 ;
4816 return 0;
4817}
4818_ACEOF
4819if ac_fn_c_try_compile "$LINENO"; then :
4820 ac_cv_c_sasl_cb_getconfpath=yes
4821else
4822 ac_cv_c_sasl_cb_getconfpath=no
4823fi
4824rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext
4825
4826fi
4827{ $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_c_sasl_cb_getconfpath" >&5
4828$as_echo "$ac_cv_c_sasl_cb_getconfpath" >&6; }
4829 if test "$ac_cv_c_sasl_cb_getconfpath" = "yes"; then :
4830
4831$as_echo "#define HAVE_SASL_CB_GETCONFPATH 1" >>confdefs.h
4832
4833fi
4834
4835
4787$as_echo "#define ENABLE_SASL 1" >>confdefs.h4836$as_echo "#define ENABLE_SASL 1" >>confdefs.h
47884837
4789 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sasl_server_init" >&54838 { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sasl_server_init" >&5
@@ -4929,6 +4978,12 @@ $as_echo "#define EXTSTORE 1" >>confdefs.h
49294978
4930fi4979fi
49314980
4981if test "x$enable_arm_crc32" = "xyes"; then
4982
4983$as_echo "#define ARM_CRC32 1" >>confdefs.h
4984
4985fi
4986
4932 if test "$build_dtrace" = "yes"; then4987 if test "$build_dtrace" = "yes"; then
4933 BUILD_DTRACE_TRUE=4988 BUILD_DTRACE_TRUE=
4934 BUILD_DTRACE_FALSE='#'4989 BUILD_DTRACE_FALSE='#'
@@ -4961,6 +5016,14 @@ else
4961 ENABLE_EXTSTORE_FALSE=5016 ENABLE_EXTSTORE_FALSE=
4962fi5017fi
49635018
5019 if test "$enable_arm_crc32" = "yes"; then
5020 ENABLE_ARM_CRC32_TRUE=
5021 ENABLE_ARM_CRC32_FALSE='#'
5022else
5023 ENABLE_ARM_CRC32_TRUE='#'
5024 ENABLE_ARM_CRC32_FALSE=
5025fi
5026
49645027
49655028
49665029
@@ -6203,6 +6266,7 @@ else
62036266
6204#include <stdlib.h>6267#include <stdlib.h>
6205#include <inttypes.h>6268#include <inttypes.h>
6269#pragma GCC optimize ("O0")
62066270
6207int6271int
6208main ()6272main ()
@@ -6292,6 +6356,7 @@ have_gcc_64atomics=no
6292$as_echo_n "checking for GCC 64bit atomics... " >&6; }6356$as_echo_n "checking for GCC 64bit atomics... " >&6; }
6293cat confdefs.h - <<_ACEOF >conftest.$ac_ext6357cat confdefs.h - <<_ACEOF >conftest.$ac_ext
6294/* end confdefs.h. */6358/* end confdefs.h. */
6359#include <inttypes.h>
62956360
6296int6361int
6297main ()6362main ()
@@ -6739,6 +6804,10 @@ if test -z "${ENABLE_EXTSTORE_TRUE}" && test -z "${ENABLE_EXTSTORE_FALSE}"; then
6739 as_fn_error $? "conditional \"ENABLE_EXTSTORE\" was never defined.6804 as_fn_error $? "conditional \"ENABLE_EXTSTORE\" was never defined.
6740Usually this means the macro was only invoked conditionally." "$LINENO" 56805Usually this means the macro was only invoked conditionally." "$LINENO" 5
6741fi6806fi
6807if test -z "${ENABLE_ARM_CRC32_TRUE}" && test -z "${ENABLE_ARM_CRC32_FALSE}"; then
6808 as_fn_error $? "conditional \"ENABLE_ARM_CRC32\" was never defined.
6809Usually this means the macro was only invoked conditionally." "$LINENO" 5
6810fi
6742if test -z "${BUILD_SOLARIS_PRIVS_TRUE}" && test -z "${BUILD_SOLARIS_PRIVS_FALSE}"; then6811if test -z "${BUILD_SOLARIS_PRIVS_TRUE}" && test -z "${BUILD_SOLARIS_PRIVS_FALSE}"; then
6743 as_fn_error $? "conditional \"BUILD_SOLARIS_PRIVS\" was never defined.6812 as_fn_error $? "conditional \"BUILD_SOLARIS_PRIVS\" was never defined.
6744Usually this means the macro was only invoked conditionally." "$LINENO" 56813Usually this means the macro was only invoked conditionally." "$LINENO" 5
@@ -7156,7 +7225,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1
7156# report actual input values of CONFIG_FILES etc. instead of their7225# report actual input values of CONFIG_FILES etc. instead of their
7157# values after options handling.7226# values after options handling.
7158ac_log="7227ac_log="
7159This file was extended by memcached $as_me 1.5.6, which was7228This file was extended by memcached $as_me 1.5.10, which was
7160generated by GNU Autoconf 2.69. Invocation command line was7229generated by GNU Autoconf 2.69. Invocation command line was
71617230
7162 CONFIG_FILES = $CONFIG_FILES7231 CONFIG_FILES = $CONFIG_FILES
@@ -7222,7 +7291,7 @@ _ACEOF
7222cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=17291cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1
7223ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"7292ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`"
7224ac_cs_version="\\7293ac_cs_version="\\
7225memcached config.status 1.5.67294memcached config.status 1.5.10
7226configured by $0, generated by GNU Autoconf 2.69,7295configured by $0, generated by GNU Autoconf 2.69,
7227 with options \\"\$ac_cs_config\\"7296 with options \\"\$ac_cs_config\\"
72287297
diff --git a/configure.ac b/configure.ac
index 9f05cfa..abcb8ae 100644
--- a/configure.ac
+++ b/configure.ac
@@ -82,11 +82,14 @@ fi
82AM_PROG_CC_C_O82AM_PROG_CC_C_O
83AC_PROG_INSTALL83AC_PROG_INSTALL
8484
85AC_ARG_ENABLE(arm_crc32,
86 [AS_HELP_STRING([--enable-arm-crc32], [Enable ARMv8 CRC32 instructions])])
87
85AC_ARG_ENABLE(extstore,88AC_ARG_ENABLE(extstore,
86 [AS_HELP_STRING([--enable-extstore], [Enable external storage EXPERIMENTAL ])])89 [AS_HELP_STRING([--enable-extstore], [Enable external storage EXPERIMENTAL ])])
8790
88AC_ARG_ENABLE(seccomp,91AC_ARG_ENABLE(seccomp,
89 [AS_HELP_STRING([--enable-seccomp],[Enable seccomp restrictions])])92 [AS_HELP_STRING([--enable-seccomp],[Enable seccomp restrictions EXPERIMENTAL])])
9093
91AC_ARG_ENABLE(sasl,94AC_ARG_ENABLE(sasl,
92 [AS_HELP_STRING([--enable-sasl],[Enable SASL authentication])])95 [AS_HELP_STRING([--enable-sasl],[Enable SASL authentication])])
@@ -122,9 +125,33 @@ unsigned long val = SASL_CB_GETCONF;
122 [Set to nonzero if your SASL implementation supports SASL_CB_GETCONF])])125 [Set to nonzero if your SASL implementation supports SASL_CB_GETCONF])])
123])126])
124127
128dnl **********************************************************************
129dnl DETECT_SASL_CB_GETCONFPATH
130dnl
131dnl check if we can use SASL_CB_GETCONFPATH
132dnl **********************************************************************
133AC_DEFUN([AC_C_DETECT_SASL_CB_GETCONFPATH],
134[
135 AC_CACHE_CHECK([for SASL_CB_GETCONFPATH],
136 [ac_cv_c_sasl_cb_getconfpath],
137 [AC_TRY_COMPILE(
138 [
139#include <sasl/sasl.h>
140 ], [
141unsigned long val = SASL_CB_GETCONFPATH;
142 ],
143 [ ac_cv_c_sasl_cb_getconfpath=yes ],
144 [ ac_cv_c_sasl_cb_getconfpath=no ])
145 ])
146 AS_IF([test "$ac_cv_c_sasl_cb_getconfpath" = "yes"],
147 [AC_DEFINE([HAVE_SASL_CB_GETCONFPATH], 1,
148 [Set to nonzero if your SASL implementation supports SASL_CB_GETCONFPATH])])
149])
150
125AC_CHECK_HEADERS([sasl/sasl.h])151AC_CHECK_HEADERS([sasl/sasl.h])
126if test "x$enable_sasl" = "xyes"; then152if test "x$enable_sasl" = "xyes"; then
127 AC_C_DETECT_SASL_CB_GETCONF153 AC_C_DETECT_SASL_CB_GETCONF
154 AC_C_DETECT_SASL_CB_GETCONFPATH
128 AC_DEFINE([ENABLE_SASL],1,[Set to nonzero if you want to include SASL])155 AC_DEFINE([ENABLE_SASL],1,[Set to nonzero if you want to include SASL])
129 AC_SEARCH_LIBS([sasl_server_init], [sasl2 sasl], [],156 AC_SEARCH_LIBS([sasl_server_init], [sasl2 sasl], [],
130 [157 [
@@ -160,13 +187,18 @@ if test "x$enable_dtrace" = "xyes"; then
160fi187fi
161188
162if test "x$enable_extstore" = "xyes"; then189if test "x$enable_extstore" = "xyes"; then
163 AC_DEFINE([EXTSTORE],1,[Set to nonzero if you want to enable extstorextstore])190 AC_DEFINE([EXTSTORE],1,[Set to nonzero if you want to enable extstore])
191fi
192
193if test "x$enable_arm_crc32" = "xyes"; then
194 AC_DEFINE([ARM_CRC32],1,[Set to nonzero if you want to enable ARMv8 crc32])
164fi195fi
165196
166AM_CONDITIONAL([BUILD_DTRACE],[test "$build_dtrace" = "yes"])197AM_CONDITIONAL([BUILD_DTRACE],[test "$build_dtrace" = "yes"])
167AM_CONDITIONAL([DTRACE_INSTRUMENT_OBJ],[test "$dtrace_instrument_obj" = "yes"])198AM_CONDITIONAL([DTRACE_INSTRUMENT_OBJ],[test "$dtrace_instrument_obj" = "yes"])
168AM_CONDITIONAL([ENABLE_SASL],[test "$enable_sasl" = "yes"])199AM_CONDITIONAL([ENABLE_SASL],[test "$enable_sasl" = "yes"])
169AM_CONDITIONAL([ENABLE_EXTSTORE],[test "$enable_extstore" = "yes"])200AM_CONDITIONAL([ENABLE_EXTSTORE],[test "$enable_extstore" = "yes"])
201AM_CONDITIONAL([ENABLE_ARM_CRC32],[test "$enable_arm_crc32" = "yes"])
170202
171AC_SUBST(DTRACE)203AC_SUBST(DTRACE)
172AC_SUBST(DTRACEFLAGS)204AC_SUBST(DTRACEFLAGS)
@@ -490,6 +522,8 @@ AC_CHECK_FUNCS(clock_gettime)
490AC_CHECK_FUNCS([accept4], [AC_DEFINE(HAVE_ACCEPT4, 1, [Define to 1 if support accept4])])522AC_CHECK_FUNCS([accept4], [AC_DEFINE(HAVE_ACCEPT4, 1, [Define to 1 if support accept4])])
491AC_CHECK_FUNCS([getopt_long], [AC_DEFINE(HAVE_GETOPT_LONG, 1, [Define to 1 if support getopt_long])])523AC_CHECK_FUNCS([getopt_long], [AC_DEFINE(HAVE_GETOPT_LONG, 1, [Define to 1 if support getopt_long])])
492524
525dnl Need to disable opt for alignment check. GCC is too clever and turns this
526dnl into wide stores and no cmp under O2.
493AC_DEFUN([AC_C_ALIGNMENT],527AC_DEFUN([AC_C_ALIGNMENT],
494[AC_CACHE_CHECK(for alignment, ac_cv_c_alignment,528[AC_CACHE_CHECK(for alignment, ac_cv_c_alignment,
495[529[
@@ -497,6 +531,7 @@ AC_DEFUN([AC_C_ALIGNMENT],
497 [AC_LANG_PROGRAM([531 [AC_LANG_PROGRAM([
498#include <stdlib.h>532#include <stdlib.h>
499#include <inttypes.h>533#include <inttypes.h>
534#pragma GCC optimize ("O0")
500 ], [535 ], [
501 char *buf = malloc(32);536 char *buf = malloc(32);
502537
@@ -551,7 +586,8 @@ dnl Check for usage of 64bit atomics
551dnl 32bit systems shouldn't have these.586dnl 32bit systems shouldn't have these.
552have_gcc_64atomics=no587have_gcc_64atomics=no
553AC_MSG_CHECKING(for GCC 64bit atomics)588AC_MSG_CHECKING(for GCC 64bit atomics)
554AC_TRY_LINK([],[589AC_TRY_LINK([#include <inttypes.h>
590 ],[
555 uint64_t a;591 uint64_t a;
556 uint64_t b;592 uint64_t b;
557 b = __sync_add_and_fetch(&a, 1);593 b = __sync_add_and_fetch(&a, 1);
diff --git a/crawler.c b/crawler.c
index 9c81bba..a1a5ead 100644
--- a/crawler.c
+++ b/crawler.c
@@ -218,7 +218,6 @@ static void crawler_expired_eval(crawler_module_t *cm, item *search, uint32_t hv
218#endif218#endif
219 do_item_unlink_nolock(search, hv);219 do_item_unlink_nolock(search, hv);
220 do_item_remove(search);220 do_item_remove(search);
221 assert(search->slabs_clsid == 0);
222 } else {221 } else {
223 s->seen++;222 s->seen++;
224 refcount_decr(search);223 refcount_decr(search);
@@ -533,6 +532,14 @@ static int do_lru_crawler_start(uint32_t id, uint32_t remaining) {
533 if (remaining == LRU_CRAWLER_CAP_REMAINING) {532 if (remaining == LRU_CRAWLER_CAP_REMAINING) {
534 remaining = do_get_lru_size(sid);533 remaining = do_get_lru_size(sid);
535 }534 }
535 /* Values for remaining:
536 * remaining = 0
537 * - scan all elements, until a NULL is reached
538 * - if empty, NULL is reached right away
539 * remaining = n + 1
540 * - first n elements are parsed (or until a NULL is reached)
541 */
542 if (remaining) remaining++;
536 crawlers[sid].remaining = remaining;543 crawlers[sid].remaining = remaining;
537 crawlers[sid].slabs_clsid = sid;544 crawlers[sid].slabs_clsid = sid;
538 crawlers[sid].reclaimed = 0;545 crawlers[sid].reclaimed = 0;
diff --git a/crc32c.c b/crc32c.c
index 13deee2..a4296a7 100644
--- a/crc32c.c
+++ b/crc32c.c
@@ -40,6 +40,10 @@
40#include <stdint.h>40#include <stdint.h>
41#include <unistd.h>41#include <unistd.h>
42#include <pthread.h>42#include <pthread.h>
43#include "config.h"
44#if defined(__linux__) && defined(__aarch64__)
45#include <sys/auxv.h>
46#endif
43#include "crc32c.h"47#include "crc32c.h"
4448
45/* CRC-32C (iSCSI) polynomial in reversed bit order. */49/* CRC-32C (iSCSI) polynomial in reversed bit order. */
@@ -109,6 +113,60 @@ static uint32_t crc32c_sw(uint32_t crci, const void *buf, size_t len)
109 return (uint32_t)crc ^ 0xffffffff;113 return (uint32_t)crc ^ 0xffffffff;
110}114}
111115
116/* Hardware CRC support for aarch64 platform */
117#if defined(__linux__) && defined(__aarch64__) && defined(ARM_CRC32)
118
119#define CRC32CX(crc, value) __asm__("crc32cx %w[c], %w[c], %x[v]":[c]"+r"(crc):[v]"r"(+value))
120#define CRC32CW(crc, value) __asm__("crc32cw %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value))
121#define CRC32CH(crc, value) __asm__("crc32ch %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value))
122#define CRC32CB(crc, value) __asm__("crc32cb %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value))
123
124#ifndef HWCAP_CRC32
125#define HWCAP_CRC32 (1 << 7)
126#endif /* HWCAP for crc32 */
127
128static uint32_t crc32c_hw_aarch64(uint32_t crc, const void* buf, size_t len)
129{
130 const uint8_t* p_buf = buf;
131 uint64_t crc64bit = crc;
132 for (size_t i = 0; i < len / sizeof(uint64_t); i++) {
133 CRC32CX(crc64bit, *(uint64_t*) p_buf);
134 p_buf += sizeof(uint64_t);
135 }
136
137 uint32_t crc32bit = (uint32_t) crc64bit;
138 len &= sizeof(uint64_t) - 1;
139 switch (len) {
140 case 7:
141 CRC32CB(crc32bit, *p_buf++);
142 case 6:
143 CRC32CH(crc32bit, *(uint16_t*) p_buf);
144 p_buf += 2;
145 case 4:
146 CRC32CW(crc32bit, *(uint32_t*) p_buf);
147 break;
148 case 3:
149 CRC32CB(crc32bit, *p_buf++);
150 case 2:
151 CRC32CH(crc32bit, *(uint16_t*) p_buf);
152 break;
153 case 5:
154 CRC32CW(crc32bit, *(uint32_t*) p_buf);
155 p_buf += 4;
156 case 1:
157 CRC32CB(crc32bit, *p_buf);
158 break;
159 case 0:
160 break;
161 }
162
163 return crc32bit;
164}
165#endif
166
167/* Apply if the platform is intel */
168#if defined(__X86_64__)||defined(__x86_64__)||defined(__ia64__)
169
112/* Multiply a matrix times a vector over the Galois field of two elements,170/* Multiply a matrix times a vector over the Galois field of two elements,
113 GF(2). Each element is a bit in an unsigned integer. mat must have at171 GF(2). Each element is a bit in an unsigned integer. mat must have at
114 least as many entries as the power of two for most significant one bit in172 least as many entries as the power of two for most significant one bit in
@@ -329,15 +387,27 @@ static uint32_t crc32c_hw(uint32_t crc, const void *buf, size_t len)
329 (have) = (ecx >> 20) & 1; \387 (have) = (ecx >> 20) & 1; \
330 } while (0)388 } while (0)
331389
390#endif
332/* Compute a CRC-32C. If the crc32 instruction is available, use the hardware391/* Compute a CRC-32C. If the crc32 instruction is available, use the hardware
333 version. Otherwise, use the software version. */392 version. Otherwise, use the software version. */
334void crc32c_init(void) {393void crc32c_init(void) {
394 #if defined(__X86_64__)||defined(__x86_64__)||defined(__ia64__)
335 int sse42;395 int sse42;
336
337 SSE42(sse42);396 SSE42(sse42);
397
338 if (sse42) {398 if (sse42) {
339 crc32c = crc32c_hw;399 crc32c = crc32c_hw;
340 } else {400 } else
401 #endif
402 /* Check if CRC instructions supported by aarch64 */
403 #if defined(__linux__) && defined(__aarch64__) && defined(ARM_CRC32)
404 unsigned long hwcap = getauxval(AT_HWCAP);
405
406 if (hwcap & HWCAP_CRC32) {
407 crc32c = crc32c_hw_aarch64;
408 } else
409 #endif
410 {
341 crc32c = crc32c_sw;411 crc32c = crc32c_sw;
342 }412 }
343}413}
diff --git a/debian/changelog b/debian/changelog
index 43c90a2..1397f89 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,11 @@
1memcached (1.5.10-0ubuntu1) cosmic; urgency=medium
2
3 * New upstream release.
4 * Includes fixes for various failures on various architectures
5 (LP: #1780838).
6
7 -- Robie Basak <robie.basak@ubuntu.com> Mon, 13 Aug 2018 13:10:37 +0100
8
1memcached (1.5.6-1) unstable; urgency=medium9memcached (1.5.6-1) unstable; urgency=medium
210
3 * New upstream release11 * New upstream release
diff --git a/debian/control b/debian/control
index 72e948f..2c63ea0 100644
--- a/debian/control
+++ b/debian/control
@@ -1,7 +1,8 @@
1Source: memcached1Source: memcached
2Section: web2Section: web
3Priority: optional3Priority: optional
4Maintainer: Guillaume Delacour <gui@iroqwa.org>4Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com>
5XSBC-Original-Maintainer: Guillaume Delacour <gui@iroqwa.org>
5Build-Depends: debhelper (>> 10), libevent-dev, libsasl2-dev, adduser,6Build-Depends: debhelper (>> 10), libevent-dev, libsasl2-dev, adduser,
6 autotools-dev, dh-autoreconf7 autotools-dev, dh-autoreconf
7Homepage: http://www.memcached.org/8Homepage: http://www.memcached.org/
diff --git a/doc/Makefile b/doc/Makefile
index 0337680..49cce71 100644
--- a/doc/Makefile
+++ b/doc/Makefile
@@ -191,10 +191,10 @@ OBJEXT = o
191PACKAGE = memcached191PACKAGE = memcached
192PACKAGE_BUGREPORT = memcached@googlegroups.com192PACKAGE_BUGREPORT = memcached@googlegroups.com
193PACKAGE_NAME = memcached193PACKAGE_NAME = memcached
194PACKAGE_STRING = memcached 1.5.6194PACKAGE_STRING = memcached 1.5.10
195PACKAGE_TARNAME = memcached195PACKAGE_TARNAME = memcached
196PACKAGE_URL = 196PACKAGE_URL =
197PACKAGE_VERSION = 1.5.6197PACKAGE_VERSION = 1.5.10
198PATH_SEPARATOR = :198PATH_SEPARATOR = :
199PROFILER = /usr/bin/gcov199PROFILER = /usr/bin/gcov
200PROFILER_FLAGS = -fprofile-arcs -ftest-coverage200PROFILER_FLAGS = -fprofile-arcs -ftest-coverage
@@ -202,7 +202,7 @@ PROFILER_LDFLAGS = -lgcov
202SET_MAKE = 202SET_MAKE =
203SHELL = /bin/bash203SHELL = /bin/bash
204STRIP = 204STRIP =
205VERSION = 1.5.6205VERSION = 1.5.10
206XML2RFC = /usr/bin/xml2rfc206XML2RFC = /usr/bin/xml2rfc
207XSLTPROC = no207XSLTPROC = no
208abs_builddir = /home/dormando/d/p/danga/git/memcached/doc208abs_builddir = /home/dormando/d/p/danga/git/memcached/doc
diff --git a/doc/memcached.1 b/doc/memcached.1
index caa53b7..04aca5f 100644
--- a/doc/memcached.1
+++ b/doc/memcached.1
@@ -62,10 +62,10 @@ caches, so consult the README and memcached homepage for configuration
62suggestions.62suggestions.
63.TP63.TP
64.B \-p, --port=<num>64.B \-p, --port=<num>
65Listen on TCP port <num>, the default is port 11211.65Listen on TCP port <num>, the default is port 11211. 0 means off.
66.TP66.TP
67.B \-U, --udp-port=<num>67.B \-U, --udp-port=<num>
68Listen on UDP port <num>, the default is port 11211, 0 is off.68Listen on UDP port <num>, the default is port 0, which is off.
69.TP69.TP
70.B \-M, --disable-evictions70.B \-M, --disable-evictions
71Disable automatic removal of items from the cache when out of memory.71Disable automatic removal of items from the cache when out of memory.
diff --git a/doc/protocol.txt b/doc/protocol.txt
index 9c8e8bc..0984594 100644
--- a/doc/protocol.txt
+++ b/doc/protocol.txt
@@ -849,6 +849,8 @@ other stats command.
849| | bool | If yes, stores numbers from VALUE response |849| | bool | If yes, stores numbers from VALUE response |
850| | | inside an item, using up to 24 bytes. |850| | | inside an item, using up to 24 bytes. |
851| | | Small slowdown for ASCII get, faster sets. |851| | | Small slowdown for ASCII get, faster sets. |
852| drop_privileges | bool | If yes, and available, drop unused syscalls |
853| | | (see seccomp on Linux, pledge on OpenBSD) |
852|-------------------+----------+----------------------------------------------|854|-------------------+----------+----------------------------------------------|
853855
854856
diff --git a/extstore.c b/extstore.c
index 02558c0..726435c 100644
--- a/extstore.c
+++ b/extstore.c
@@ -62,6 +62,7 @@ typedef struct _store_page {
62 unsigned int allocated;62 unsigned int allocated;
63 unsigned int written; /* item offsets can be past written if wbuf not flushed */63 unsigned int written; /* item offsets can be past written if wbuf not flushed */
64 unsigned int bucket; /* which bucket the page is linked into */64 unsigned int bucket; /* which bucket the page is linked into */
65 unsigned int free_bucket; /* which bucket this page returns to when freed */
65 int fd;66 int fd;
66 unsigned short id;67 unsigned short id;
67 bool active; /* actively being written to */68 bool active; /* actively being written to */
@@ -95,6 +96,7 @@ struct store_engine {
95 store_maint_thread *maint_thread;96 store_maint_thread *maint_thread;
96 store_page *page_freelist;97 store_page *page_freelist;
97 store_page **page_buckets; /* stack of pages currently allocated to each bucket */98 store_page **page_buckets; /* stack of pages currently allocated to each bucket */
99 store_page **free_page_buckets; /* stack of use-case isolated free pages */
98 size_t page_size;100 size_t page_size;
99 unsigned int version; /* global version counter */101 unsigned int version; /* global version counter */
100 unsigned int last_io_thread; /* round robin the IO threads */102 unsigned int last_io_thread; /* round robin the IO threads */
@@ -102,6 +104,7 @@ struct store_engine {
102 unsigned int page_count;104 unsigned int page_count;
103 unsigned int page_free; /* unallocated pages */105 unsigned int page_free; /* unallocated pages */
104 unsigned int page_bucketcount; /* count of potential page buckets */106 unsigned int page_bucketcount; /* count of potential page buckets */
107 unsigned int free_page_bucketcount; /* count of free page buckets */
105 unsigned int io_depth; /* FIXME: Might cache into thr struct */108 unsigned int io_depth; /* FIXME: Might cache into thr struct */
106 pthread_mutex_t stats_mutex;109 pthread_mutex_t stats_mutex;
107 struct extstore_stats stats;110 struct extstore_stats stats;
@@ -192,11 +195,11 @@ const char *extstore_err(enum extstore_res res) {
192 return rv;195 return rv;
193}196}
194197
195void *extstore_init(char *fn, struct extstore_conf *cf,198// TODO: #define's for DEFAULT_BUCKET, FREE_VERSION, etc
199void *extstore_init(struct extstore_conf_file *fh, struct extstore_conf *cf,
196 enum extstore_res *res) {200 enum extstore_res *res) {
197 int i;201 int i;
198 int fd;202 struct extstore_conf_file *f = NULL;
199 uint64_t offset = 0;
200 pthread_t thread;203 pthread_t thread;
201204
202 if (cf->page_size % cf->wbuf_size != 0) {205 if (cf->page_size % cf->wbuf_size != 0) {
@@ -227,43 +230,72 @@ void *extstore_init(char *fn, struct extstore_conf *cf,
227 }230 }
228231
229 e->page_size = cf->page_size;232 e->page_size = cf->page_size;
230 fd = open(fn, O_RDWR | O_CREAT | O_TRUNC, 0644);233 for (f = fh; f != NULL; f = f->next) {
231 if (fd < 0) {234 f->fd = open(f->file, O_RDWR | O_CREAT | O_TRUNC, 0644);
232 *res = EXTSTORE_INIT_OPEN_FAIL;235 if (f->fd < 0) {
236 *res = EXTSTORE_INIT_OPEN_FAIL;
233#ifdef EXTSTORE_DEBUG237#ifdef EXTSTORE_DEBUG
234 perror("open");238 perror("open");
235#endif239#endif
236 free(e);240 free(e);
237 return NULL;241 return NULL;
242 }
243 e->page_count += f->page_count;
244 f->offset = 0;
238 }245 }
239246
240 e->pages = calloc(cf->page_count, sizeof(store_page));247 e->pages = calloc(e->page_count, sizeof(store_page));
241 if (e->pages == NULL) {248 if (e->pages == NULL) {
242 *res = EXTSTORE_INIT_OOM;249 *res = EXTSTORE_INIT_OOM;
243 close(fd);250 // FIXME: loop-close. make error label
244 free(e);251 free(e);
245 return NULL;252 return NULL;
246 }253 }
247254
248 for (i = 0; i < cf->page_count; i++) {255 // interleave the pages between devices
256 f = NULL; // start at the first device.
257 for (i = 0; i < e->page_count; i++) {
258 // find next device with available pages
259 while (1) {
260 // restart the loop
261 if (f == NULL || f->next == NULL) {
262 f = fh;
263 } else {
264 f = f->next;
265 }
266 if (f->page_count) {
267 f->page_count--;
268 break;
269 }
270 }
249 pthread_mutex_init(&e->pages[i].mutex, NULL);271 pthread_mutex_init(&e->pages[i].mutex, NULL);
250 e->pages[i].id = i;272 e->pages[i].id = i;
251 e->pages[i].fd = fd;273 e->pages[i].fd = f->fd;
252 e->pages[i].offset = offset;274 e->pages[i].free_bucket = f->free_bucket;
275 e->pages[i].offset = f->offset;
253 e->pages[i].free = true;276 e->pages[i].free = true;
254 offset += e->page_size;277 f->offset += e->page_size;
255 }278 }
256279
257 for (i = cf->page_count-1; i > 0; i--) {280 // free page buckets allows the app to organize devices by use case
258 e->pages[i].next = e->page_freelist;281 e->free_page_buckets = calloc(cf->page_buckets, sizeof(store_page *));
259 e->page_freelist = &e->pages[i];282 e->page_bucketcount = cf->page_buckets;
283
284 for (i = e->page_count-1; i > 0; i--) {
260 e->page_free++;285 e->page_free++;
286 if (e->pages[i].free_bucket == 0) {
287 e->pages[i].next = e->page_freelist;
288 e->page_freelist = &e->pages[i];
289 } else {
290 int fb = e->pages[i].free_bucket;
291 e->pages[i].next = e->free_page_buckets[fb];
292 e->free_page_buckets[fb] = &e->pages[i];
293 }
261 }294 }
262295
263 // 0 is magic "page is freed" version296 // 0 is magic "page is freed" version
264 e->version = 1;297 e->version = 1;
265298
266 e->page_count = cf->page_count;
267 // scratch data for stats. TODO: malloc failure handle299 // scratch data for stats. TODO: malloc failure handle
268 e->stats.page_data =300 e->stats.page_data =
269 calloc(e->page_count, sizeof(struct extstore_page_data));301 calloc(e->page_count, sizeof(struct extstore_page_data));
@@ -305,8 +337,12 @@ void *extstore_init(char *fn, struct extstore_conf *cf,
305 e->maint_thread = calloc(1, sizeof(store_maint_thread));337 e->maint_thread = calloc(1, sizeof(store_maint_thread));
306 e->maint_thread->e = e;338 e->maint_thread->e = e;
307 // FIXME: error handling339 // FIXME: error handling
340 pthread_mutex_init(&e->maint_thread->mutex, NULL);
341 pthread_cond_init(&e->maint_thread->cond, NULL);
308 pthread_create(&thread, NULL, extstore_maint_thread, e->maint_thread);342 pthread_create(&thread, NULL, extstore_maint_thread, e->maint_thread);
309343
344 extstore_run_maint(e);
345
310 return (void *)e;346 return (void *)e;
311}347}
312348
@@ -316,13 +352,25 @@ void extstore_run_maint(void *ptr) {
316}352}
317353
318// call with *e locked354// call with *e locked
319static store_page *_allocate_page(store_engine *e, unsigned int bucket) {355static store_page *_allocate_page(store_engine *e, unsigned int bucket,
356 unsigned int free_bucket) {
320 assert(!e->page_buckets[bucket] || e->page_buckets[bucket]->allocated == e->page_size);357 assert(!e->page_buckets[bucket] || e->page_buckets[bucket]->allocated == e->page_size);
321 store_page *tmp = e->page_freelist;358 store_page *tmp = NULL;
322 E_DEBUG("EXTSTORE: allocating new page\n");359 // if a specific free bucket was requested, check there first
323 if (e->page_free > 0) {360 if (free_bucket != 0 && e->free_page_buckets[free_bucket] != NULL) {
324 assert(e->page_freelist != NULL);361 assert(e->page_free > 0);
362 tmp = e->free_page_buckets[free_bucket];
363 e->free_page_buckets[free_bucket] = tmp->next;
364 }
365 // failing that, try the global list.
366 if (tmp == NULL && e->page_freelist != NULL) {
367 tmp = e->page_freelist;
325 e->page_freelist = tmp->next;368 e->page_freelist = tmp->next;
369 }
370 E_DEBUG("EXTSTORE: allocating new page\n");
371 // page_freelist can be empty if the only free pages are specialized and
372 // we didn't just request one.
373 if (e->page_free > 0 && tmp != NULL) {
326 tmp->next = e->page_buckets[bucket];374 tmp->next = e->page_buckets[bucket];
327 e->page_buckets[bucket] = tmp;375 e->page_buckets[bucket] = tmp;
328 tmp->active = true;376 tmp->active = true;
@@ -432,7 +480,8 @@ static void _submit_wbuf(store_engine *e, store_page *p) {
432 * new page. best if used from a background thread that can harmlessly retry.480 * new page. best if used from a background thread that can harmlessly retry.
433 */481 */
434482
435int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) {483int extstore_write_request(void *ptr, unsigned int bucket,
484 unsigned int free_bucket, obj_io *io) {
436 store_engine *e = (store_engine *)ptr;485 store_engine *e = (store_engine *)ptr;
437 store_page *p;486 store_page *p;
438 int ret = -1;487 int ret = -1;
@@ -442,7 +491,7 @@ int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) {
442 pthread_mutex_lock(&e->mutex);491 pthread_mutex_lock(&e->mutex);
443 p = e->page_buckets[bucket];492 p = e->page_buckets[bucket];
444 if (!p) {493 if (!p) {
445 p = _allocate_page(e, bucket);494 p = _allocate_page(e, bucket, free_bucket);
446 }495 }
447 pthread_mutex_unlock(&e->mutex);496 pthread_mutex_unlock(&e->mutex);
448 if (!p)497 if (!p)
@@ -456,7 +505,7 @@ int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) {
456 ((!p->wbuf || p->wbuf->full) && p->allocated >= e->page_size)) {505 ((!p->wbuf || p->wbuf->full) && p->allocated >= e->page_size)) {
457 pthread_mutex_unlock(&p->mutex);506 pthread_mutex_unlock(&p->mutex);
458 pthread_mutex_lock(&e->mutex);507 pthread_mutex_lock(&e->mutex);
459 _allocate_page(e, bucket);508 _allocate_page(e, bucket, free_bucket);
460 pthread_mutex_unlock(&e->mutex);509 pthread_mutex_unlock(&e->mutex);
461 return ret;510 return ret;
462 }511 }
@@ -510,7 +559,7 @@ void extstore_write(void *ptr, obj_io *io) {
510/* engine submit function; takes engine, item_io stack.559/* engine submit function; takes engine, item_io stack.
511 * lock io_thread context and add stack?560 * lock io_thread context and add stack?
512 * signal io thread to wake.561 * signal io thread to wake.
513 * return sucess.562 * return success.
514 */563 */
515int extstore_submit(void *ptr, obj_io *io) {564int extstore_submit(void *ptr, obj_io *io) {
516 store_engine *e = (store_engine *)ptr;565 store_engine *e = (store_engine *)ptr;
@@ -603,7 +652,7 @@ void extstore_close_page(void *ptr, unsigned int page_id, uint64_t page_version)
603652
604/* Finds an attached wbuf that can satisfy the read.653/* Finds an attached wbuf that can satisfy the read.
605 * Since wbufs can potentially be flushed to disk out of order, they are only654 * Since wbufs can potentially be flushed to disk out of order, they are only
606 * removed as the head of the list successfuly flushes to disk.655 * removed as the head of the list successfully flushes to disk.
607 */656 */
608// call with *p locked657// call with *p locked
609// FIXME: protect from reading past wbuf658// FIXME: protect from reading past wbuf
@@ -762,8 +811,14 @@ static void _free_page(store_engine *e, store_page *p) {
762 p->closed = false;811 p->closed = false;
763 p->free = true;812 p->free = true;
764 // add to page stack813 // add to page stack
765 p->next = e->page_freelist;814 // TODO: free_page_buckets first class and remove redundancy?
766 e->page_freelist = p;815 if (p->free_bucket != 0) {
816 p->next = e->free_page_buckets[p->free_bucket];
817 e->free_page_buckets[p->free_bucket] = p;
818 } else {
819 p->next = e->page_freelist;
820 e->page_freelist = p;
821 }
767 e->page_free++;822 e->page_free++;
768 pthread_mutex_unlock(&e->mutex);823 pthread_mutex_unlock(&e->mutex);
769}824}
@@ -795,7 +850,9 @@ static void *extstore_maint_thread(void *arg) {
795850
796 pthread_cond_wait(&me->cond, &me->mutex);851 pthread_cond_wait(&me->cond, &me->mutex);
797 pthread_mutex_lock(&e->mutex);852 pthread_mutex_lock(&e->mutex);
798 if (e->page_free == 0) {853 // default freelist requires at least one page free.
854 // specialized freelists fall back to default once full.
855 if (e->page_free == 0 || e->page_freelist == NULL) {
799 do_evict = true;856 do_evict = true;
800 }857 }
801 pthread_mutex_unlock(&e->mutex);858 pthread_mutex_unlock(&e->mutex);
@@ -804,6 +861,7 @@ static void *extstore_maint_thread(void *arg) {
804 for (i = 0; i < e->page_count; i++) {861 for (i = 0; i < e->page_count; i++) {
805 store_page *p = &e->pages[i];862 store_page *p = &e->pages[i];
806 pthread_mutex_lock(&p->mutex);863 pthread_mutex_lock(&p->mutex);
864 pd[p->id].free_bucket = p->free_bucket;
807 if (p->active || p->free) {865 if (p->active || p->free) {
808 pthread_mutex_unlock(&p->mutex);866 pthread_mutex_unlock(&p->mutex);
809 continue;867 continue;
@@ -812,7 +870,13 @@ static void *extstore_maint_thread(void *arg) {
812 pd[p->id].version = p->version;870 pd[p->id].version = p->version;
813 pd[p->id].bytes_used = p->bytes_used;871 pd[p->id].bytes_used = p->bytes_used;
814 pd[p->id].bucket = p->bucket;872 pd[p->id].bucket = p->bucket;
815 if (p->version < low_version) {873 // low_version/low_page are only used in the eviction
874 // scenario. when we evict, it's only to fill the default page
875 // bucket again.
876 // TODO: experiment with allowing evicting up to a single page
877 // for any specific free bucket. this is *probably* required
878 // since it could cause a load bias on default-only devices?
879 if (p->free_bucket == 0 && p->version < low_version) {
816 low_version = p->version;880 low_version = p->version;
817 low_page = i;881 low_page = i;
818 }882 }
diff --git a/extstore.h b/extstore.h
index a466562..6814415 100644
--- a/extstore.h
+++ b/extstore.h
@@ -8,6 +8,7 @@ struct extstore_page_data {
8 uint64_t version;8 uint64_t version;
9 uint64_t bytes_used;9 uint64_t bytes_used;
10 unsigned int bucket;10 unsigned int bucket;
11 unsigned int free_bucket;
11};12};
1213
13/* Pages can have objects deleted from them at any time. This creates holes14/* Pages can have objects deleted from them at any time. This creates holes
@@ -43,12 +44,23 @@ struct extstore_conf {
43 unsigned int page_size; // ideally 64-256M in size44 unsigned int page_size; // ideally 64-256M in size
44 unsigned int page_count;45 unsigned int page_count;
45 unsigned int page_buckets; // number of different writeable pages46 unsigned int page_buckets; // number of different writeable pages
47 unsigned int free_page_buckets; // buckets of dedicated pages (see code)
46 unsigned int wbuf_size; // must divide cleanly into page_size48 unsigned int wbuf_size; // must divide cleanly into page_size
47 unsigned int wbuf_count; // this might get locked to "2 per active page"49 unsigned int wbuf_count; // this might get locked to "2 per active page"
48 unsigned int io_threadcount;50 unsigned int io_threadcount;
49 unsigned int io_depth; // with normal I/O, hits locks less. req'd for AIO51 unsigned int io_depth; // with normal I/O, hits locks less. req'd for AIO
50};52};
5153
54struct extstore_conf_file {
55 unsigned int page_count;
56 char *file;
57 int fd; // internal usage
58 uint64_t offset; // internal usage
59 unsigned int bucket; // free page bucket
60 unsigned int free_bucket; // specialized free bucket
61 struct extstore_conf_file *next;
62};
63
52enum obj_io_mode {64enum obj_io_mode {
53 OBJ_IO_READ = 0,65 OBJ_IO_READ = 0,
54 OBJ_IO_WRITE,66 OBJ_IO_WRITE,
@@ -87,8 +99,8 @@ enum extstore_res {
87};99};
88100
89const char *extstore_err(enum extstore_res res);101const char *extstore_err(enum extstore_res res);
90void *extstore_init(char *fn, struct extstore_conf *cf, enum extstore_res *res);102void *extstore_init(struct extstore_conf_file *fh, struct extstore_conf *cf, enum extstore_res *res);
91int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io);103int extstore_write_request(void *ptr, unsigned int bucket, unsigned int free_bucket, obj_io *io);
92void extstore_write(void *ptr, obj_io *io);104void extstore_write(void *ptr, obj_io *io);
93int extstore_submit(void *ptr, obj_io *io);105int extstore_submit(void *ptr, obj_io *io);
94/* count are the number of objects being removed, bytes are the original106/* count are the number of objects being removed, bytes are the original
diff --git a/items.c b/items.c
index 0aefaf0..d4ce5a1 100644
--- a/items.c
+++ b/items.c
@@ -285,6 +285,13 @@ item *do_item_alloc(char *key, const size_t nkey, const unsigned int flags,
285 if (settings.use_cas) {285 if (settings.use_cas) {
286 htotal += sizeof(uint64_t);286 htotal += sizeof(uint64_t);
287 }287 }
288#ifdef NEED_ALIGN
289 // header chunk needs to be padded on some systems
290 int remain = htotal % 8;
291 if (remain != 0) {
292 htotal += 8 - remain;
293 }
294#endif
288 hdr_id = slabs_clsid(htotal);295 hdr_id = slabs_clsid(htotal);
289 it = do_item_alloc_pull(htotal, hdr_id);296 it = do_item_alloc_pull(htotal, hdr_id);
290 /* setting ITEM_CHUNKED is fine here because we aren't LINKED yet. */297 /* setting ITEM_CHUNKED is fine here because we aren't LINKED yet. */
@@ -336,7 +343,7 @@ item *do_item_alloc(char *key, const size_t nkey, const unsigned int flags,
336343
337 /* Initialize internal chunk. */344 /* Initialize internal chunk. */
338 if (it->it_flags & ITEM_CHUNKED) {345 if (it->it_flags & ITEM_CHUNKED) {
339 item_chunk *chunk = (item_chunk *) ITEM_data(it);346 item_chunk *chunk = (item_chunk *) ITEM_schunk(it);
340347
341 chunk->next = 0;348 chunk->next = 0;
342 chunk->prev = 0;349 chunk->prev = 0;
@@ -1538,7 +1545,6 @@ static void *lru_maintainer_thread(void *arg) {
1538 void *storage = arg;1545 void *storage = arg;
1539 if (storage != NULL)1546 if (storage != NULL)
1540 sam = &slab_automove_extstore;1547 sam = &slab_automove_extstore;
1541 int x;
1542#endif1548#endif
1543 int i;1549 int i;
1544 useconds_t to_sleep = MIN_LRU_MAINTAINER_SLEEP;1550 useconds_t to_sleep = MIN_LRU_MAINTAINER_SLEEP;
@@ -1592,20 +1598,6 @@ static void *lru_maintainer_thread(void *arg) {
1592 }1598 }
15931599
1594 int did_moves = lru_maintainer_juggle(i);1600 int did_moves = lru_maintainer_juggle(i);
1595#ifdef EXTSTORE
1596 // Deeper loop to speed up pushing to storage.
1597 if (storage) {
1598 for (x = 0; x < 500; x++) {
1599 int found;
1600 found = lru_maintainer_store(storage, i);
1601 if (found) {
1602 did_moves += found;
1603 } else {
1604 break;
1605 }
1606 }
1607 }
1608#endif
1609 if (did_moves == 0) {1601 if (did_moves == 0) {
1610 if (backoff_juggles[i] != 0) {1602 if (backoff_juggles[i] != 0) {
1611 backoff_juggles[i] += backoff_juggles[i] / 8;1603 backoff_juggles[i] += backoff_juggles[i] / 8;
diff --git a/linux_priv.c b/linux_priv.c
index 04155dd..cc9aef3 100644
--- a/linux_priv.c
+++ b/linux_priv.c
@@ -2,11 +2,11 @@
2#include <seccomp.h>2#include <seccomp.h>
3#include <errno.h>3#include <errno.h>
4#include <stdlib.h>4#include <stdlib.h>
5#include <sys/ioctl.h>
5#include "memcached.h"6#include "memcached.h"
67
7// In the future when the system is more tested this could be switched8// If anything crosses the policy, kill the process.
8// to SCMP_ACT_KILL instead.9#define DENY_ACTION SCMP_ACT_KILL
9#define DENY_ACTION SCMP_ACT_ERRNO(EACCES)
1010
11void drop_privileges(void) {11void drop_privileges(void) {
12 scmp_filter_ctx ctx = seccomp_init(DENY_ACTION);12 scmp_filter_ctx ctx = seccomp_init(DENY_ACTION);
@@ -16,24 +16,41 @@ void drop_privileges(void) {
1616
17 int rc = 0;17 int rc = 0;
18 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0);18 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0);
19 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0);
19 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);20 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);
20 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0);21 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0);
22 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_pwait), 0);
21 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept4), 0);23 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept4), 0);
22 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept), 0);24 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept), 0);
23 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);25 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
26 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 0);
24 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0);27 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0);
25 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mmap), 0);28 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mmap), 0);
26 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0);29 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0);
27 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(shmctl), 0);30 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(shmctl), 0);
28 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);31 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
32 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0);
33 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TIOCGWINSZ));
34 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TCGETS));
35
36#if defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC)
37 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clock_gettime), 0);
38#endif
2939
30#ifdef MEMCACHED_DEBUG40#ifdef MEMCACHED_DEBUG
31 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0);41 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0);
32 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0);42 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0);
33 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);43 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
44 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(readv), 0);
34 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0);45 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0);
35 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0);46 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0);
36 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0);47 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0);
48
49 if (settings.relaxed_privileges) {
50 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(openat), 0);
51 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mkdir), 0);
52 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(access), 0);
53 }
37#endif54#endif
3855
39 if (rc != 0) {56 if (rc != 0) {
@@ -45,8 +62,13 @@ void drop_privileges(void) {
45 goto fail;62 goto fail;
46 }63 }
4764
65 seccomp_release(ctx);
66 return;
67
48fail:68fail:
49 seccomp_release(ctx);69 seccomp_release(ctx);
70 fprintf(stderr, "Failed to set a seccomp profile on the main thread\n");
71 exit(EXIT_FAILURE);
50}72}
5173
52void drop_worker_privileges(void) {74void drop_worker_privileges(void) {
@@ -57,10 +79,14 @@ void drop_worker_privileges(void) {
5779
58 int rc = 0;80 int rc = 0;
59 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0);81 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0);
82 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0);
60 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);83 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0);
61 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0);84 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0);
85 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_pwait), 0);
62 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_ctl), 0);86 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_ctl), 0);
87 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(poll), 0);
63 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);88 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0);
89 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(readv), 0);
64 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mprotect), 0);90 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mprotect), 0);
65 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpeername), 0);91 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpeername), 0);
66 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0);92 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0);
@@ -70,6 +96,8 @@ void drop_worker_privileges(void) {
70 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mremap), 0);96 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mremap), 0);
71 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0);97 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0);
72 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(recvfrom), 0);98 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(recvfrom), 0);
99 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0);
100 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TIOCGWINSZ));
73101
74 // for spawning the LRU crawler102 // for spawning the LRU crawler
75 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clone), 0);103 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clone), 0);
@@ -83,20 +111,29 @@ void drop_worker_privileges(void) {
83111
84 if (settings.shutdown_command) {112 if (settings.shutdown_command) {
85 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tgkill), 0);113 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tgkill), 0);
114 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tkill), 0);
86 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);115 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0);
87 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0);116 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0);
88 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0);
89 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0);117 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0);
90 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(gettid), 0);118 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(gettid), 0);
91 }119 }
92120
93 if (settings.relaxed_privileges) {121 if (settings.relaxed_privileges) {
122 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(openat), 0);
123 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mkdir), 0);
124 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(access), 0);
94 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0);125 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0);
95 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0);126 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0);
96 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0);127 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0);
97 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);128 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0);
129 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 0);
98 } else {130 } else {
131 // stdout
99 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 1));132 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 1));
133 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 1, SCMP_A0(SCMP_CMP_EQ, 1));
134 // stderr
135 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 2));
136 rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 1, SCMP_A0(SCMP_CMP_EQ, 2));
100 }137 }
101138
102 if (rc != 0) {139 if (rc != 0) {
@@ -108,6 +145,11 @@ void drop_worker_privileges(void) {
108 goto fail;145 goto fail;
109 }146 }
110147
148 seccomp_release(ctx);
149 return;
150
111fail:151fail:
112 seccomp_release(ctx);152 seccomp_release(ctx);
153 fprintf(stderr, "Failed to set a seccomp profile on a worker thread\n");
154 exit(EXIT_FAILURE);
113}155}
diff --git a/logger.c b/logger.c
index 7af9917..1322d7d 100644
--- a/logger.c
+++ b/logger.c
@@ -363,7 +363,7 @@ static int logger_thread_read(logger *l, struct logger_stats *ls) {
363 } else {363 } else {
364 logger_thread_write_entry(e, ls, scratch, scratch_len);364 logger_thread_write_entry(e, ls, scratch, scratch_len);
365 }365 }
366 pos += sizeof(logentry) + e->size;366 pos += sizeof(logentry) + e->size + e->pad;
367 }367 }
368 assert(pos <= size);368 assert(pos <= size);
369369
@@ -699,8 +699,9 @@ enum logger_ret_type logger_log(logger *l, const enum log_entry_type event, cons
699 l->dropped++;699 l->dropped++;
700 return LOGGER_RET_NOSPACE;700 return LOGGER_RET_NOSPACE;
701 }701 }
702 e->gid = logger_get_gid();
703 e->event = d->subtype;702 e->event = d->subtype;
703 e->pad = 0;
704 e->gid = logger_get_gid();
704 /* TODO: Could pass this down as an argument now that we're using705 /* TODO: Could pass this down as an argument now that we're using
705 * LOGGER_LOG() macro.706 * LOGGER_LOG() macro.
706 */707 */
@@ -751,11 +752,19 @@ enum logger_ret_type logger_log(logger *l, const enum log_entry_type event, cons
751 rel_time_t sttl = va_arg(ap, rel_time_t);752 rel_time_t sttl = va_arg(ap, rel_time_t);
752 uint8_t sclsid = va_arg(ap, int);753 uint8_t sclsid = va_arg(ap, int);
753 _logger_log_item_store(e, status, comm, skey, snkey, sttl, sclsid);754 _logger_log_item_store(e, status, comm, skey, snkey, sttl, sclsid);
755 va_end(ap);
754 break;756 break;
755 }757 }
756758
759#ifdef NEED_ALIGN
760 /* Need to ensure *next* request is aligned. */
761 if (sizeof(logentry) + e->size % 8 != 0) {
762 e->pad = 8 - (sizeof(logentry) + e->size % 8);
763 }
764#endif
765
757 /* Push pointer forward by the actual amount required */766 /* Push pointer forward by the actual amount required */
758 if (bipbuf_push(buf, (sizeof(logentry) + e->size)) == 0) {767 if (bipbuf_push(buf, (sizeof(logentry) + e->size + e->pad)) == 0) {
759 fprintf(stderr, "LOGGER: Failed to bipbuf push a text entry\n");768 fprintf(stderr, "LOGGER: Failed to bipbuf push a text entry\n");
760 pthread_mutex_unlock(&l->mutex);769 pthread_mutex_unlock(&l->mutex);
761 return LOGGER_RET_ERR;770 return LOGGER_RET_ERR;
diff --git a/logger.h b/logger.h
index 3d4c44c..730c86f 100644
--- a/logger.h
+++ b/logger.h
@@ -100,6 +100,7 @@ struct logentry_item_store {
100100
101typedef struct _logentry {101typedef struct _logentry {
102 enum log_entry_subtype event;102 enum log_entry_subtype event;
103 uint8_t pad;
103 uint16_t eflags;104 uint16_t eflags;
104 uint64_t gid;105 uint64_t gid;
105 struct timeval tv; /* not monotonic! */106 struct timeval tv; /* not monotonic! */
diff --git a/memcached.c b/memcached.c
index 7178666..621b317 100644
--- a/memcached.c
+++ b/memcached.c
@@ -271,7 +271,7 @@ static void settings_init(void) {
271 settings.crawls_persleep = 1000;271 settings.crawls_persleep = 1000;
272 settings.logger_watcher_buf_size = LOGGER_WATCHER_BUF_SIZE;272 settings.logger_watcher_buf_size = LOGGER_WATCHER_BUF_SIZE;
273 settings.logger_buf_size = LOGGER_BUF_SIZE;273 settings.logger_buf_size = LOGGER_BUF_SIZE;
274 settings.drop_privileges = true;274 settings.drop_privileges = false;
275#ifdef MEMCACHED_DEBUG275#ifdef MEMCACHED_DEBUG
276 settings.relaxed_privileges = false;276 settings.relaxed_privileges = false;
277#endif277#endif
@@ -612,6 +612,7 @@ conn *conn_new(const int sfd, enum conn_states init_state,
612 c->iovused = 0;612 c->iovused = 0;
613 c->msgcurr = 0;613 c->msgcurr = 0;
614 c->msgused = 0;614 c->msgused = 0;
615 c->sasl_started = false;
615 c->authenticated = false;616 c->authenticated = false;
616 c->last_cmd_time = current_time; /* initialize for idle kicker */617 c->last_cmd_time = current_time; /* initialize for idle kicker */
617#ifdef EXTSTORE618#ifdef EXTSTORE
@@ -648,8 +649,19 @@ static void recache_or_free(conn *c, io_wrap *wrap) {
648 item *it;649 item *it;
649 it = (item *)wrap->io.buf;650 it = (item *)wrap->io.buf;
650 bool do_free = true;651 bool do_free = true;
651 // If request was ultimately a miss, unlink the header.652 if (wrap->active) {
652 if (wrap->miss) {653 // If request never dispatched, free the read buffer but leave the
654 // item header alone.
655 do_free = false;
656 size_t ntotal = ITEM_ntotal(wrap->hdr_it);
657 slabs_free(it, ntotal, slabs_clsid(ntotal));
658 c->io_wrapleft--;
659 assert(c->io_wrapleft >= 0);
660 pthread_mutex_lock(&c->thread->stats.mutex);
661 c->thread->stats.get_aborted_extstore++;
662 pthread_mutex_unlock(&c->thread->stats.mutex);
663 } else if (wrap->miss) {
664 // If request was ultimately a miss, unlink the header.
653 do_free = false;665 do_free = false;
654 size_t ntotal = ITEM_ntotal(wrap->hdr_it);666 size_t ntotal = ITEM_ntotal(wrap->hdr_it);
655 item_unlink(wrap->hdr_it);667 item_unlink(wrap->hdr_it);
@@ -1020,7 +1032,7 @@ static int add_iov(conn *c, const void *buf, int len) {
10201032
1021static int add_chunked_item_iovs(conn *c, item *it, int len) {1033static int add_chunked_item_iovs(conn *c, item *it, int len) {
1022 assert(it->it_flags & ITEM_CHUNKED);1034 assert(it->it_flags & ITEM_CHUNKED);
1023 item_chunk *ch = (item_chunk *) ITEM_data(it);1035 item_chunk *ch = (item_chunk *) ITEM_schunk(it);
1024 while (ch) {1036 while (ch) {
1025 int todo = (len > ch->used) ? ch->used : len;1037 int todo = (len > ch->used) ? ch->used : len;
1026 if (add_iov(c, ch->data, todo) != 0) {1038 if (add_iov(c, ch->data, todo) != 0) {
@@ -1640,13 +1652,8 @@ static void process_bin_get_or_touch(conn *c) {
1640 rsp->message.header.response.cas = htonll(ITEM_get_cas(it));1652 rsp->message.header.response.cas = htonll(ITEM_get_cas(it));
16411653
1642 // add the flags1654 // add the flags
1643 if (settings.inline_ascii_response) {1655 FLAGS_CONV(settings.inline_ascii_response, it, rsp->message.body.flags);
1644 rsp->message.body.flags = htonl(strtoul(ITEM_suffix(it), NULL, 10));1656 rsp->message.body.flags = htonl(rsp->message.body.flags);
1645 } else if (it->nsuffix > 0) {
1646 rsp->message.body.flags = htonl(*((uint32_t *)ITEM_suffix(it)));
1647 } else {
1648 rsp->message.body.flags = 0;
1649 }
1650 add_iov(c, &rsp->message.body, sizeof(rsp->message.body));1657 add_iov(c, &rsp->message.body, sizeof(rsp->message.body));
16511658
1652 if (should_return_key) {1659 if (should_return_key) {
@@ -1663,10 +1670,14 @@ static void process_bin_get_or_touch(conn *c) {
1663 iovcnt = 3;1670 iovcnt = 3;
1664 iovst = c->iovused - 2;1671 iovst = c->iovused - 2;
1665 }1672 }
1666 // FIXME: this can return an error, but code flow doesn't1673
1667 // allow bailing here.1674 if (_get_extstore(c, it, iovst, iovcnt) != 0) {
1668 if (_get_extstore(c, it, iovst, iovcnt) != 0)1675 pthread_mutex_lock(&c->thread->stats.mutex);
1676 c->thread->stats.get_oom_extstore++;
1677 pthread_mutex_unlock(&c->thread->stats.mutex);
1678
1669 failed = true;1679 failed = true;
1680 }
1670 } else if ((it->it_flags & ITEM_CHUNKED) == 0) {1681 } else if ((it->it_flags & ITEM_CHUNKED) == 0) {
1671 add_iov(c, ITEM_data(it), it->nbytes - 2);1682 add_iov(c, ITEM_data(it), it->nbytes - 2);
1672 } else {1683 } else {
@@ -1686,10 +1697,11 @@ static void process_bin_get_or_touch(conn *c) {
1686 c->write_and_go = conn_new_cmd;1697 c->write_and_go = conn_new_cmd;
1687 /* Remember this command so we can garbage collect it later */1698 /* Remember this command so we can garbage collect it later */
1688#ifdef EXTSTORE1699#ifdef EXTSTORE
1689 if ((it->it_flags & ITEM_HDR) == 0) {1700 if ((it->it_flags & ITEM_HDR) != 0 && should_return_value) {
1690 c->item = it;1701 // Only have extstore clean if header and returning value.
1691 } else {
1692 c->item = NULL;1702 c->item = NULL;
1703 } else {
1704 c->item = it;
1693 }1705 }
1694#else1706#else
1695 c->item = it;1707 c->item = it;
@@ -2105,8 +2117,16 @@ static void process_bin_complete_sasl_auth(conn *c) {
2105 result = sasl_server_start(c->sasl_conn, mech,2117 result = sasl_server_start(c->sasl_conn, mech,
2106 challenge, vlen,2118 challenge, vlen,
2107 &out, &outlen);2119 &out, &outlen);
2120 c->sasl_started = (result == SASL_OK || result == SASL_CONTINUE);
2108 break;2121 break;
2109 case PROTOCOL_BINARY_CMD_SASL_STEP:2122 case PROTOCOL_BINARY_CMD_SASL_STEP:
2123 if (!c->sasl_started) {
2124 if (settings.verbose) {
2125 fprintf(stderr, "%d: SASL_STEP called but sasl_server_start "
2126 "not called for this connection!\n", c->sfd);
2127 }
2128 break;
2129 }
2110 result = sasl_server_step(c->sasl_conn,2130 result = sasl_server_step(c->sasl_conn,
2111 challenge, vlen,2131 challenge, vlen,
2112 &out, &outlen);2132 &out, &outlen);
@@ -2465,7 +2485,15 @@ static void process_bin_update(conn *c) {
2465 }2485 }
24662486
2467 c->item = it;2487 c->item = it;
2488#ifdef NEED_ALIGN
2489 if (it->it_flags & ITEM_CHUNKED) {
2490 c->ritem = ITEM_schunk(it);
2491 } else {
2492 c->ritem = ITEM_data(it);
2493 }
2494#else
2468 c->ritem = ITEM_data(it);2495 c->ritem = ITEM_data(it);
2496#endif
2469 c->rlbytes = vlen;2497 c->rlbytes = vlen;
2470 conn_set_state(c, conn_nread);2498 conn_set_state(c, conn_nread);
2471 c->substate = bin_read_set_value;2499 c->substate = bin_read_set_value;
@@ -2520,7 +2548,15 @@ static void process_bin_append_prepend(conn *c) {
2520 }2548 }
25212549
2522 c->item = it;2550 c->item = it;
2551#ifdef NEED_ALIGN
2552 if (it->it_flags & ITEM_CHUNKED) {
2553 c->ritem = ITEM_schunk(it);
2554 } else {
2555 c->ritem = ITEM_data(it);
2556 }
2557#else
2523 c->ritem = ITEM_data(it);2558 c->ritem = ITEM_data(it);
2559#endif
2524 c->rlbytes = vlen;2560 c->rlbytes = vlen;
2525 conn_set_state(c, conn_nread);2561 conn_set_state(c, conn_nread);
2526 c->substate = bin_read_set_value;2562 c->substate = bin_read_set_value;
@@ -2681,7 +2717,7 @@ static void complete_nread(conn *c) {
2681/* Destination must always be chunked */2717/* Destination must always be chunked */
2682/* This should be part of item.c */2718/* This should be part of item.c */
2683static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) {2719static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) {
2684 item_chunk *dch = (item_chunk *) ITEM_data(d_it);2720 item_chunk *dch = (item_chunk *) ITEM_schunk(d_it);
2685 /* Advance dch until we find free space */2721 /* Advance dch until we find free space */
2686 while (dch->size == dch->used) {2722 while (dch->size == dch->used) {
2687 if (dch->next) {2723 if (dch->next) {
@@ -2693,7 +2729,7 @@ static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) {
26932729
2694 if (s_it->it_flags & ITEM_CHUNKED) {2730 if (s_it->it_flags & ITEM_CHUNKED) {
2695 int remain = len;2731 int remain = len;
2696 item_chunk *sch = (item_chunk *) ITEM_data(s_it);2732 item_chunk *sch = (item_chunk *) ITEM_schunk(s_it);
2697 int copied = 0;2733 int copied = 0;
2698 /* Fills dch's to capacity, not straight copy sch in case data is2734 /* Fills dch's to capacity, not straight copy sch in case data is
2699 * being added or removed (ie append/prepend)2735 * being added or removed (ie append/prepend)
@@ -2857,15 +2893,7 @@ enum store_item_type do_store_item(item *it, int comm, conn *c, const uint32_t h
2857 if (stored == NOT_STORED) {2893 if (stored == NOT_STORED) {
2858 /* we have it and old_it here - alloc memory to hold both */2894 /* we have it and old_it here - alloc memory to hold both */
2859 /* flags was already lost - so recover them from ITEM_suffix(it) */2895 /* flags was already lost - so recover them from ITEM_suffix(it) */
28602896 FLAGS_CONV(settings.inline_ascii_response, old_it, flags);
2861 if (settings.inline_ascii_response) {
2862 flags = (uint32_t) strtoul(ITEM_suffix(old_it), (char **) NULL, 10);
2863 } else if (old_it->nsuffix > 0) {
2864 flags = *((uint32_t *)ITEM_suffix(old_it));
2865 } else {
2866 flags = 0;
2867 }
2868
2869 new_it = do_item_alloc(key, it->nkey, flags, old_it->exptime, it->nbytes + old_it->nbytes - 2 /* CRLF */);2897 new_it = do_item_alloc(key, it->nkey, flags, old_it->exptime, it->nbytes + old_it->nbytes - 2 /* CRLF */);
28702898
2871 /* copy data from it and old_it to new_it */2899 /* copy data from it and old_it to new_it */
@@ -3105,6 +3133,8 @@ static void server_stats(ADD_STAT add_stats, conn *c) {
3105#ifdef EXTSTORE3133#ifdef EXTSTORE
3106 if (c->thread->storage) {3134 if (c->thread->storage) {
3107 APPEND_STAT("get_extstore", "%llu", (unsigned long long)thread_stats.get_extstore);3135 APPEND_STAT("get_extstore", "%llu", (unsigned long long)thread_stats.get_extstore);
3136 APPEND_STAT("get_aborted_extstore", "%llu", (unsigned long long)thread_stats.get_aborted_extstore);
3137 APPEND_STAT("get_oom_extstore", "%llu", (unsigned long long)thread_stats.get_oom_extstore);
3108 APPEND_STAT("recache_from_extstore", "%llu", (unsigned long long)thread_stats.recache_from_extstore);3138 APPEND_STAT("recache_from_extstore", "%llu", (unsigned long long)thread_stats.recache_from_extstore);
3109 APPEND_STAT("miss_from_extstore", "%llu", (unsigned long long)thread_stats.miss_from_extstore);3139 APPEND_STAT("miss_from_extstore", "%llu", (unsigned long long)thread_stats.miss_from_extstore);
3110 APPEND_STAT("badcrc_from_extstore", "%llu", (unsigned long long)thread_stats.badcrc_from_extstore);3140 APPEND_STAT("badcrc_from_extstore", "%llu", (unsigned long long)thread_stats.badcrc_from_extstore);
@@ -3242,6 +3272,9 @@ static void process_stat_settings(ADD_STAT add_stats, void *c) {
3242 APPEND_STAT("worker_logbuf_size", "%u", settings.logger_buf_size);3272 APPEND_STAT("worker_logbuf_size", "%u", settings.logger_buf_size);
3243 APPEND_STAT("track_sizes", "%s", item_stats_sizes_status() ? "yes" : "no");3273 APPEND_STAT("track_sizes", "%s", item_stats_sizes_status() ? "yes" : "no");
3244 APPEND_STAT("inline_ascii_response", "%s", settings.inline_ascii_response ? "yes" : "no");3274 APPEND_STAT("inline_ascii_response", "%s", settings.inline_ascii_response ? "yes" : "no");
3275#ifdef HAVE_DROP_PRIVILEGES
3276 APPEND_STAT("drop_privileges", "%s", settings.drop_privileges ? "yes" : "no");
3277#endif
3245#ifdef EXTSTORE3278#ifdef EXTSTORE
3246 APPEND_STAT("ext_item_size", "%u", settings.ext_item_size);3279 APPEND_STAT("ext_item_size", "%u", settings.ext_item_size);
3247 APPEND_STAT("ext_item_age", "%u", settings.ext_item_age);3280 APPEND_STAT("ext_item_age", "%u", settings.ext_item_age);
@@ -3384,6 +3417,8 @@ static void process_extstore_stats(ADD_STAT add_stats, conn *c) {
3384 (unsigned long long) st.page_data[i].bytes_used);3417 (unsigned long long) st.page_data[i].bytes_used);
3385 APPEND_NUM_STAT(i, "bucket", "%u",3418 APPEND_NUM_STAT(i, "bucket", "%u",
3386 st.page_data[i].bucket);3419 st.page_data[i].bucket);
3420 APPEND_NUM_STAT(i, "free_bucket", "%u",
3421 st.page_data[i].free_bucket);
3387 }3422 }
3388}3423}
3389#endif3424#endif
@@ -3583,14 +3618,14 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) {
3583 // item is chunked, crc the iov's3618 // item is chunked, crc the iov's
3584 if (io->iov != NULL) {3619 if (io->iov != NULL) {
3585 // first iov is the header, which we don't use beyond crc3620 // first iov is the header, which we don't use beyond crc
3586 crc2 = crc32c(0, (char *)io->iov[0].iov_base+32, io->iov[0].iov_len-32);3621 crc2 = crc32c(0, (char *)io->iov[0].iov_base+STORE_OFFSET, io->iov[0].iov_len-STORE_OFFSET);
3587 // make sure it's not sent. hack :(3622 // make sure it's not sent. hack :(
3588 io->iov[0].iov_len = 0;3623 io->iov[0].iov_len = 0;
3589 for (x = 1; x < io->iovcnt; x++) {3624 for (x = 1; x < io->iovcnt; x++) {
3590 crc2 = crc32c(crc2, (char *)io->iov[x].iov_base, io->iov[x].iov_len);3625 crc2 = crc32c(crc2, (char *)io->iov[x].iov_base, io->iov[x].iov_len);
3591 }3626 }
3592 } else {3627 } else {
3593 crc2 = crc32c(0, (char *)read_it+32, io->len-32);3628 crc2 = crc32c(0, (char *)read_it+STORE_OFFSET, io->len-STORE_OFFSET);
3594 }3629 }
35953630
3596 if (crc != crc2) {3631 if (crc != crc2) {
@@ -3639,6 +3674,7 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) {
3639 }3674 }
3640 }3675 }
3641 }3676 }
3677 wrap->miss = false;
3642 // iov_len is already set3678 // iov_len is already set
3643 // TODO: Should do that here instead and cuddle in the wrap object3679 // TODO: Should do that here instead and cuddle in the wrap object
3644 }3680 }
@@ -3657,23 +3693,20 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) {
36573693
3658// FIXME: This completely breaks UDP support.3694// FIXME: This completely breaks UDP support.
3659static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {3695static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
3696#ifdef NEED_ALIGN
3697 item_hdr hdr;
3698 memcpy(&hdr, ITEM_data(it), sizeof(hdr));
3699#else
3660 item_hdr *hdr = (item_hdr *)ITEM_data(it);3700 item_hdr *hdr = (item_hdr *)ITEM_data(it);
3701#endif
3661 size_t ntotal = ITEM_ntotal(it);3702 size_t ntotal = ITEM_ntotal(it);
3662 unsigned int clsid = slabs_clsid(ntotal);3703 unsigned int clsid = slabs_clsid(ntotal);
3663 item *new_it;3704 item *new_it;
3664 bool chunked = false;3705 bool chunked = false;
3665 if (ntotal > settings.slab_chunk_size_max) {3706 if (ntotal > settings.slab_chunk_size_max) {
3666 // Pull a chunked item header.3707 // Pull a chunked item header.
3667 // FIXME: make a func. used in several places.
3668 uint32_t flags;3708 uint32_t flags;
3669 if (settings.inline_ascii_response) {3709 FLAGS_CONV(settings.inline_ascii_response, it, flags);
3670 flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10);
3671 } else if (it->nsuffix > 0) {
3672 flags = *((uint32_t *)ITEM_suffix(it));
3673 } else {
3674 flags = 0;
3675 }
3676
3677 new_it = item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, it->nbytes);3710 new_it = item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, it->nbytes);
3678 assert(new_it == NULL || (new_it->it_flags & ITEM_CHUNKED));3711 assert(new_it == NULL || (new_it->it_flags & ITEM_CHUNKED));
3679 chunked = true;3712 chunked = true;
@@ -3702,13 +3735,12 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
3702 if (chunked) {3735 if (chunked) {
3703 unsigned int ciovcnt = 1;3736 unsigned int ciovcnt = 1;
3704 size_t remain = new_it->nbytes;3737 size_t remain = new_it->nbytes;
3705 item_chunk *chunk = (item_chunk *) ITEM_data(new_it);3738 item_chunk *chunk = (item_chunk *) ITEM_schunk(new_it);
3706 io->io.iov = &c->iov[c->iovused];3739 io->io.iov = &c->iov[c->iovused];
3707 // fill the header so we can get the full data + crc back.3740 // fill the header so we can get the full data + crc back.
3708 add_iov(c, new_it, ITEM_ntotal(new_it) - new_it->nbytes);3741 add_iov(c, new_it, ITEM_ntotal(new_it) - new_it->nbytes);
3709 while (remain > 0) {3742 while (remain > 0) {
3710 chunk = do_item_alloc_chunk(chunk, remain);3743 chunk = do_item_alloc_chunk(chunk, remain);
3711 // TODO: counter bump
3712 if (chunk == NULL) {3744 if (chunk == NULL) {
3713 item_remove(new_it);3745 item_remove(new_it);
3714 do_cache_free(c->thread->io_cache, io);3746 do_cache_free(c->thread->io_cache, io);
@@ -3745,9 +3777,15 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
3745 io->io.data = (void *)io;3777 io->io.data = (void *)io;
37463778
3747 // Now, fill in io->io based on what was in our header.3779 // Now, fill in io->io based on what was in our header.
3780#ifdef NEED_ALIGN
3781 io->io.page_version = hdr.page_version;
3782 io->io.page_id = hdr.page_id;
3783 io->io.offset = hdr.offset;
3784#else
3748 io->io.page_version = hdr->page_version;3785 io->io.page_version = hdr->page_version;
3749 io->io.page_id = hdr->page_id;3786 io->io.page_id = hdr->page_id;
3750 io->io.offset = hdr->offset;3787 io->io.offset = hdr->offset;
3788#endif
3751 io->io.len = ntotal;3789 io->io.len = ntotal;
3752 io->io.mode = OBJ_IO_READ;3790 io->io.mode = OBJ_IO_READ;
3753 io->io.cb = _get_extstore_cb;3791 io->io.cb = _get_extstore_cb;
@@ -3763,8 +3801,6 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) {
3763 return 0;3801 return 0;
3764}3802}
3765#endif3803#endif
3766// FIXME: the 'breaks' around memory malloc's should break all the way down,
3767// fill ileft/suffixleft, then run conn_releaseitems()
3768/* ntokens is overwritten here... shrug.. */3804/* ntokens is overwritten here... shrug.. */
3769static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, bool return_cas, bool should_touch) {3805static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, bool return_cas, bool should_touch) {
3770 char *key;3806 char *key;
@@ -3776,6 +3812,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3776 char *suffix;3812 char *suffix;
3777 int32_t exptime_int = 0;3813 int32_t exptime_int = 0;
3778 rel_time_t exptime = 0;3814 rel_time_t exptime = 0;
3815 bool fail_length = false;
3779 assert(c != NULL);3816 assert(c != NULL);
37803817
3781 if (should_touch) {3818 if (should_touch) {
@@ -3795,14 +3832,8 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3795 nkey = key_token->length;3832 nkey = key_token->length;
37963833
3797 if (nkey > KEY_MAX_LENGTH) {3834 if (nkey > KEY_MAX_LENGTH) {
3798 out_string(c, "CLIENT_ERROR bad command line format");3835 fail_length = true;
3799 while (i-- > 0) {3836 goto stop;
3800 item_remove(*(c->ilist + i));
3801 if (return_cas || !settings.inline_ascii_response) {
3802 do_cache_free(c->thread->suffix_cache, *(c->suffixlist + i));
3803 }
3804 }
3805 return;
3806 }3837 }
38073838
3808 it = limited_get(key, nkey, c, exptime, should_touch);3839 it = limited_get(key, nkey, c, exptime, should_touch);
@@ -3812,7 +3843,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3812 if (it) {3843 if (it) {
3813 if (_ascii_get_expand_ilist(c, i) != 0) {3844 if (_ascii_get_expand_ilist(c, i) != 0) {
3814 item_remove(it);3845 item_remove(it);
3815 break; // FIXME: Should bail down to error.3846 goto stop;
3816 }3847 }
38173848
3818 /*3849 /*
@@ -3831,7 +3862,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3831 suffix = _ascii_get_suffix_buf(c, si);3862 suffix = _ascii_get_suffix_buf(c, si);
3832 if (suffix == NULL) {3863 if (suffix == NULL) {
3833 item_remove(it);3864 item_remove(it);
3834 break;3865 goto stop;
3835 }3866 }
3836 si++;3867 si++;
3837 nbytes = it->nbytes;3868 nbytes = it->nbytes;
@@ -3842,13 +3873,17 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3842 add_iov(c, suffix, suffix_len) != 0)3873 add_iov(c, suffix, suffix_len) != 0)
3843 {3874 {
3844 item_remove(it);3875 item_remove(it);
3845 break;3876 goto stop;
3846 }3877 }
3847#ifdef EXTSTORE3878#ifdef EXTSTORE
3848 if (it->it_flags & ITEM_HDR) {3879 if (it->it_flags & ITEM_HDR) {
3849 if (_get_extstore(c, it, c->iovused-3, 4) != 0) {3880 if (_get_extstore(c, it, c->iovused-3, 4) != 0) {
3881 pthread_mutex_lock(&c->thread->stats.mutex);
3882 c->thread->stats.get_oom_extstore++;
3883 pthread_mutex_unlock(&c->thread->stats.mutex);
3884
3850 item_remove(it);3885 item_remove(it);
3851 break;3886 goto stop;
3852 }3887 }
3853 } else if ((it->it_flags & ITEM_CHUNKED) == 0) {3888 } else if ((it->it_flags & ITEM_CHUNKED) == 0) {
3854#else3889#else
@@ -3857,7 +3892,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3857 add_iov(c, ITEM_data(it), it->nbytes);3892 add_iov(c, ITEM_data(it), it->nbytes);
3858 } else if (add_chunked_item_iovs(c, it, it->nbytes) != 0) {3893 } else if (add_chunked_item_iovs(c, it, it->nbytes) != 0) {
3859 item_remove(it);3894 item_remove(it);
3860 break;3895 goto stop;
3861 }3896 }
3862 }3897 }
3863 else3898 else
@@ -3868,19 +3903,19 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3868 add_iov(c, ITEM_key(it), it->nkey) != 0)3903 add_iov(c, ITEM_key(it), it->nkey) != 0)
3869 {3904 {
3870 item_remove(it);3905 item_remove(it);
3871 break;3906 goto stop;
3872 }3907 }
3873 if ((it->it_flags & ITEM_CHUNKED) == 0)3908 if ((it->it_flags & ITEM_CHUNKED) == 0)
3874 {3909 {
3875 if (add_iov(c, ITEM_suffix(it), it->nsuffix + it->nbytes) != 0)3910 if (add_iov(c, ITEM_suffix(it), it->nsuffix + it->nbytes) != 0)
3876 {3911 {
3877 item_remove(it);3912 item_remove(it);
3878 break;3913 goto stop;
3879 }3914 }
3880 } else if (add_iov(c, ITEM_suffix(it), it->nsuffix) != 0 ||3915 } else if (add_iov(c, ITEM_suffix(it), it->nsuffix) != 0 ||
3881 add_chunked_item_iovs(c, it, it->nbytes) != 0) {3916 add_chunked_item_iovs(c, it, it->nbytes) != 0) {
3882 item_remove(it);3917 item_remove(it);
3883 break;3918 goto stop;
3884 }3919 }
3885 }3920 }
38863921
@@ -3940,6 +3975,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3940 }3975 }
39413976
3942 } while(key_token->value != NULL);3977 } while(key_token->value != NULL);
3978stop:
39433979
3944 c->icurr = c->ilist;3980 c->icurr = c->ilist;
3945 c->ileft = i;3981 c->ileft = i;
@@ -3958,7 +3994,12 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens,
3958 */3994 */
3959 if (key_token->value != NULL || add_iov(c, "END\r\n", 5) != 03995 if (key_token->value != NULL || add_iov(c, "END\r\n", 5) != 0
3960 || (IS_UDP(c->transport) && build_udp_headers(c) != 0)) {3996 || (IS_UDP(c->transport) && build_udp_headers(c) != 0)) {
3961 out_of_memory(c, "SERVER_ERROR out of memory writing get response");3997 if (fail_length) {
3998 out_string(c, "CLIENT_ERROR bad command line format");
3999 } else {
4000 out_of_memory(c, "SERVER_ERROR out of memory writing get response");
4001 }
4002 conn_release_items(c);
3962 }4003 }
3963 else {4004 else {
3964 conn_set_state(c, conn_mwrite);4005 conn_set_state(c, conn_mwrite);
@@ -4055,7 +4096,15 @@ static void process_update_command(conn *c, token_t *tokens, const size_t ntoken
4055 ITEM_set_cas(it, req_cas_id);4096 ITEM_set_cas(it, req_cas_id);
40564097
4057 c->item = it;4098 c->item = it;
4099#ifdef NEED_ALIGN
4100 if (it->it_flags & ITEM_CHUNKED) {
4101 c->ritem = ITEM_schunk(it);
4102 } else {
4103 c->ritem = ITEM_data(it);
4104 }
4105#else
4058 c->ritem = ITEM_data(it);4106 c->ritem = ITEM_data(it);
4107#endif
4059 c->rlbytes = it->nbytes;4108 c->rlbytes = it->nbytes;
4060 c->cmd = comm;4109 c->cmd = comm;
4061 conn_set_state(c, conn_nread);4110 conn_set_state(c, conn_nread);
@@ -4239,13 +4288,7 @@ enum delta_result_type do_add_delta(conn *c, const char *key, const size_t nkey,
4239 } else if (it->refcount > 1) {4288 } else if (it->refcount > 1) {
4240 item *new_it;4289 item *new_it;
4241 uint32_t flags;4290 uint32_t flags;
4242 if (settings.inline_ascii_response) {4291 FLAGS_CONV(settings.inline_ascii_response, it, flags);
4243 flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10);
4244 } else if (it->nsuffix > 0) {
4245 flags = *((uint32_t *)ITEM_suffix(it));
4246 } else {
4247 flags = 0;
4248 }
4249 new_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, res + 2);4292 new_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, res + 2);
4250 if (new_it == 0) {4293 if (new_it == 0) {
4251 do_item_remove(it);4294 do_item_remove(it);
@@ -5274,7 +5317,6 @@ static int read_into_chunked_item(conn *c) {
52745317
5275 while (c->rlbytes > 0) {5318 while (c->rlbytes > 0) {
5276 item_chunk *ch = (item_chunk *)c->ritem;5319 item_chunk *ch = (item_chunk *)c->ritem;
5277 assert(ch->used <= ch->size);
5278 if (ch->size == ch->used) {5320 if (ch->size == ch->used) {
5279 // FIXME: ch->next is currently always 0. remove this?5321 // FIXME: ch->next is currently always 0. remove this?
5280 if (ch->next) {5322 if (ch->next) {
@@ -6156,7 +6198,7 @@ static void clock_handler(const int fd, const short which, void *arg) {
6156static void usage(void) {6198static void usage(void) {
6157 printf(PACKAGE " " VERSION "\n");6199 printf(PACKAGE " " VERSION "\n");
6158 printf("-p, --port=<num> TCP port to listen on (default: 11211)\n"6200 printf("-p, --port=<num> TCP port to listen on (default: 11211)\n"
6159 "-U, --udp-port=<num> UDP port to listen on (default: 11211, 0 is off)\n"6201 "-U, --udp-port=<num> UDP port to listen on (default: 0, off)\n"
6160 "-s, --unix-socket=<file> UNIX socket to listen on (disables network support)\n"6202 "-s, --unix-socket=<file> UNIX socket to listen on (disables network support)\n"
6161 "-A, --enable-shutdown enable ascii \"shutdown\" command\n"6203 "-A, --enable-shutdown enable ascii \"shutdown\" command\n"
6162 "-a, --unix-mask=<mask> access mask for UNIX socket, in octal (default: 0700)\n"6204 "-a, --unix-mask=<mask> access mask for UNIX socket, in octal (default: 0700)\n"
@@ -6236,7 +6278,8 @@ static void usage(void) {
6236 " currently: nothing\n"6278 " currently: nothing\n"
6237 " - no_modern: uses defaults of previous major version (1.4.x)\n"6279 " - no_modern: uses defaults of previous major version (1.4.x)\n"
6238#ifdef HAVE_DROP_PRIVILEGES6280#ifdef HAVE_DROP_PRIVILEGES
6239 " - no_drop_privileges: Disable drop_privileges in case it causes issues with\n"6281 " - drop_privileges: enable dropping extra syscall privileges\n"
6282 " - no_drop_privileges: disable drop_privileges in case it causes issues with\n"
6240 " some customisation.\n"6283 " some customisation.\n"
6241#ifdef MEMCACHED_DEBUG6284#ifdef MEMCACHED_DEBUG
6242 " - relaxed_privileges: Running tests requires extra privileges.\n"6285 " - relaxed_privileges: Running tests requires extra privileges.\n"
@@ -6244,6 +6287,7 @@ static void usage(void) {
6244#endif6287#endif
6245#ifdef EXTSTORE6288#ifdef EXTSTORE
6246 " - ext_path: file to write to for external storage.\n"6289 " - ext_path: file to write to for external storage.\n"
6290 " ie: ext_path=/mnt/d1/extstore:1G\n"
6247 " - ext_page_size: size in megabytes of storage pages.\n"6291 " - ext_page_size: size in megabytes of storage pages.\n"
6248 " - ext_wbuf_size: size in megabytes of page write buffers.\n"6292 " - ext_wbuf_size: size in megabytes of page write buffers.\n"
6249 " - ext_threads: number of IO threads to run.\n"6293 " - ext_threads: number of IO threads to run.\n"
@@ -6255,6 +6299,7 @@ static void usage(void) {
6255 " - ext_compact_under: compact when fewer than this many free pages\n"6299 " - ext_compact_under: compact when fewer than this many free pages\n"
6256 " - ext_drop_under: drop COLD items when fewer than this many free pages\n"6300 " - ext_drop_under: drop COLD items when fewer than this many free pages\n"
6257 " - ext_max_frag: max page fragmentation to tolerage\n"6301 " - ext_max_frag: max page fragmentation to tolerage\n"
6302 " - slab_automove_freeratio: ratio of memory to hold free as buffer.\n"
6258 " (see doc/storage.txt for more info)\n"6303 " (see doc/storage.txt for more info)\n"
6259#endif6304#endif
6260 );6305 );
@@ -6436,6 +6481,16 @@ static int enable_large_pages(void) {
6436 }6481 }
64376482
6438 return ret;6483 return ret;
6484#elif defined(__linux__) && defined(MADV_HUGEPAGE)
6485 /* check if transparent hugepages is compiled into the kernel */
6486 struct stat st;
6487 int ret = stat("/sys/kernel/mm/transparent_hugepage/enabled", &st);
6488 if (ret || !(st.st_mode & S_IFREG)) {
6489 fprintf(stderr, "Transparent huge pages support not detected.\n");
6490 fprintf(stderr, "Will use default page size.\n");
6491 return -1;
6492 }
6493 return 0;
6439#else6494#else
6440 return -1;6495 return -1;
6441#endif6496#endif
@@ -6534,7 +6589,7 @@ int main (int argc, char **argv) {
6534 bool slab_chunk_size_changed = false;6589 bool slab_chunk_size_changed = false;
6535#ifdef EXTSTORE6590#ifdef EXTSTORE
6536 void *storage = NULL;6591 void *storage = NULL;
6537 char *storage_file = NULL;6592 struct extstore_conf_file *storage_file = NULL;
6538 struct extstore_conf ext_cf;6593 struct extstore_conf ext_cf;
6539#endif6594#endif
6540 char *subopts, *subopts_orig;6595 char *subopts, *subopts_orig;
@@ -6575,12 +6630,12 @@ int main (int argc, char **argv) {
6575 NO_LRU_CRAWLER,6630 NO_LRU_CRAWLER,
6576 NO_LRU_MAINTAINER,6631 NO_LRU_MAINTAINER,
6577 NO_DROP_PRIVILEGES,6632 NO_DROP_PRIVILEGES,
6633 DROP_PRIVILEGES,
6578#ifdef MEMCACHED_DEBUG6634#ifdef MEMCACHED_DEBUG
6579 RELAXED_PRIVILEGES,6635 RELAXED_PRIVILEGES,
6580#endif6636#endif
6581#ifdef EXTSTORE6637#ifdef EXTSTORE
6582 EXT_PAGE_SIZE,6638 EXT_PAGE_SIZE,
6583 EXT_PAGE_COUNT,
6584 EXT_WBUF_SIZE,6639 EXT_WBUF_SIZE,
6585 EXT_THREADS,6640 EXT_THREADS,
6586 EXT_IO_DEPTH,6641 EXT_IO_DEPTH,
@@ -6632,12 +6687,12 @@ int main (int argc, char **argv) {
6632 [NO_LRU_CRAWLER] = "no_lru_crawler",6687 [NO_LRU_CRAWLER] = "no_lru_crawler",
6633 [NO_LRU_MAINTAINER] = "no_lru_maintainer",6688 [NO_LRU_MAINTAINER] = "no_lru_maintainer",
6634 [NO_DROP_PRIVILEGES] = "no_drop_privileges",6689 [NO_DROP_PRIVILEGES] = "no_drop_privileges",
6690 [DROP_PRIVILEGES] = "drop_privileges",
6635#ifdef MEMCACHED_DEBUG6691#ifdef MEMCACHED_DEBUG
6636 [RELAXED_PRIVILEGES] = "relaxed_privileges",6692 [RELAXED_PRIVILEGES] = "relaxed_privileges",
6637#endif6693#endif
6638#ifdef EXTSTORE6694#ifdef EXTSTORE
6639 [EXT_PAGE_SIZE] = "ext_page_size",6695 [EXT_PAGE_SIZE] = "ext_page_size",
6640 [EXT_PAGE_COUNT] = "ext_page_count",
6641 [EXT_WBUF_SIZE] = "ext_wbuf_size",6696 [EXT_WBUF_SIZE] = "ext_wbuf_size",
6642 [EXT_THREADS] = "ext_threads",6697 [EXT_THREADS] = "ext_threads",
6643 [EXT_IO_DEPTH] = "ext_io_depth",6698 [EXT_IO_DEPTH] = "ext_io_depth",
@@ -6677,7 +6732,6 @@ int main (int argc, char **argv) {
6677 settings.ext_drop_under = 0;6732 settings.ext_drop_under = 0;
6678 settings.slab_automove_freeratio = 0.01;6733 settings.slab_automove_freeratio = 0.01;
6679 ext_cf.page_size = 1024 * 1024 * 64;6734 ext_cf.page_size = 1024 * 1024 * 64;
6680 ext_cf.page_count = 64;
6681 ext_cf.wbuf_size = settings.ext_wbuf_size;6735 ext_cf.wbuf_size = settings.ext_wbuf_size;
6682 ext_cf.io_threadcount = 1;6736 ext_cf.io_threadcount = 1;
6683 ext_cf.io_depth = 1;6737 ext_cf.io_depth = 1;
@@ -7203,16 +7257,6 @@ int main (int argc, char **argv) {
7203 }7257 }
7204 ext_cf.page_size *= 1024 * 1024; /* megabytes */7258 ext_cf.page_size *= 1024 * 1024; /* megabytes */
7205 break;7259 break;
7206 case EXT_PAGE_COUNT:
7207 if (subopts_value == NULL) {
7208 fprintf(stderr, "Missing ext_page_count argument\n");
7209 return 1;
7210 }
7211 if (!safe_strtoul(subopts_value, &ext_cf.page_count)) {
7212 fprintf(stderr, "could not parse argument to ext_page_count\n");
7213 return 1;
7214 }
7215 break;
7216 case EXT_WBUF_SIZE:7260 case EXT_WBUF_SIZE:
7217 if (subopts_value == NULL) {7261 if (subopts_value == NULL) {
7218 fprintf(stderr, "Missing ext_wbuf_size argument\n");7262 fprintf(stderr, "Missing ext_wbuf_size argument\n");
@@ -7329,7 +7373,20 @@ int main (int argc, char **argv) {
7329 settings.ext_drop_unread = true;7373 settings.ext_drop_unread = true;
7330 break;7374 break;
7331 case EXT_PATH:7375 case EXT_PATH:
7332 storage_file = strdup(subopts_value);7376 if (subopts_value) {
7377 struct extstore_conf_file *tmp = storage_conf_parse(subopts_value, ext_cf.page_size);
7378 if (tmp == NULL) {
7379 fprintf(stderr, "failed to parse ext_path argument\n");
7380 return 1;
7381 }
7382 if (storage_file != NULL) {
7383 tmp->next = storage_file;
7384 }
7385 storage_file = tmp;
7386 } else {
7387 fprintf(stderr, "missing argument to ext_path, ie: ext_path=/d/file:5G\n");
7388 return 1;
7389 }
7333 break;7390 break;
7334#endif7391#endif
7335 case MODERN:7392 case MODERN:
@@ -7351,6 +7408,9 @@ int main (int argc, char **argv) {
7351 case NO_DROP_PRIVILEGES:7408 case NO_DROP_PRIVILEGES:
7352 settings.drop_privileges = false;7409 settings.drop_privileges = false;
7353 break;7410 break;
7411 case DROP_PRIVILEGES:
7412 settings.drop_privileges = true;
7413 break;
7354#ifdef MEMCACHED_DEBUG7414#ifdef MEMCACHED_DEBUG
7355 case RELAXED_PRIVILEGES:7415 case RELAXED_PRIVILEGES:
7356 settings.relaxed_privileges = true;7416 settings.relaxed_privileges = true;
@@ -7536,6 +7596,10 @@ int main (int argc, char **argv) {
7536 fprintf(stderr, "can't find the user %s to switch to\n", username);7596 fprintf(stderr, "can't find the user %s to switch to\n", username);
7537 exit(EX_NOUSER);7597 exit(EX_NOUSER);
7538 }7598 }
7599 if (setgroups(0, NULL) < 0) {
7600 fprintf(stderr, "failed to drop supplementary groups\n");
7601 exit(EX_OSERR);
7602 }
7539 if (setgid(pw->pw_gid) < 0 || setuid(pw->pw_uid) < 0) {7603 if (setgid(pw->pw_gid) < 0 || setuid(pw->pw_uid) < 0) {
7540 fprintf(stderr, "failed to assume identity of user %s\n", username);7604 fprintf(stderr, "failed to assume identity of user %s\n", username);
7541 exit(EX_OSERR);7605 exit(EX_OSERR);
@@ -7596,9 +7660,9 @@ int main (int argc, char **argv) {
7596 if (storage_file) {7660 if (storage_file) {
7597 enum extstore_res eres;7661 enum extstore_res eres;
7598 if (settings.ext_compact_under == 0) {7662 if (settings.ext_compact_under == 0) {
7599 settings.ext_compact_under = ext_cf.page_count / 4;7663 settings.ext_compact_under = storage_file->page_count / 4;
7600 /* Only rescues non-COLD items if below this threshold */7664 /* Only rescues non-COLD items if below this threshold */
7601 settings.ext_drop_under = ext_cf.page_count / 4;7665 settings.ext_drop_under = storage_file->page_count / 4;
7602 }7666 }
7603 crc32c_init();7667 crc32c_init();
7604 /* Init free chunks to zero. */7668 /* Init free chunks to zero. */
@@ -7649,6 +7713,10 @@ int main (int argc, char **argv) {
7649 fprintf(stderr, "Failed to start storage compaction thread\n");7713 fprintf(stderr, "Failed to start storage compaction thread\n");
7650 exit(EXIT_FAILURE);7714 exit(EXIT_FAILURE);
7651 }7715 }
7716 if (storage && start_storage_write_thread(storage) != 0) {
7717 fprintf(stderr, "Failed to start storage writer thread\n");
7718 exit(EXIT_FAILURE);
7719 }
76527720
7653 if (start_lru_maintainer && start_lru_maintainer_thread(storage) != 0) {7721 if (start_lru_maintainer && start_lru_maintainer_thread(storage) != 0) {
7654#else7722#else
diff --git a/memcached.h b/memcached.h
index 22893f5..a8dd59d 100644
--- a/memcached.h
+++ b/memcached.h
@@ -18,6 +18,7 @@
18#include <pthread.h>18#include <pthread.h>
19#include <unistd.h>19#include <unistd.h>
20#include <assert.h>20#include <assert.h>
21#include <grp.h>
2122
22#include "itoa_ljust.h"23#include "itoa_ljust.h"
23#include "protocol_binary.h"24#include "protocol_binary.h"
@@ -140,6 +141,17 @@
140#define APPEND_NUM_STAT(num, name, fmt, val) \141#define APPEND_NUM_STAT(num, name, fmt, val) \
141 APPEND_NUM_FMT_STAT("%d:%s", num, name, fmt, val)142 APPEND_NUM_FMT_STAT("%d:%s", num, name, fmt, val)
142143
144/** Item client flag conversion */
145#define FLAGS_CONV(iar, it, flag) { \
146 if ((iar)) { \
147 flag = (uint32_t) strtoul(ITEM_suffix((it)), (char **) NULL, 10); \
148 } else if ((it)->nsuffix > 0) { \
149 flag = *((uint32_t *)ITEM_suffix((it))); \
150 } else { \
151 flag = 0; \
152 } \
153}
154
143/**155/**
144 * Callback for any function producing stats.156 * Callback for any function producing stats.
145 *157 *
@@ -274,6 +286,8 @@ struct slab_stats {
274#ifdef EXTSTORE286#ifdef EXTSTORE
275#define EXTSTORE_THREAD_STATS_FIELDS \287#define EXTSTORE_THREAD_STATS_FIELDS \
276 X(get_extstore) \288 X(get_extstore) \
289 X(get_aborted_extstore) \
290 X(get_oom_extstore) \
277 X(recache_from_extstore) \291 X(recache_from_extstore) \
278 X(miss_from_extstore) \292 X(miss_from_extstore) \
279 X(badcrc_from_extstore)293 X(badcrc_from_extstore)
@@ -510,6 +524,23 @@ typedef struct _strchunk {
510 uint8_t slabs_clsid; /* Same as above. */524 uint8_t slabs_clsid; /* Same as above. */
511 char data[];525 char data[];
512} item_chunk;526} item_chunk;
527
528#ifdef NEED_ALIGN
529static inline char *ITEM_schunk(item *it) {
530 int offset = it->nkey + 1 + it->nsuffix
531 + ((it->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0);
532 int remain = offset % 8;
533 if (remain != 0) {
534 offset += 8 - remain;
535 }
536 return ((char *) &(it->data)) + offset;
537}
538#else
539#define ITEM_schunk(item) ((char*) &((item)->data) + (item)->nkey + 1 \
540 + (item)->nsuffix \
541 + (((item)->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0))
542#endif
543
513#ifdef EXTSTORE544#ifdef EXTSTORE
514typedef struct {545typedef struct {
515 unsigned int page_version; /* from IO header */546 unsigned int page_version; /* from IO header */
@@ -545,7 +576,7 @@ typedef struct _io_wrap {
545 unsigned int iovec_data; /* specific index of data iovec */576 unsigned int iovec_data; /* specific index of data iovec */
546 bool miss; /* signal a miss to unlink hdr_it */577 bool miss; /* signal a miss to unlink hdr_it */
547 bool badcrc; /* signal a crc failure */578 bool badcrc; /* signal a crc failure */
548 bool active; // FIXME: canary for test. remove579 bool active; /* tells if IO was dispatched or not */
549} io_wrap;580} io_wrap;
550#endif581#endif
551/**582/**
@@ -554,6 +585,7 @@ typedef struct _io_wrap {
554struct conn {585struct conn {
555 int sfd;586 int sfd;
556 sasl_conn_t *sasl_conn;587 sasl_conn_t *sasl_conn;
588 bool sasl_started;
557 bool authenticated;589 bool authenticated;
558 enum conn_states state;590 enum conn_states state;
559 enum bin_substates substate;591 enum bin_substates substate;
@@ -713,7 +745,7 @@ void sidethread_conn_close(conn *c);
713745
714/* Lock wrappers for cache functions that are called from main loop. */746/* Lock wrappers for cache functions that are called from main loop. */
715enum delta_result_type add_delta(conn *c, const char *key,747enum delta_result_type add_delta(conn *c, const char *key,
716 const size_t nkey, const int incr,748 const size_t nkey, bool incr,
717 const int64_t delta, char *buf,749 const int64_t delta, char *buf,
718 uint64_t *cas);750 uint64_t *cas);
719void accept_new_conns(const bool do_accept);751void accept_new_conns(const bool do_accept);
diff --git a/memcached.spec b/memcached.spec
index 697c38f..295b7d9 100644
--- a/memcached.spec
+++ b/memcached.spec
@@ -17,7 +17,7 @@ BuildRequires: systemd-units
17%endif17%endif
1818
19Name: memcached19Name: memcached
20Version: 1.5.620Version: 1.5.10
21Release: 1%{?dist}21Release: 1%{?dist}
22Summary: High Performance, Distributed Memory Object Cache22Summary: High Performance, Distributed Memory Object Cache
2323
@@ -79,9 +79,9 @@ install -Dp -m0755 scripts/memcached.service %{buildroot}%{_unitdir}/%{name}.ser
79install -Dp -m0755 scripts/memcached@.service %{buildroot}%{_unitdir}/%{name}@.service79install -Dp -m0755 scripts/memcached@.service %{buildroot}%{_unitdir}/%{name}@.service
8080
81if [ %{safer_systemd} -gt 0 ]; then81if [ %{safer_systemd} -gt 0 ]; then
82 sed -e -i 's/^##safer##//g' %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service82 sed -e 's/^##safer##//g' -i %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service
83else83else
84 sed -e -i 's/^##safer##/#/g' %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service84 sed -e 's/^##safer##/#/g' -i %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service
85fi85fi
86%else86%else
87install -Dp -m0755 scripts/memcached.sysv %{buildroot}%{_initrddir}/%{name}87install -Dp -m0755 scripts/memcached.sysv %{buildroot}%{_initrddir}/%{name}
@@ -187,7 +187,7 @@ exit 0
187- above suggestions from Bernard Johnson187- above suggestions from Bernard Johnson
188188
189* Mon May 7 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-2189* Mon May 7 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-2
190- Tidyness improvements suggested by Ruben Kerkhof in bugzilla #238994190- Tidiness improvements suggested by Ruben Kerkhof in bugzilla #238994
191191
192* Fri May 4 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-1192* Fri May 4 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-1
193- Initial spec file created via rpmdev-newspec193- Initial spec file created via rpmdev-newspec
diff --git a/sasl_defs.c b/sasl_defs.c
index 93a7b8b..c60d1bf 100644
--- a/sasl_defs.c
+++ b/sasl_defs.c
@@ -7,7 +7,7 @@
77
8char my_sasl_hostname[1025];8char my_sasl_hostname[1025];
99
10#ifdef HAVE_SASL_CB_GETCONF10#if defined(HAVE_SASL_CB_GETCONF) || defined(HAVE_SASL_CB_GETCONFPATH)
11/* The locations we may search for a SASL config file if the user didn't11/* The locations we may search for a SASL config file if the user didn't
12 * specify one in the environment variable SASL_CONF_PATH12 * specify one in the environment variable SASL_CONF_PATH
13 */13 */
@@ -82,7 +82,7 @@ static int sasl_server_userdb_checkpass(sasl_conn_t *conn,
82}82}
83#endif83#endif
8484
85#ifdef HAVE_SASL_CB_GETCONF85#if defined(HAVE_SASL_CB_GETCONF) || defined(HAVE_SASL_CB_GETCONFPATH)
86static int sasl_getconf(void *context, const char **path)86static int sasl_getconf(void *context, const char **path)
87{87{
88 *path = getenv("SASL_CONF_PATH");88 *path = getenv("SASL_CONF_PATH");
@@ -152,6 +152,10 @@ static sasl_callback_t sasl_callbacks[] = {
152152
153#ifdef HAVE_SASL_CB_GETCONF153#ifdef HAVE_SASL_CB_GETCONF
154 { SASL_CB_GETCONF, sasl_getconf, NULL },154 { SASL_CB_GETCONF, sasl_getconf, NULL },
155#else
156#ifdef HAVE_SASL_CB_GETCONFPATH
157 { SASL_CB_GETCONFPATH, (sasl_callback_ft)sasl_getconf, NULL },
158#endif
155#endif159#endif
156160
157 { SASL_CB_LIST_END, NULL, NULL }161 { SASL_CB_LIST_END, NULL, NULL }
diff --git a/scripts/memcached-automove b/scripts/memcached-automove
index f881129..4a3d3dd 100755
--- a/scripts/memcached-automove
+++ b/scripts/memcached-automove
@@ -54,7 +54,7 @@ def determine_move(history, diffs, totals):
54 - use age as average over window. smooths over items falling out of WARM.54 - use age as average over window. smooths over items falling out of WARM.
55 also gives a little weight: if still evicting, a few more pages than55 also gives a little weight: if still evicting, a few more pages than
56 necessary may be moved, pulling the classes closer together. Hopefully56 necessary may be moved, pulling the classes closer together. Hopefully
57 avoidnig ping-ponging.57 avoiding ping-ponging.
58 """58 """
59 # rotate windows59 # rotate windows
60 history['w'].append({})60 history['w'].append({})
diff --git a/scripts/memcached-automove-extstore b/scripts/memcached-automove-extstore
index db66d8c..2bb0e35 100755
--- a/scripts/memcached-automove-extstore
+++ b/scripts/memcached-automove-extstore
@@ -64,7 +64,7 @@ def determine_move(history, stats, diffs, memfree):
64 - if global pool is below minimum remove pages from oldest large class.64 - if global pool is below minimum remove pages from oldest large class.
65 - if global pool is above maximum, move pages to youngest large class.65 - if global pool is above maximum, move pages to youngest large class.
66 - extstore manages a desired number of free chunks in each slab class.66 - extstore manages a desired number of free chunks in each slab class.
67 - autmover adjusts above limits once per minute based on current sizes.67 - automover adjusts above limits once per minute based on current sizes.
68 - if youngest is below the age ratio limit of oldest, move a page to it.68 - if youngest is below the age ratio limit of oldest, move a page to it.
69 """69 """
70 # rotate windows70 # rotate windows
@@ -85,7 +85,6 @@ def determine_move(history, stats, diffs, memfree):
85 print("global pool: [{}]".format(stats['slab_global_page_pool']))85 print("global pool: [{}]".format(stats['slab_global_page_pool']))
8686
87 pool_low = window_key_check(history, 'slab_pool_low')87 pool_low = window_key_check(history, 'slab_pool_low')
88 pool_high = window_key_check(history, 'slab_pool_high')
89 for sid, slab in diffs.items():88 for sid, slab in diffs.items():
90 small_slab = False89 small_slab = False
91 free_enough = False90 free_enough = False
diff --git a/scripts/memcached-tool b/scripts/memcached-tool
index 79f3770..cab36a5 100755
--- a/scripts/memcached-tool
+++ b/scripts/memcached-tool
@@ -20,6 +20,7 @@ use IO::Socket::INET;
20my $addr = shift;20my $addr = shift;
21my $mode = shift || "display";21my $mode = shift || "display";
22my ($from, $to);22my ($from, $to);
23my $limit;
2324
24if ($mode eq "display") {25if ($mode eq "display") {
25 undef $mode if @ARGV;26 undef $mode if @ARGV;
@@ -30,7 +31,11 @@ if ($mode eq "display") {
30 undef $mode if $to < 6 || $to > 17;31 undef $mode if $to < 6 || $to > 17;
31 print STDERR "ERROR: parameters out of range\n\n" unless $mode;32 print STDERR "ERROR: parameters out of range\n\n" unless $mode;
32} elsif ($mode eq 'dump') {33} elsif ($mode eq 'dump') {
33 ;34 if (@ARGV) {
35 $limit = shift;
36 undef $mode if $limit < 1;
37 print STDERR "ERROR: invalid limit (should be a positive number)\n\n" unless $mode;
38 }
34} elsif ($mode eq 'stats') {39} elsif ($mode eq 'stats') {
35 ;40 ;
36} elsif ($mode eq 'settings') {41} elsif ($mode eq 'settings') {
@@ -45,12 +50,12 @@ undef $mode if @ARGV;
4550
46die51die
47 "Usage: memcached-tool <host[:port] | /path/to/socket> [mode]\n52 "Usage: memcached-tool <host[:port] | /path/to/socket> [mode]\n
48 memcached-tool 10.0.0.5:11211 display # shows slabs53 memcached-tool 10.0.0.5:11211 display # shows slabs
49 memcached-tool 10.0.0.5:11211 # same. (default is display)54 memcached-tool 10.0.0.5:11211 # same. (default is display)
50 memcached-tool 10.0.0.5:11211 stats # shows general stats55 memcached-tool 10.0.0.5:11211 stats # shows general stats
51 memcached-tool 10.0.0.5:11211 settings # shows settings stats56 memcached-tool 10.0.0.5:11211 settings # shows settings stats
52 memcached-tool 10.0.0.5:11211 sizes # shows sizes stats57 memcached-tool 10.0.0.5:11211 sizes # shows sizes stats
53 memcached-tool 10.0.0.5:11211 dump # dumps keys and values58 memcached-tool 10.0.0.5:11211 dump [limit] # dumps keys and values
5459
55WARNING! sizes is a development command.60WARNING! sizes is a development command.
56As of 1.4 it is still the only command which will lock your memcached instance for some time.61As of 1.4 it is still the only command which will lock your memcached instance for some time.
@@ -60,65 +65,74 @@ or at least speed it up.
60" unless $addr && $mode;65" unless $addr && $mode;
6166
6267
63my $sock;68sub server_connect {
64if ($addr =~ m:/:) {69 my $sock;
65 $sock = IO::Socket::UNIX->new(70 if ($addr =~ m:/:) {
66 Peer => $addr,71 $sock = IO::Socket::UNIX->new(
67 );72 Peer => $addr,
68}73 );
69else {74 }
70 $addr .= ':11211' unless $addr =~ /:\d+$/;75 else {
76 $addr .= ':11211' unless $addr =~ /:\d+$/;
7177
72 $sock = IO::Socket::INET->new(78 $sock = IO::Socket::INET->new(
73 PeerAddr => $addr,79 PeerAddr => $addr,
74 Proto => 'tcp',80 Proto => 'tcp',
75 );81 );
82 }
83 die "Couldn't connect to $addr\n" unless $sock;
84 return $sock;
76}85}
77die "Couldn't connect to $addr\n" unless $sock;
7886
79if ($mode eq 'dump') {87my $sock = server_connect();
80 my %items;
81 my $totalitems;
82
83 print $sock "stats items\r\n";
8488
89if ($mode eq 'dump') {
90 print STDERR "Dumping memcache contents";
91 print STDERR " (limiting to $limit keys)" unless !$limit;
92 print STDERR "\n";
93 print $sock "lru_crawler metadump all\r\n";
94 my %keyexp;
95 my $keycount = 0;
85 while (<$sock>) {96 while (<$sock>) {
86 last if /^END/;97 last if /^END/ or ($limit and $keycount == $limit);
87 if (/^STAT items:(\d*):number (\d*)/) {98 # return format looks like this
88 $items{$1} = $2;99 # key=foo exp=2147483647 la=1521046038 cas=717111 fetch=no cls=13 size=1232
89 $totalitems += $2;100 if (/^key=(\S+) exp=(-?\d+) .*/) {
90 }101 my $k = $1;
91 }102 $k =~ s/%(.{2})/chr hex $1/eg;
92 print STDERR "Dumping memcache contents\n";103
93 print STDERR " Number of buckets: " . scalar(keys(%items)) . "\n";104 if ($2 == -1) {
94 print STDERR " Number of items : $totalitems\n";105 $keyexp{$k} = 0;
95106 } else {
96 foreach my $bucket (sort(keys(%items))) {107 $keyexp{$k} = $2;
97 print STDERR "Dumping bucket $bucket - " . $items{$bucket} . " total items\n";
98 print $sock "stats cachedump $bucket $items{$bucket}\r\n";
99 my %keyexp;
100 while (<$sock>) {
101 last if /^END/;
102 # return format looks like this
103 # ITEM foo [6 b; 1176415152 s]
104 if (/^ITEM (\S+) \[.* (\d+) s\]/) {
105 $keyexp{$1} = $2;
106 }108 }
107 }109 }
110 $keycount++;
111 }
108112
109 foreach my $k (keys(%keyexp)) {113 if ($limit) {
110 print $sock "get $k\r\n";114 # Need to reopen the connection here to stop the metadump in
111 my $response = <$sock>;115 # case the key limit was reached.
112 if ($response =~ /VALUE (\S+) (\d+) (\d+)/) {116 #
113 my $flags = $2;117 # XXX: Once a limit on # of keys returned is introduced in
114 my $len = $3;118 # `lru_crawler metadump`, this should be removed and the proper
115 my $val;119 # parameter passed in the query above.
116 read $sock, $val, $len;120 close($sock);
117 print "add $k $flags $keyexp{$k} $len\r\n$val\r\n";121 $sock = server_connect();
118 # get the END122 }
119 $_ = <$sock>;123
120 $_ = <$sock>;124 foreach my $k (keys(%keyexp)) {
121 }125 print $sock "get $k\r\n";
126 my $response = <$sock>;
127 if ($response =~ /VALUE (\S+) (\d+) (\d+)/) {
128 my $flags = $2;
129 my $len = $3;
130 my $val;
131 read $sock, $val, $len;
132 print "add $k $flags $keyexp{$k} $len\r\n$val\r\n";
133 # get the END
134 $_ = <$sock>;
135 $_ = <$sock>;
122 }136 }
123 }137 }
124 exit;138 exit;
diff --git a/scripts/memcached-tool.1 b/scripts/memcached-tool.1
index 6bb021b..a863bd4 100644
--- a/scripts/memcached-tool.1
+++ b/scripts/memcached-tool.1
@@ -57,9 +57,12 @@ Number of times the underlying slab class was unable to store a new item.
57Print general-purpose statistics of the daemon. Each line contains the name of57Print general-purpose statistics of the daemon. Each line contains the name of
58the statistic and its value.58the statistic and its value.
59.TP59.TP
60.B dump60.B dump [limit]
61Make a partial dump of the cache written in the add statements of the61Make a partial dump of the cache written in the add statements of the
62memcached protocol.62memcached protocol. If
63.B limit
64is given and is a strictly positive
65integer, then the dump is limited to that number of items.
6366
64.SH SEE ALSO67.SH SEE ALSO
65.BR memcached (1),68.BR memcached (1),
diff --git a/slabs.c b/slabs.c
index 200d575..2bcc350 100644
--- a/slabs.c
+++ b/slabs.c
@@ -8,6 +8,7 @@
8 * memcached protocol.8 * memcached protocol.
9 */9 */
10#include "memcached.h"10#include "memcached.h"
11#include <sys/mman.h>
11#include <sys/stat.h>12#include <sys/stat.h>
12#include <sys/socket.h>13#include <sys/socket.h>
13#include <sys/resource.h>14#include <sys/resource.h>
@@ -98,6 +99,57 @@ unsigned int slabs_clsid(const size_t size) {
98 return res;99 return res;
99}100}
100101
102#if defined(__linux__) && defined(MADV_HUGEPAGE)
103/* Function split out for better error path handling */
104static void * alloc_large_chunk_linux(const size_t limit)
105{
106 size_t pagesize = 0;
107 void *ptr = NULL;
108 FILE *fp;
109 int ret;
110
111 /* Get the size of huge pages */
112 fp = fopen("/proc/meminfo", "r");
113 if (fp != NULL) {
114 char buf[64];
115
116 while ((fgets(buf, sizeof(buf), fp)))
117 if (!strncmp(buf, "Hugepagesize:", 13)) {
118 ret = sscanf(buf + 13, "%zu\n", &pagesize);
119
120 /* meminfo huge page size is in KiBs */
121 pagesize <<= 10;
122 }
123 fclose(fp);
124 }
125
126 if (!pagesize) {
127 fprintf(stderr, "Failed to get supported huge page size\n");
128 return NULL;
129 }
130
131 if (settings.verbose > 1)
132 fprintf(stderr, "huge page size: %zu\n", pagesize);
133
134 /* This works because glibc simply uses mmap when the alignment is
135 * above a certain limit. */
136 ret = posix_memalign(&ptr, pagesize, limit);
137 if (ret != 0) {
138 fprintf(stderr, "Failed to get aligned memory chunk: %d\n", ret);
139 return NULL;
140 }
141
142 ret = madvise(ptr, limit, MADV_HUGEPAGE);
143 if (ret < 0) {
144 fprintf(stderr, "Failed to set transparent hugepage hint: %d\n", ret);
145 free(ptr);
146 ptr = NULL;
147 }
148
149 return ptr;
150}
151#endif
152
101/**153/**
102 * Determines the chunk sizes and initializes the slab class descriptors154 * Determines the chunk sizes and initializes the slab class descriptors
103 * accordingly.155 * accordingly.
@@ -106,11 +158,24 @@ void slabs_init(const size_t limit, const double factor, const bool prealloc, co
106 int i = POWER_SMALLEST - 1;158 int i = POWER_SMALLEST - 1;
107 unsigned int size = sizeof(item) + settings.chunk_size;159 unsigned int size = sizeof(item) + settings.chunk_size;
108160
161 /* Some platforms use runtime transparent hugepages. If for any reason
162 * the initial allocation fails, the required settings do not persist
163 * for remaining allocations. As such it makes little sense to do slab
164 * preallocation. */
165 bool __attribute__ ((unused)) do_slab_prealloc = false;
166
109 mem_limit = limit;167 mem_limit = limit;
110168
111 if (prealloc) {169 if (prealloc) {
170#if defined(__linux__) && defined(MADV_HUGEPAGE)
171 mem_base = alloc_large_chunk_linux(mem_limit);
172 if (mem_base)
173 do_slab_prealloc = true;
174#else
112 /* Allocate everything in a big chunk with malloc */175 /* Allocate everything in a big chunk with malloc */
113 mem_base = malloc(mem_limit);176 mem_base = malloc(mem_limit);
177 do_slab_prealloc = true;
178#endif
114 if (mem_base != NULL) {179 if (mem_base != NULL) {
115 mem_current = mem_base;180 mem_current = mem_base;
116 mem_avail = mem_limit;181 mem_avail = mem_limit;
@@ -161,7 +226,7 @@ void slabs_init(const size_t limit, const double factor, const bool prealloc, co
161226
162 }227 }
163228
164 if (prealloc) {229 if (prealloc && do_slab_prealloc) {
165 slabs_preallocate(power_largest);230 slabs_preallocate(power_largest);
166 }231 }
167}232}
@@ -315,7 +380,7 @@ static void *do_slabs_alloc(const size_t size, unsigned int id, uint64_t *total_
315}380}
316381
317static void do_slabs_free_chunked(item *it, const size_t size) {382static void do_slabs_free_chunked(item *it, const size_t size) {
318 item_chunk *chunk = (item_chunk *) ITEM_data(it);383 item_chunk *chunk = (item_chunk *) ITEM_schunk(it);
319 slabclass_t *p;384 slabclass_t *p;
320385
321 it->it_flags = ITEM_SLABBED;386 it->it_flags = ITEM_SLABBED;
@@ -339,7 +404,15 @@ static void do_slabs_free_chunked(item *it, const size_t size) {
339 p->slots = it;404 p->slots = it;
340 p->sl_curr++;405 p->sl_curr++;
341 // TODO: macro406 // TODO: macro
407#ifdef NEED_ALIGN
408 int total = it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk);
409 if (total % 8 != 0) {
410 total += 8 - (total % 8);
411 }
412 p->requested -= total;
413#else
342 p->requested -= it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk);414 p->requested -= it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk);
415#endif
343 if (settings.use_cas) {416 if (settings.use_cas) {
344 p->requested -= sizeof(uint64_t);417 p->requested -= sizeof(uint64_t);
345 }418 }
@@ -948,7 +1021,7 @@ static int slab_rebalance_move(void) {
948 do_item_replace(it, new_it, hv);1021 do_item_replace(it, new_it, hv);
949 /* Need to walk the chunks and repoint head */1022 /* Need to walk the chunks and repoint head */
950 if (new_it->it_flags & ITEM_CHUNKED) {1023 if (new_it->it_flags & ITEM_CHUNKED) {
951 item_chunk *fch = (item_chunk *) ITEM_data(new_it);1024 item_chunk *fch = (item_chunk *) ITEM_schunk(new_it);
952 fch->next->prev = fch;1025 fch->next->prev = fch;
953 while (fch) {1026 while (fch) {
954 fch->head = new_it;1027 fch->head = new_it;
diff --git a/storage.c b/storage.c
index 6e61cd8..45554cf 100644
--- a/storage.c
+++ b/storage.c
@@ -6,27 +6,18 @@
6#include <stdlib.h>6#include <stdlib.h>
7#include <string.h>7#include <string.h>
8#include <limits.h>8#include <limits.h>
9#include <ctype.h>
910
10#define PAGE_BUCKET_DEFAULT 011#define PAGE_BUCKET_DEFAULT 0
11#define PAGE_BUCKET_COMPACT 112#define PAGE_BUCKET_COMPACT 1
12#define PAGE_BUCKET_CHUNKED 213#define PAGE_BUCKET_CHUNKED 2
13#define PAGE_BUCKET_LOWTTL 314#define PAGE_BUCKET_LOWTTL 3
1415
15int lru_maintainer_store(void *storage, const int clsid) {16/*** WRITE FLUSH THREAD ***/
16 //int i;17
18static int storage_write(void *storage, const int clsid, const int item_age) {
17 int did_moves = 0;19 int did_moves = 0;
18 int item_age = settings.ext_item_age;
19 bool mem_limit_reached = false;
20 unsigned int chunks_free;
21 struct lru_pull_tail_return it_info;20 struct lru_pull_tail_return it_info;
22 // FIXME: need to directly ask the slabber how big a class is
23 if (slabs_clsid(settings.ext_item_size) > clsid)
24 return 0;
25 chunks_free = slabs_available_chunks(clsid, &mem_limit_reached,
26 NULL, NULL);
27 // if we are low on chunks and no spare, push out early.
28 if (chunks_free < settings.ext_free_memchunks[clsid] && mem_limit_reached)
29 item_age = 0;
3021
31 it_info.it = NULL;22 it_info.it = NULL;
32 lru_pull_tail(clsid, COLD_LRU, 0, LRU_PULL_RETURN_ITEM, 0, &it_info);23 lru_pull_tail(clsid, COLD_LRU, 0, LRU_PULL_RETURN_ITEM, 0, &it_info);
@@ -42,14 +33,7 @@ int lru_maintainer_store(void *storage, const int clsid) {
42 uint32_t flags;33 uint32_t flags;
43 if ((it->it_flags & ITEM_HDR) == 0 &&34 if ((it->it_flags & ITEM_HDR) == 0 &&
44 (item_age == 0 || current_time - it->time > item_age)) {35 (item_age == 0 || current_time - it->time > item_age)) {
45 // FIXME: flag conversion again36 FLAGS_CONV(settings.inline_ascii_response, it, flags);
46 if (settings.inline_ascii_response) {
47 flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10);
48 } else if (it->nsuffix > 0) {
49 flags = *((uint32_t *)ITEM_suffix(it));
50 } else {
51 flags = 0;
52 }
53 item *hdr_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, sizeof(item_hdr));37 item *hdr_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, sizeof(item_hdr));
54 /* Run the storage write understanding the start of the item is dirty.38 /* Run the storage write understanding the start of the item is dirty.
55 * We will fill it (time/exptime/etc) from the header item on read.39 * We will fill it (time/exptime/etc) from the header item on read.
@@ -57,7 +41,7 @@ int lru_maintainer_store(void *storage, const int clsid) {
57 if (hdr_it != NULL) {41 if (hdr_it != NULL) {
58 int bucket = (it->it_flags & ITEM_CHUNKED) ?42 int bucket = (it->it_flags & ITEM_CHUNKED) ?
59 PAGE_BUCKET_CHUNKED : PAGE_BUCKET_DEFAULT;43 PAGE_BUCKET_CHUNKED : PAGE_BUCKET_DEFAULT;
60 // Compres soon to expire items into similar pages.44 // Compress soon to expire items into similar pages.
61 if (it->exptime - current_time < settings.ext_low_ttl) {45 if (it->exptime - current_time < settings.ext_low_ttl) {
62 bucket = PAGE_BUCKET_LOWTTL;46 bucket = PAGE_BUCKET_LOWTTL;
63 }47 }
@@ -67,7 +51,9 @@ int lru_maintainer_store(void *storage, const int clsid) {
67 // NOTE: when the item is read back in, the slab mover51 // NOTE: when the item is read back in, the slab mover
68 // may see it. Important to have refcount>=2 or ~ITEM_LINKED52 // may see it. Important to have refcount>=2 or ~ITEM_LINKED
69 assert(it->refcount >= 2);53 assert(it->refcount >= 2);
70 if (extstore_write_request(storage, bucket, &io) == 0) {54 // NOTE: write bucket vs free page bucket will disambiguate once
55 // lowttl feature is better understood.
56 if (extstore_write_request(storage, bucket, bucket, &io) == 0) {
71 // cuddle the hash value into the time field so we don't have57 // cuddle the hash value into the time field so we don't have
72 // to recalculate it.58 // to recalculate it.
73 item *buf_it = (item *) io.buf;59 item *buf_it = (item *) io.buf;
@@ -76,12 +62,12 @@ int lru_maintainer_store(void *storage, const int clsid) {
76 // TODO: should be in items.c62 // TODO: should be in items.c
77 if (it->it_flags & ITEM_CHUNKED) {63 if (it->it_flags & ITEM_CHUNKED) {
78 // Need to loop through the item and copy64 // Need to loop through the item and copy
79 item_chunk *sch = (item_chunk *) ITEM_data(it);65 item_chunk *sch = (item_chunk *) ITEM_schunk(it);
80 int remain = orig_ntotal;66 int remain = orig_ntotal;
81 int copied = 0;67 int copied = 0;
82 // copy original header68 // copy original header
83 int hdrtotal = ITEM_ntotal(it) - it->nbytes;69 int hdrtotal = ITEM_ntotal(it) - it->nbytes;
84 memcpy((char *)io.buf+32, (char *)it+32, hdrtotal - 32);70 memcpy((char *)io.buf+STORE_OFFSET, (char *)it+STORE_OFFSET, hdrtotal - STORE_OFFSET);
85 copied = hdrtotal;71 copied = hdrtotal;
86 // copy data in like it were one large object.72 // copy data in like it were one large object.
87 while (sch && remain) {73 while (sch && remain) {
@@ -93,11 +79,11 @@ int lru_maintainer_store(void *storage, const int clsid) {
93 sch = sch->next;79 sch = sch->next;
94 }80 }
95 } else {81 } else {
96 memcpy((char *)io.buf+32, (char *)it+32, io.len-32);82 memcpy((char *)io.buf+STORE_OFFSET, (char *)it+STORE_OFFSET, io.len-STORE_OFFSET);
97 }83 }
98 // crc what we copied so we can do it sequentially.84 // crc what we copied so we can do it sequentially.
99 buf_it->it_flags &= ~ITEM_LINKED;85 buf_it->it_flags &= ~ITEM_LINKED;
100 buf_it->exptime = crc32c(0, (char*)io.buf+32, orig_ntotal-32);86 buf_it->exptime = crc32c(0, (char*)io.buf+STORE_OFFSET, orig_ntotal-STORE_OFFSET);
101 extstore_write(storage, &io);87 extstore_write(storage, &io);
102 item_hdr *hdr = (item_hdr *) ITEM_data(hdr_it);88 item_hdr *hdr = (item_hdr *) ITEM_data(hdr_it);
103 hdr->page_version = io.page_version;89 hdr->page_version = io.page_version;
@@ -125,6 +111,128 @@ int lru_maintainer_store(void *storage, const int clsid) {
125 return did_moves;111 return did_moves;
126}112}
127113
114static pthread_t storage_write_tid;
115static pthread_mutex_t storage_write_plock;
116#define WRITE_SLEEP_MAX 1000000
117#define WRITE_SLEEP_MIN 500
118
119static void *storage_write_thread(void *arg) {
120 void *storage = arg;
121 // NOTE: ignoring overflow since that would take years of uptime in a
122 // specific load pattern of never going to sleep.
123 unsigned int backoff[MAX_NUMBER_OF_SLAB_CLASSES] = {0};
124 unsigned int counter = 0;
125 useconds_t to_sleep = WRITE_SLEEP_MIN;
126 logger *l = logger_create();
127 if (l == NULL) {
128 fprintf(stderr, "Failed to allocate logger for storage compaction thread\n");
129 abort();
130 }
131
132 pthread_mutex_lock(&storage_write_plock);
133
134 while (1) {
135 // cache per-loop to avoid calls to the slabs_clsid() search loop
136 int min_class = slabs_clsid(settings.ext_item_size);
137 bool do_sleep = true;
138 counter++;
139 if (to_sleep > WRITE_SLEEP_MAX)
140 to_sleep = WRITE_SLEEP_MAX;
141
142 for (int x = 0; x < MAX_NUMBER_OF_SLAB_CLASSES; x++) {
143 bool did_move = false;
144 bool mem_limit_reached = false;
145 unsigned int chunks_free;
146 int item_age;
147 int target = settings.ext_free_memchunks[x];
148 if (min_class > x || (backoff[x] && (counter % backoff[x] != 0))) {
149 // Long sleeps means we should retry classes sooner.
150 if (to_sleep > WRITE_SLEEP_MIN * 10)
151 backoff[x] /= 2;
152 continue;
153 }
154
155 // Avoid extra slab lock calls during heavy writing.
156 chunks_free = slabs_available_chunks(x, &mem_limit_reached,
157 NULL, NULL);
158
159 // storage_write() will fail and cut loop after filling write buffer.
160 while (1) {
161 // if we are low on chunks and no spare, push out early.
162 if (chunks_free < target && mem_limit_reached) {
163 item_age = 0;
164 } else {
165 item_age = settings.ext_item_age;
166 }
167 if (storage_write(storage, x, item_age)) {
168 chunks_free++; // Allow stopping if we've done enough this loop
169 did_move = true;
170 do_sleep = false;
171 if (to_sleep > WRITE_SLEEP_MIN)
172 to_sleep /= 2;
173 } else {
174 break;
175 }
176 }
177
178 if (!did_move) {
179 backoff[x]++;
180 } else if (backoff[x]) {
181 backoff[x] /= 2;
182 }
183 }
184
185 // flip lock so we can be paused or stopped
186 pthread_mutex_unlock(&storage_write_plock);
187 if (do_sleep) {
188 usleep(to_sleep);
189 to_sleep *= 2;
190 }
191 pthread_mutex_lock(&storage_write_plock);
192 }
193 return NULL;
194}
195
196// TODO
197// logger needs logger_destroy() to exist/work before this is safe.
198/*int stop_storage_write_thread(void) {
199 int ret;
200 pthread_mutex_lock(&lru_maintainer_lock);
201 do_run_lru_maintainer_thread = 0;
202 pthread_mutex_unlock(&lru_maintainer_lock);
203 // WAKEUP SIGNAL
204 if ((ret = pthread_join(lru_maintainer_tid, NULL)) != 0) {
205 fprintf(stderr, "Failed to stop LRU maintainer thread: %s\n", strerror(ret));
206 return -1;
207 }
208 settings.lru_maintainer_thread = false;
209 return 0;
210}*/
211
212void storage_write_pause(void) {
213 pthread_mutex_lock(&storage_write_plock);
214}
215
216void storage_write_resume(void) {
217 pthread_mutex_unlock(&storage_write_plock);
218}
219
220int start_storage_write_thread(void *arg) {
221 int ret;
222
223 pthread_mutex_init(&storage_write_plock, NULL);
224 if ((ret = pthread_create(&storage_write_tid, NULL,
225 storage_write_thread, arg)) != 0) {
226 fprintf(stderr, "Can't create storage_write thread: %s\n",
227 strerror(ret));
228 return -1;
229 }
230
231 return 0;
232}
233
234/*** COMPACTOR ***/
235
128/* Fetch stats from the external storage system and decide to compact.236/* Fetch stats from the external storage system and decide to compact.
129 * If we're more than half full, start skewing how aggressively to run237 * If we're more than half full, start skewing how aggressively to run
130 * compaction, up to a desired target when all pages are full.238 * compaction, up to a desired target when all pages are full.
@@ -260,7 +368,7 @@ static void storage_compact_readback(void *storage, logger *l,
260 io.len = ntotal;368 io.len = ntotal;
261 io.mode = OBJ_IO_WRITE;369 io.mode = OBJ_IO_WRITE;
262 for (tries = 10; tries > 0; tries--) {370 for (tries = 10; tries > 0; tries--) {
263 if (extstore_write_request(storage, PAGE_BUCKET_COMPACT, &io) == 0) {371 if (extstore_write_request(storage, PAGE_BUCKET_COMPACT, PAGE_BUCKET_COMPACT, &io) == 0) {
264 memcpy(io.buf, it, io.len);372 memcpy(io.buf, it, io.len);
265 extstore_write(storage, &io);373 extstore_write(storage, &io);
266 do_update = true;374 do_update = true;
@@ -456,4 +564,88 @@ int start_storage_compact_thread(void *arg) {
456 return 0;564 return 0;
457}565}
458566
567/*** UTILITY ***/
568// /path/to/file:100G:bucket1
569// FIXME: Modifies argument. copy instead?
570struct extstore_conf_file *storage_conf_parse(char *arg, unsigned int page_size) {
571 struct extstore_conf_file *cf = NULL;
572 char *b = NULL;
573 char *p = strtok_r(arg, ":", &b);
574 char unit = 0;
575 uint64_t multiplier = 0;
576 int base_size = 0;
577 if (p == NULL)
578 goto error;
579 // First arg is the filepath.
580 cf = calloc(1, sizeof(struct extstore_conf_file));
581 cf->file = strdup(p);
582
583 p = strtok_r(NULL, ":", &b);
584 if (p == NULL) {
585 fprintf(stderr, "must supply size to ext_path, ie: ext_path=/f/e:64m (M|G|T|P supported)\n");
586 goto error;
587 }
588 unit = tolower(p[strlen(p)-1]);
589 p[strlen(p)-1] = '\0';
590 // sigh.
591 switch (unit) {
592 case 'm':
593 multiplier = 1024 * 1024;
594 break;
595 case 'g':
596 multiplier = 1024 * 1024 * 1024;
597 break;
598 case 't':
599 multiplier = 1024 * 1024;
600 multiplier *= 1024 * 1024;
601 break;
602 case 'p':
603 multiplier = 1024 * 1024;
604 multiplier *= 1024 * 1024 * 1024;
605 break;
606 }
607 base_size = atoi(p);
608 multiplier *= base_size;
609 // page_count is nearest-but-not-larger-than pages * psize
610 cf->page_count = multiplier / page_size;
611 assert(page_size * cf->page_count <= multiplier);
612
613 // final token would be a default free bucket
614 p = strtok_r(NULL, ",", &b);
615 // TODO: We reuse the original DEFINES for now,
616 // but if lowttl gets split up this needs to be its own set.
617 if (p != NULL) {
618 if (strcmp(p, "compact") == 0) {
619 cf->free_bucket = PAGE_BUCKET_COMPACT;
620 } else if (strcmp(p, "lowttl") == 0) {
621 cf->free_bucket = PAGE_BUCKET_LOWTTL;
622 } else if (strcmp(p, "chunked") == 0) {
623 cf->free_bucket = PAGE_BUCKET_CHUNKED;
624 } else if (strcmp(p, "default") == 0) {
625 cf->free_bucket = PAGE_BUCKET_DEFAULT;
626 } else {
627 fprintf(stderr, "Unknown extstore bucket: %s\n", p);
628 goto error;
629 }
630 } else {
631 // TODO: is this necessary?
632 cf->free_bucket = PAGE_BUCKET_DEFAULT;
633 }
634
635 // TODO: disabling until compact algorithm is improved.
636 if (cf->free_bucket != PAGE_BUCKET_DEFAULT) {
637 fprintf(stderr, "ext_path only presently supports the default bucket\n");
638 goto error;
639 }
640
641 return cf;
642error:
643 if (cf) {
644 if (cf->file)
645 free(cf->file);
646 free(cf);
647 }
648 return NULL;
649}
650
459#endif651#endif
diff --git a/storage.h b/storage.h
index 7af6cc0..60f499b 100644
--- a/storage.h
+++ b/storage.h
@@ -1,9 +1,15 @@
1#ifndef STORAGE_H1#ifndef STORAGE_H
2#define STORAGE_H2#define STORAGE_H
33
4int lru_maintainer_store(void *storage, const int clsid);4int start_storage_write_thread(void *arg);
5void storage_write_pause(void);
6void storage_write_resume(void);
5int start_storage_compact_thread(void *arg);7int start_storage_compact_thread(void *arg);
6void storage_compact_pause(void);8void storage_compact_pause(void);
7void storage_compact_resume(void);9void storage_compact_resume(void);
10struct extstore_conf_file *storage_conf_parse(char *arg, unsigned int page_size);
11
12// Ignore pointers and header bits from the CRC
13#define STORE_OFFSET offsetof(item, nbytes)
814
9#endif15#endif
diff --git a/t/binary-extstore.t b/t/binary-extstore.t
index 390e26a..676d13b 100755
--- a/t/binary-extstore.t
+++ b/t/binary-extstore.t
@@ -17,7 +17,7 @@ if (!supports_extstore()) {
1717
18$ext_path = "/tmp/extstore.$$";18$ext_path = "/tmp/extstore.$$";
1919
20my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,no_lru_crawler,slab_automove=0");20my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,no_lru_crawler,slab_automove=0");
21ok($server, "started the server");21ok($server, "started the server");
2222
23# Based almost 100% off testClient.py which is:23# Based almost 100% off testClient.py which is:
@@ -176,7 +176,15 @@ $set->('x', 10, 19, "somevalue");
176 $set->("mfoo$_", 0, 19, $value);176 $set->("mfoo$_", 0, 19, $value);
177 }177 }
178 sleep 4;178 sleep 4;
179 $empty->('mfoo1');179 # FIXME: Need to sample through a few values, or fix eviction to be
180 # more accurate. On 32bit systems some pages unused to this point get
181 # filled after the first few items, then the eviction algo pulls those
182 # pages since they have the lowest version number, leaving older objects
183 # in memory and evicting newer ones.
184 for (1 .. ($keycount*3)) {
185 next unless $_ % 100 == 0;
186 eval { $mc->get("mfoo$_"); };
187 }
180188
181 my %s = $mc->stats('');189 my %s = $mc->stats('');
182 cmp_ok($s{extstore_objects_evicted}, '>', 0);190 cmp_ok($s{extstore_objects_evicted}, '>', 0);
diff --git a/t/binary-sasl.t b/t/binary-sasl.t
index 85ef069..e923584 100755
--- a/t/binary-sasl.t
+++ b/t/binary-sasl.t
@@ -13,7 +13,7 @@ use Test::More;
1313
14if (supports_sasl()) {14if (supports_sasl()) {
15 if ($ENV{'RUN_SASL_TESTS'}) {15 if ($ENV{'RUN_SASL_TESTS'}) {
16 plan tests => 33;16 plan tests => 34;
17 } else {17 } else {
18 plan skip_all => 'Skipping SASL tests';18 plan skip_all => 'Skipping SASL tests';
19 exit 0;19 exit 0;
@@ -92,7 +92,7 @@ use constant RES_MAGIC => 0x81;
92my $pwd=getcwd;92my $pwd=getcwd;
93$ENV{'SASL_CONF_PATH'} = "$pwd/t/sasl";93$ENV{'SASL_CONF_PATH'} = "$pwd/t/sasl";
9494
95my $server = new_memcached('-B binary -S ');95my $server = new_memcached('-B binary -S -l 127.0.0.1 ');
9696
97my $mc = MC::Client->new;97my $mc = MC::Client->new;
9898
@@ -192,7 +192,9 @@ for my $dir (split(/:/, $ENV{PATH}),
192 }192 }
193}193}
194194
195system("echo testpass | $saslpasswd_path -a memcached -c -p testuser");195my $sasl_realm = 'memcached.realm';
196
197system("echo testpass | $saslpasswd_path -a memcached -u $sasl_realm -c -p testuser");
196198
197$mc = MC::Client->new;199$mc = MC::Client->new;
198200
@@ -261,6 +263,11 @@ $empty->('x', 'somevalue');
261 cmp_ok($status,'==',ERR_AUTH_ERROR, "error code matches");263 cmp_ok($status,'==',ERR_AUTH_ERROR, "error code matches");
262}264}
263265
266{
267 my $mc = MC::Client->new;
268 is ($mc->sasl_step('testuser', 'testpass'), 0x20, "sasl_step_fails_no_segfault");
269}
270
264# check the SASL stats, make sure they track things correctly271# check the SASL stats, make sure they track things correctly
265# note: the enabled or not is presence checked in stats.t272# note: the enabled or not is presence checked in stats.t
266273
@@ -273,8 +280,8 @@ $empty->('x', 'somevalue');
273280
274{281{
275 my %stats = $mc->stats('');282 my %stats = $mc->stats('');
276 is ($stats{'auth_cmds'}, 5, "auth commands counted");283 is ($stats{'auth_cmds'}, 6, "auth commands counted");
277 is ($stats{'auth_errors'}, 3, "auth errors correct");284 is ($stats{'auth_errors'}, 4, "auth errors correct");
278}285}
279286
280287
@@ -309,10 +316,17 @@ sub new {
309sub authenticate {316sub authenticate {
310 my ($self, $user, $pass, $mech)= @_;317 my ($self, $user, $pass, $mech)= @_;
311 $mech ||= 'PLAIN';318 $mech ||= 'PLAIN';
312 my $buf = sprintf("%c%s%c%s", 0, $user, 0, $pass);319 my $buf = sprintf("%c%s@%s%c%s", 0, $user, $sasl_realm, 0, $pass);
313 my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_AUTH, $mech, $buf, '');320 my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_AUTH, $mech, $buf, '');
314 return $status;321 return $status;
315}322}
323sub sasl_step {
324 my ($self, $user, $pass, $mech)= @_;
325 $mech ||= 'PLAIN';
326 my $buf = sprintf("%c%s@%s%c%s", 0, $user, $sasl_realm, 0, $pass);
327 my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_STEP, $mech, $buf, '');
328 return $status;
329}
316sub list_mechs {330sub list_mechs {
317 my ($self)= @_;331 my ($self)= @_;
318 my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_LIST_MECHS, '', '', '');332 my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_LIST_MECHS, '', '', '');
@@ -661,4 +675,3 @@ sub auth_error {
661unlink $sasldb;675unlink $sasldb;
662676
663# vim: filetype=perl677# vim: filetype=perl
664
diff --git a/t/chunked-extstore.t b/t/chunked-extstore.t
index ebd0f0b..3515f21 100644
--- a/t/chunked-extstore.t
+++ b/t/chunked-extstore.t
@@ -18,9 +18,26 @@ if (!supports_extstore()) {
1818
19$ext_path = "/tmp/extstore.$$";19$ext_path = "/tmp/extstore.$$";
2020
21my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,slab_chunk_max=16384,slab_automove=0");21my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_chunk_max=16384,slab_automove=0,ext_compact_under=1");
22my $sock = $server->sock;22my $sock = $server->sock;
2323
24# Wait until all items have flushed
25sub wait_for_ext {
26 my $sum = 1;
27 while ($sum != 0) {
28 my $s = mem_stats($sock, "items");
29 $sum = 0;
30 for my $key (keys %$s) {
31 if ($key =~ m/items:(\d+):number/) {
32 # Ignore classes which can contain extstore items
33 next if $1 < 3;
34 $sum += $s->{$key};
35 }
36 }
37 sleep 1 if $sum != 0;
38 }
39}
40
24# We're testing to ensure item chaining doesn't corrupt or poorly overlap41# We're testing to ensure item chaining doesn't corrupt or poorly overlap
25# data, so create a non-repeating pattern.42# data, so create a non-repeating pattern.
26my @parts = ();43my @parts = ();
@@ -58,7 +75,7 @@ for (1..5) {
58 print $sock "set toast$_ 0 0 $biglen\r\n$big\r\n";75 print $sock "set toast$_ 0 0 $biglen\r\n$big\r\n";
59 is(scalar <$sock>, "STORED\r\n", "stored big");76 is(scalar <$sock>, "STORED\r\n", "stored big");
60 }77 }
61 sleep 4;78 wait_for_ext();
6279
63 for (1..40) {80 for (1..40) {
64 mem_get_is($sock, "toast$_", $big);81 mem_get_is($sock, "toast$_", $big);
@@ -76,17 +93,19 @@ for (1..5) {
7693
77# fill to eviction94# fill to eviction
78{95{
79 my $keycount = 1000;96 my $keycount = 1250;
80 for (1 .. $keycount) {97 for (1 .. $keycount) {
81 print $sock "set mfoo$_ 0 0 $plen noreply\r\n$pattern\r\n";98 print $sock "set mfoo$_ 0 0 $plen noreply\r\n$pattern\r\n";
99 wait_for_ext() if $_ % 500 == 0;
82 }100 }
83 sleep 6;101 # because item_age is set to 2s.
102 wait_for_ext();
84103
85 my $stats = mem_stats($sock);104 my $stats = mem_stats($sock);
86 cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted');105 cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted');
87 cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted');106 cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted');
88 cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted');107 cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted');
89 is($stats->{extstore_pages_free}, 1, '1 page is free');108 cmp_ok($stats->{extstore_pages_free}, '<', 2, 'most pages are used');
90 is($stats->{miss_from_extstore}, 0, 'no misses');109 is($stats->{miss_from_extstore}, 0, 'no misses');
91110
92 # original "pattern" key should be gone.111 # original "pattern" key should be gone.
@@ -106,14 +125,18 @@ for (1..5) {
106 print $sock "delete toast$_ noreply\r\n" if $_ % 2 == 0;125 print $sock "delete toast$_ noreply\r\n" if $_ % 2 == 0;
107 }126 }
108127
109 for (1..1000) {128 for (1..1250) {
110 # Force a read so objects don't get skipped.129 # Force a read so objects don't get skipped.
111 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;130 print $sock "add mfoo$_ 0 0 1 noreply\r\n1\r\n" if $_ % 2 == 1;
112 }131 }
113 for (1..1000) {132 for (1..1250) {
114 # Force a read so objects don't get skipped.133 # Delete lots of objects to trigger compaction.
115 print $sock "delete mfoo$_ noreply\r\n" if $_ % 2 == 0;134 print $sock "delete mfoo$_ noreply\r\n" if $_ % 2 == 0;
116 }135 }
136 print $sock "extstore compact_under 4\r\n";
137 my $res = <$sock>;
138 print $sock "extstore drop_under 3\r\n";
139 $res = <$sock>;
117140
118 sleep 4;141 sleep 4;
119142
@@ -122,7 +145,7 @@ for (1..5) {
122 cmp_ok($stats->{extstore_compact_rescues}, '>', 0, 'some compaction rescues happened');145 cmp_ok($stats->{extstore_compact_rescues}, '>', 0, 'some compaction rescues happened');
123146
124 # Some of the early items got evicted147 # Some of the early items got evicted
125 for (100..1000) {148 for (750..1250) {
126 # everything should validate properly.149 # everything should validate properly.
127 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;150 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;
128 }151 }
@@ -133,20 +156,12 @@ for (1..5) {
133 print $sock "extstore recache_rate 1\r\n";156 print $sock "extstore recache_rate 1\r\n";
134 is(scalar <$sock>, "OK\r\n", "upped recache rate");157 is(scalar <$sock>, "OK\r\n", "upped recache rate");
135158
136 for (800..1000) {159 for (1150..1250) {
137 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;160 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;
138 }161 }
139162
140 my $stats = mem_stats($sock);163 my $stats = mem_stats($sock);
141 cmp_ok($stats->{recache_from_extstore}, '>', 100, 'recaching happening');164 cmp_ok($stats->{recache_from_extstore}, '>', 25, 'recaching happening');
142
143 for (800..1000) {
144 mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1;
145 }
146
147 my $stats2 = mem_stats($sock);
148 is($stats->{recache_from_extstore}, $stats2->{recache_from_extstore},
149 'values already recached');
150}165}
151166
152done_testing();167done_testing();
diff --git a/t/error-extstore.t b/t/error-extstore.t
153new file mode 100644168new file mode 100644
index 0000000..6df1528
--- /dev/null
+++ b/t/error-extstore.t
@@ -0,0 +1,130 @@
1#!/usr/bin/perl
2# Test the "Error on get" path for extstore.
3# the entire error handling code for process_get_command() never worked, and
4# would infinite loop. get_extstore() can hit it sometimes.
5
6use strict;
7use warnings;
8
9use Test::More;
10use FindBin qw($Bin);
11use lib "$Bin/lib";
12use MemcachedTest;
13
14my $ext_path;
15
16if (!supports_extstore()) {
17 plan skip_all => 'extstore not enabled';
18 exit 0;
19}
20
21$ext_path = "/tmp/extstore.$$";
22
23my $server = new_memcached("-m 64 -I 4m -U 0 -o ext_page_size=8,ext_wbuf_size=8,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_automove=0,ext_compact_under=1");
24my $sock = $server->sock;
25
26# Wait until all items have flushed
27sub wait_for_ext {
28 my $sum = 1;
29 while ($sum != 0) {
30 my $s = mem_stats($sock, "items");
31 $sum = 0;
32 for my $key (keys %$s) {
33 if ($key =~ m/items:(\d+):number/) {
34 # Ignore classes which can contain extstore items
35 next if $1 < 3;
36 $sum += $s->{$key};
37 }
38 }
39 sleep 1 if $sum != 0;
40 }
41}
42
43# We're testing to ensure item chaining doesn't corrupt or poorly overlap
44# data, so create a non-repeating pattern.
45my @parts = ();
46for (1 .. 8000) {
47 push(@parts, $_);
48}
49my $pattern = join(':', @parts);
50my $plen = length($pattern);
51
52# Set some large items and let them flush to extstore.
53for (1..5) {
54 my $size = 3000 * 1024;
55 my $data = "x" x $size;
56 print $sock "set foo$_ 0 0 $size\r\n$data\r\n";
57 my $res = <$sock>;
58 is($res, "STORED\r\n", "stored some big items");
59}
60
61wait_for_ext();
62
63{
64 my $long_key = "f" x 512;
65 print $sock "get foo1 foo2 foo3 $long_key\r\n";
66 ok(scalar <$sock> =~ /CLIENT_ERROR bad command line format/, 'long key fails');
67 my $stats = mem_stats($sock);
68 cmp_ok($stats->{get_aborted_extstore}, '>', 1, 'some extstore queries aborted');
69}
70
71# Disable automatic page balancing, then move enough pages that the large
72# items can no longer be loaded from extstore
73{
74 print $sock "slabs automove 0\r\n";
75 my $res = <$sock>;
76 my $source = 0;
77 while (1) {
78 print $sock "slabs reassign $source 1\r\n";
79 $res = <$sock>;
80 if ($res =~ m/NOSPARE/) {
81 $source = -1;
82 my $stats = mem_stats($sock, 'slabs');
83 for my $key (grep { /total_pages/ } keys %$stats) {
84 if ($key =~ m/(\d+):total_pages/) {
85 next if $1 < 3;
86 $source = $1 if $stats->{$key} > 1;
87 }
88 }
89 last if $source == -1;
90 }
91 select undef, undef, undef, 0.10;
92 }
93}
94
95# fetching the large keys should now fail.
96{
97 print $sock "get foo1\r\n";
98 my $res = <$sock>;
99 $res =~ s/[\r\n]//g;
100 is($res, 'SERVER_ERROR out of memory writing get response', 'can no longer read back item');
101 my $stats = mem_stats($sock);
102 is($stats->{get_oom_extstore}, 1, 'check extstore oom counter');
103}
104
105# Leaving this for future generations.
106# The process_get_command() function had several memory leaks.
107my $LEAK_TEST = 0;
108if ($LEAK_TEST) {
109 my $tries = 0;
110 while ($tries) {
111 print $sock "slabs reassign 1 39\r\n";
112 my $res = <$sock>;
113 if ($res =~ m/BUSY/) {
114 select undef, undef, undef, 0.10;
115 } else {
116 $tries--;
117 }
118 }
119 my $long_key = "f" x 512;
120 while (1) {
121 print $sock "get foo1 foo2 foo3 $long_key\r\n";
122 my $res = <$sock>;
123 }
124}
125
126done_testing();
127
128END {
129 unlink $ext_path if $ext_path;
130}
diff --git a/t/extstore-buckets.t b/t/extstore-buckets.t
index a2c1c90..e3027ad 100644
--- a/t/extstore-buckets.t
+++ b/t/extstore-buckets.t
@@ -17,7 +17,7 @@ if (!supports_extstore()) {
1717
18$ext_path = "/tmp/extstore.$$";18$ext_path = "/tmp/extstore.$$";
1919
20my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0,ext_path=$ext_path,ext_low_ttl=60,slab_automove=1");20my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0,ext_path=$ext_path:64m,ext_low_ttl=60,slab_automove=1");
21my $sock = $server->sock;21my $sock = $server->sock;
2222
23my $value;23my $value;
diff --git a/t/extstore-jbod.t b/t/extstore-jbod.t
24new file mode 10064424new file mode 100644
index 0000000..1618803
--- /dev/null
+++ b/t/extstore-jbod.t
@@ -0,0 +1,69 @@
1#!/usr/bin/perl
2
3use strict;
4use warnings;
5use Test::More;
6use FindBin qw($Bin);
7use lib "$Bin/lib";
8use MemcachedTest;
9use Data::Dumper qw/Dumper/;
10
11my $ext_path;
12my $ext_path2;
13
14if (!supports_extstore()) {
15 plan skip_all => 'extstore not enabled';
16 exit 0;
17}
18
19$ext_path = "/tmp/extstore1.$$";
20$ext_path2 = "/tmp/extstore2.$$";
21
22my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,ext_path=$ext_path2:96m,slab_automove=1");
23my $sock = $server->sock;
24
25my $value;
26{
27 my @chars = ("C".."Z");
28 for (1 .. 20000) {
29 $value .= $chars[rand @chars];
30 }
31}
32
33# fill some larger objects
34{
35 # interleave sets with 0 ttl vs long ttl's.
36 my $keycount = 3700;
37 for (1 .. $keycount) {
38 print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n";
39 print $sock "set lfoo$_ 0 0 20000 noreply\r\n$value\r\n";
40 }
41 # wait for a flush
42 wait_ext_flush($sock);
43 # delete half
44 mem_get_is($sock, "nfoo1", $value);
45 for (1 .. $keycount) {
46 print $sock "delete lfoo$_ noreply\r\n";
47 }
48 print $sock "lru_crawler crawl all\r\n";
49 <$sock>;
50 sleep 10;
51 # fetch
52 # check extstore counters
53 my $stats = mem_stats($sock);
54 is($stats->{evictions}, 0, 'no RAM evictions');
55 cmp_ok($stats->{extstore_page_allocs}, '>', 0, 'at least one page allocated');
56 cmp_ok($stats->{extstore_objects_written}, '>', $keycount / 2, 'some objects written');
57 cmp_ok($stats->{extstore_bytes_written}, '>', length($value) * 2, 'some bytes written');
58 cmp_ok($stats->{get_extstore}, '>', 0, 'one object was fetched');
59 cmp_ok($stats->{extstore_objects_read}, '>', 0, 'one object read');
60 cmp_ok($stats->{extstore_bytes_read}, '>', length($value), 'some bytes read');
61 cmp_ok($stats->{extstore_page_reclaims}, '>', 1, 'at least two pages reclaimed');
62}
63
64done_testing();
65
66END {
67 unlink $ext_path if $ext_path;
68 unlink $ext_path2 if $ext_path2;
69}
diff --git a/t/extstore.t b/t/extstore.t
index df4ca05..1790a54 100644
--- a/t/extstore.t
+++ b/t/extstore.t
@@ -17,9 +17,27 @@ if (!supports_extstore()) {
1717
18$ext_path = "/tmp/extstore.$$";18$ext_path = "/tmp/extstore.$$";
1919
20my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,slab_automove=0");20my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_automove=0,ext_compact_under=1");
21my $sock = $server->sock;21my $sock = $server->sock;
2222
23# Wait until all items have flushed
24sub wait_for_ext {
25 my $target = shift || 0;
26 my $sum = $target + 1;
27 while ($sum > $target) {
28 my $s = mem_stats($sock, "items");
29 $sum = 0;
30 for my $key (keys %$s) {
31 if ($key =~ m/items:(\d+):number/) {
32 # Ignore classes which can contain extstore items
33 next if $1 < 3;
34 $sum += $s->{$key};
35 }
36 }
37 sleep 1 if $sum > $target;
38 }
39}
40
23my $value;41my $value;
24{42{
25 my @chars = ("C".."Z");43 my @chars = ("C".."Z");
@@ -47,7 +65,7 @@ mem_get_is($sock, "foo", "hi");
47 print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n";65 print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n";
48 }66 }
49 # wait for a flush67 # wait for a flush
50 sleep 4;68 wait_for_ext();
51 # fetch69 # fetch
52 # TODO: Fetch back all values70 # TODO: Fetch back all values
53 mem_get_is($sock, "nfoo1", $value);71 mem_get_is($sock, "nfoo1", $value);
@@ -83,22 +101,28 @@ mem_get_is($sock, "foo", "hi");
83101
84# fill to eviction102# fill to eviction
85{103{
86 my $keycount = 3000;104 my $keycount = 4000;
87 for (1 .. $keycount) {105 for (1 .. $keycount) {
88 print $sock "set mfoo$_ 0 0 20000 noreply\r\n$value\r\n";106 print $sock "set mfoo$_ 0 0 20000 noreply\r\n$value\r\n";
107 # wait to avoid evictions
108 wait_for_ext(500) if ($_ % 2000 == 0);
89 }109 }
90 sleep 4;110 # because item_age is set to 2s
111 wait_for_ext();
91 my $stats = mem_stats($sock);112 my $stats = mem_stats($sock);
113 is($stats->{evictions}, 0, 'no evictions');
92 is($stats->{miss_from_extstore}, 0, 'no misses');114 is($stats->{miss_from_extstore}, 0, 'no misses');
93 mem_get_is($sock, "canary", undef);115 # FIXME: test is flaky; something can rescue the canary because of a race
116 # condition. might need to roundtrip twice or disable compaction?
117 #mem_get_is($sock, "canary", undef);
94118
95 # check counters119 # check counters
96 $stats = mem_stats($sock);120 $stats = mem_stats($sock);
97 cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted');121 cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted');
98 cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted');122 cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted');
99 cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted');123 cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted');
100 is($stats->{extstore_pages_free}, 0, '0 pages are free');124 cmp_ok($stats->{extstore_pages_free}, '<', 2, 'few pages are free');
101 is($stats->{miss_from_extstore}, 1, 'exactly one miss');125 #is($stats->{miss_from_extstore}, 1, 'exactly one miss');
102126
103 # refresh some keys so rescues happen while drop_unread == 1.127 # refresh some keys so rescues happen while drop_unread == 1.
104 for (1 .. $keycount / 2) {128 for (1 .. $keycount / 2) {
@@ -111,6 +135,10 @@ mem_get_is($sock, "foo", "hi");
111 my $res = <$sock>;135 my $res = <$sock>;
112 print $sock "extstore max_frag 0\r\n";136 print $sock "extstore max_frag 0\r\n";
113 $res = <$sock>;137 $res = <$sock>;
138 print $sock "extstore compact_under 4\r\n";
139 $res = <$sock>;
140 print $sock "extstore drop_under 3\r\n";
141 $res = <$sock>;
114 for (1 .. $keycount) {142 for (1 .. $keycount) {
115 next unless $_ % 2 == 0;143 next unless $_ % 2 == 0;
116 print $sock "delete mfoo$_ noreply\r\n";144 print $sock "delete mfoo$_ noreply\r\n";
@@ -131,7 +159,7 @@ mem_get_is($sock, "foo", "hi");
131 for (1 .. $keycount) {159 for (1 .. $keycount) {
132 print $sock "set bfoo$_ 0 0 20000 noreply\r\n$value\r\n";160 print $sock "set bfoo$_ 0 0 20000 noreply\r\n$value\r\n";
133 }161 }
134 sleep 4;162 wait_for_ext();
135163
136 # incr should be blocked.164 # incr should be blocked.
137 print $sock "incr bfoo1 1\r\n";165 print $sock "incr bfoo1 1\r\n";
@@ -139,7 +167,7 @@ mem_get_is($sock, "foo", "hi");
139167
140 # append/prepend *could* work, but it would require pulling the item back in.168 # append/prepend *could* work, but it would require pulling the item back in.
141 print $sock "append bfoo1 0 0 2\r\nhi\r\n";169 print $sock "append bfoo1 0 0 2\r\nhi\r\n";
142 is(scalar <$sock>, "NOT_STORED\r\n", 'append falis');170 is(scalar <$sock>, "NOT_STORED\r\n", 'append fails');
143 print $sock "prepend bfoo1 0 0 2\r\nhi\r\n";171 print $sock "prepend bfoo1 0 0 2\r\nhi\r\n";
144 is(scalar <$sock>, "NOT_STORED\r\n", 'prepend fails');172 is(scalar <$sock>, "NOT_STORED\r\n", 'prepend fails');
145}173}
diff --git a/t/flush-all.t b/t/flush-all.t
index e30d819..d9c536a 100755
--- a/t/flush-all.t
+++ b/t/flush-all.t
@@ -40,7 +40,7 @@ is(scalar <$sock>, "OK\r\n", "did flush_all in future");
40print $sock "set foo 0 0 4\r\n1234\r\n";40print $sock "set foo 0 0 4\r\n1234\r\n";
41is(scalar <$sock>, "STORED\r\n", "stored foo = '1234'");41is(scalar <$sock>, "STORED\r\n", "stored foo = '1234'");
42mem_get_is($sock, "foo", '1234');42mem_get_is($sock, "foo", '1234');
43sleep(3);43sleep(5);
44mem_get_is($sock, "foo", undef);44mem_get_is($sock, "foo", undef);
4545
46print $sock "set foo 0 0 5\r\n12345\r\n";46print $sock "set foo 0 0 5\r\n12345\r\n";
diff --git a/t/issue_67.t b/t/issue_67.t
index b2d374f..1dbaba2 100644
--- a/t/issue_67.t
+++ b/t/issue_67.t
@@ -1,11 +1,12 @@
1#!/usr/bin/perl1#!/usr/bin/perl
22
3use strict;3use strict;
4use Test::More tests => 22;4use Test::More tests => 24;
5use FindBin qw($Bin);5use FindBin qw($Bin);
6use lib "$Bin/lib";6use lib "$Bin/lib";
7use MemcachedTest;7use MemcachedTest;
8use Carp qw(croak);8use Carp qw(croak);
9use Socket qw(sockaddr_in INADDR_ANY PF_INET SOCK_STREAM);
910
10use Cwd;11use Cwd;
11my $builddir = getcwd;12my $builddir = getcwd;
@@ -29,12 +30,25 @@ sub validate_port {
29 if ($expected == -1) {30 if ($expected == -1) {
30 ok(!defined($got), "$name expected no port, got $got");31 ok(!defined($got), "$name expected no port, got $got");
31 } elsif ($expected == 0) {32 } elsif ($expected == 0) {
32 ok($got != 11211, "$name expected random port (got $got)");33 ok(defined($got) && $got != 11211, "$name expected random port (got $got)");
33 } else {34 } else {
34 is($got, $expected, "$name");35 is($got, $expected, "$name");
35 }36 }
36}37}
3738
39sub skip_if_default_addr_in_use(&) {
40 my ($block) = @_;
41
42 socket(my $socket, PF_INET, SOCK_STREAM, 0) or die $!;
43 my $addr_in_use = !bind($socket, sockaddr_in(11211, INADDR_ANY));
44 close($socket);
45
46 SKIP: {
47 skip 'Default address is in use. Do you have a running instance?', 2 if $addr_in_use;
48 return $block->();
49 }
50}
51
38sub run_server {52sub run_server {
39 my ($args) = @_;53 my ($args) = @_;
4054
@@ -75,14 +89,13 @@ sub when {
75 validate_port($name, $ports{'UDP INET'}, $expected_udp);89 validate_port($name, $ports{'UDP INET'}, $expected_udp);
76}90}
7791
78# Disabling the defaults since it conflicts with a running instance.92skip_if_default_addr_in_use { when('no arguments', '', 11211, -1) };
79# when('no arguments', '', 11211, 11211);
80when('specifying tcp port', '-p 11212', 11212, -1);93when('specifying tcp port', '-p 11212', 11212, -1);
81when('specifying udp port', '-U 11222', 11222, 11222);94when('specifying udp port', '-U 11222', 11222, 11222);
82when('specifying tcp ephemeral port', '-p -1', 0, 0);95when('specifying tcp ephemeral port', '-p -1', 0, -1);
83when('specifying udp ephemeral port', '-U -1', 0, 0);96when('specifying udp ephemeral port', '-U -1', 0, 0);
84when('tcp port disabled', '-p 0', -1, -1);97when('tcp port disabled', '-p 0', -1, -1);
85when('udp port disabled', '-U 0', 11211, -1);98skip_if_default_addr_in_use { when('udp port disabled', '-U 0', 11211, -1) };
86when('specifying tcp and udp ports', '-p 11232 -U 11233', 11232, 11233);99when('specifying tcp and udp ports', '-p 11232 -U 11233', 11232, 11233);
87when('specifying tcp and disabling udp', '-p 11242 -U 0', 11242, -1);100when('specifying tcp and disabling udp', '-p 11242 -U 0', 11242, -1);
88when('specifying udp and disabling tcp', '-p -1 -U 11252', 0, 11252);101when('specifying udp and disabling tcp', '-p -1 -U 11252', 0, 11252);
diff --git a/t/lib/MemcachedTest.pm b/t/lib/MemcachedTest.pm
index 4e1da66..416daaf 100644
--- a/t/lib/MemcachedTest.pm
+++ b/t/lib/MemcachedTest.pm
@@ -14,13 +14,33 @@ my $builddir = getcwd;
14my @unixsockets = ();14my @unixsockets = ();
1515
16@EXPORT = qw(new_memcached sleep mem_get_is mem_gets mem_gets_is mem_stats16@EXPORT = qw(new_memcached sleep mem_get_is mem_gets mem_gets_is mem_stats
17 supports_sasl free_port supports_drop_priv supports_extstore);17 supports_sasl free_port supports_drop_priv supports_extstore
18 wait_ext_flush);
1819
19sub sleep {20sub sleep {
20 my $n = shift;21 my $n = shift;
21 select undef, undef, undef, $n;22 select undef, undef, undef, $n;
22}23}
2324
25# Wait until all items have flushed
26sub wait_ext_flush {
27 my $sock = shift;
28 my $target = shift || 0;
29 my $sum = $target + 1;
30 while ($sum > $target) {
31 my $s = mem_stats($sock, "items");
32 $sum = 0;
33 for my $key (keys %$s) {
34 if ($key =~ m/items:(\d+):number/) {
35 # Ignore classes which can contain extstore items
36 next if $1 < 3;
37 $sum += $s->{$key};
38 }
39 }
40 sleep 1 if $sum > $target;
41 }
42}
43
24sub mem_stats {44sub mem_stats {
25 my ($sock, $type) = @_;45 my ($sock, $type) = @_;
26 $type = $type ? " $type" : "";46 $type = $type ? " $type" : "";
diff --git a/t/lru-crawler.t b/t/lru-crawler.t
index 18375c2..4e94fb3 100644
--- a/t/lru-crawler.t
+++ b/t/lru-crawler.t
@@ -2,7 +2,7 @@
22
3use strict;3use strict;
4use warnings;4use warnings;
5use Test::More tests => 221;5use Test::More tests => 222;
6use FindBin qw($Bin);6use FindBin qw($Bin);
7use lib "$Bin/lib";7use lib "$Bin/lib";
8use MemcachedTest;8use MemcachedTest;
@@ -60,6 +60,18 @@ while (1) {
60 is($items->{"items:1:crawler_reclaimed"}, 30, "slab1 has 30 reclaims");60 is($items->{"items:1:crawler_reclaimed"}, 30, "slab1 has 30 reclaims");
61}61}
6262
63# Check that crawler metadump works correctly.
64{
65 print $sock "lru_crawler metadump all\r\n";
66 my $count = 0;
67 while (<$sock>) {
68 last if /^(\.|END)/;
69 /^(key=) (\S+).*([^\r\n]+)/;
70 $count++;
71 }
72 is ($count, 60);
73}
74
63for (1 .. 30) {75for (1 .. 30) {
64 mem_get_is($sock, "ifoo$_", "ok");76 mem_get_is($sock, "ifoo$_", "ok");
65 mem_get_is($sock, "lfoo$_", "ok");77 mem_get_is($sock, "lfoo$_", "ok");
diff --git a/t/lru-maintainer.t b/t/lru-maintainer.t
index ac2ba7b..04fa76a 100644
--- a/t/lru-maintainer.t
+++ b/t/lru-maintainer.t
@@ -56,6 +56,18 @@ for (my $key = 0; $key < 100; $key++) {
56 # Items need two fetches to become active56 # Items need two fetches to become active
57 mem_get_is($sock, "canary", $value);57 mem_get_is($sock, "canary", $value);
58 mem_get_is($sock, "canary", $value);58 mem_get_is($sock, "canary", $value);
59 $stats = mem_stats($sock);
60 # The maintainer thread needs to juggle a bit to actually rescue an
61 # item. If it's slow we could evict after resuming setting.
62 sleep 1;
63 for (0..4) {
64 my $s2 = mem_stats($sock);
65 if ($s2->{lru_maintainer_juggles} - $stats->{lru_maintainer_juggles} < 5) {
66 sleep 1;
67 next;
68 }
69 last;
70 }
59 }71 }
60 print $sock "set key$key 0 0 66560\r\n$value\r\n";72 print $sock "set key$key 0 0 66560\r\n$value\r\n";
61 is(scalar <$sock>, "STORED\r\n", "stored key$key");73 is(scalar <$sock>, "STORED\r\n", "stored key$key");
diff --git a/t/misbehave.t b/t/misbehave.t
62old mode 10064474old mode 100644
63new mode 10075575new mode 100755
index ccf88f5..13cb7f3
--- a/t/misbehave.t
+++ b/t/misbehave.t
@@ -3,6 +3,7 @@
3use strict;3use strict;
4use Test::More;4use Test::More;
5use FindBin qw($Bin);5use FindBin qw($Bin);
6use Socket qw(MSG_PEEK MSG_DONTWAIT);
6use lib "$Bin/lib";7use lib "$Bin/lib";
7use MemcachedTest;8use MemcachedTest;
89
@@ -13,8 +14,15 @@ if (supports_drop_priv()) {
13 exit 0;14 exit 0;
14}15}
1516
16my $server = new_memcached();17my $server = new_memcached('-o drop_privileges');
17my $sock = $server->sock;18my $sock = $server->sock;
1819
19print $sock "misbehave\r\n";20print $sock "misbehave\r\n";
20is(scalar <$sock>, "OK\r\n", "did not allow misbehaving");21sleep(1);
22
23# check if the socket is dead now
24my $buff;
25my $ret = recv($sock, $buff, 1, MSG_PEEK | MSG_DONTWAIT);
26is($ret, undef, "did not allow misbehaving");
27
28$server->DESTROY();
diff --git a/thread.c b/thread.c
index d17a387..618ffac 100644
--- a/thread.c
+++ b/thread.c
@@ -139,22 +139,24 @@ void pause_threads(enum pause_thread_types type) {
139 buf[0] = 0;139 buf[0] = 0;
140 switch (type) {140 switch (type) {
141 case PAUSE_ALL_THREADS:141 case PAUSE_ALL_THREADS:
142 lru_maintainer_pause();
143 slabs_rebalancer_pause();142 slabs_rebalancer_pause();
143 lru_maintainer_pause();
144 lru_crawler_pause();144 lru_crawler_pause();
145#ifdef EXTSTORE145#ifdef EXTSTORE
146 storage_compact_pause();146 storage_compact_pause();
147 storage_write_pause();
147#endif148#endif
148 case PAUSE_WORKER_THREADS:149 case PAUSE_WORKER_THREADS:
149 buf[0] = 'p';150 buf[0] = 'p';
150 pthread_mutex_lock(&worker_hang_lock);151 pthread_mutex_lock(&worker_hang_lock);
151 break;152 break;
152 case RESUME_ALL_THREADS:153 case RESUME_ALL_THREADS:
153 lru_maintainer_resume();
154 slabs_rebalancer_resume();154 slabs_rebalancer_resume();
155 lru_maintainer_resume();
155 lru_crawler_resume();156 lru_crawler_resume();
156#ifdef EXTSTORE157#ifdef EXTSTORE
157 storage_compact_resume();158 storage_compact_resume();
159 storage_write_resume();
158#endif160#endif
159 case RESUME_WORKER_THREADS:161 case RESUME_WORKER_THREADS:
160 pthread_mutex_unlock(&worker_hang_lock);162 pthread_mutex_unlock(&worker_hang_lock);
@@ -625,7 +627,7 @@ void item_unlink(item *item) {
625 * Does arithmetic on a numeric item value.627 * Does arithmetic on a numeric item value.
626 */628 */
627enum delta_result_type add_delta(conn *c, const char *key,629enum delta_result_type add_delta(conn *c, const char *key,
628 const size_t nkey, int incr,630 const size_t nkey, bool incr,
629 const int64_t delta, char *buf,631 const int64_t delta, char *buf,
630 uint64_t *cas) {632 uint64_t *cas) {
631 enum delta_result_type ret;633 enum delta_result_type ret;
diff --git a/timedrun.c b/timedrun.c
index 6888453..a869004 100644
--- a/timedrun.c
+++ b/timedrun.c
@@ -7,11 +7,11 @@
77
8#include <assert.h>8#include <assert.h>
99
10static int caught = 0;10volatile sig_atomic_t caught_sig = 0;
1111
12static void caught_signal(int which)12static void signal_handler(int which)
13{13{
14 caught = which;14 caught_sig = which;
15}15}
1616
17static int wait_for_process(pid_t pid)17static int wait_for_process(pid_t pid)
@@ -21,7 +21,7 @@ static int wait_for_process(pid_t pid)
21 int i = 0;21 int i = 0;
22 struct sigaction sig_handler;22 struct sigaction sig_handler;
2323
24 sig_handler.sa_handler = caught_signal;24 sig_handler.sa_handler = signal_handler;
25 sig_handler.sa_flags = 0;25 sig_handler.sa_flags = 0;
2626
27 sigaction(SIGALRM, &sig_handler, NULL);27 sigaction(SIGALRM, &sig_handler, NULL);
@@ -44,8 +44,8 @@ static int wait_for_process(pid_t pid)
44 switch (i) {44 switch (i) {
45 case 0:45 case 0:
46 /* On the first iteration, pass the signal through */46 /* On the first iteration, pass the signal through */
47 sig = caught > 0 ? caught : SIGTERM;47 sig = caught_sig > 0 ? caught_sig : SIGTERM;
48 if (caught == SIGALRM) {48 if (caught_sig == SIGALRM) {
49 fprintf(stderr, "Timeout.. killing the process\n");49 fprintf(stderr, "Timeout.. killing the process\n");
50 }50 }
51 break;51 break;
diff --git a/version.m4 b/version.m4
index 4a02156..ff2636d 100644
--- a/version.m4
+++ b/version.m4
@@ -1 +1 @@
1m4_define([VERSION_NUMBER], [1.5.6])1m4_define([VERSION_NUMBER], [1.5.10])

Subscribers

People subscribed via source and target branches