Merge ~racb/ubuntu/+source/memcached:upstream-bump-1.5.10 into ubuntu/+source/memcached:ubuntu/devel
- Git
- lp:~racb/ubuntu/+source/memcached
- upstream-bump-1.5.10
- Merge into ubuntu/devel
Status: | Merged | ||||
---|---|---|---|---|---|
Merged at revision: | 05743ddcb21ad054c5102c4cd3660bea10c6f17e | ||||
Proposed branch: | ~racb/ubuntu/+source/memcached:upstream-bump-1.5.10 | ||||
Merge into: | ubuntu/+source/memcached:ubuntu/devel | ||||
Diff against target: |
3924 lines (+1746/-336) 46 files modified
INSTALL (+370/-0) Makefile.in (+2/-2) README.md (+2/-2) config.h.in (+7/-1) configure (+80/-11) configure.ac (+39/-3) crawler.c (+8/-1) crc32c.c (+72/-2) debian/changelog (+8/-0) debian/control (+2/-1) doc/Makefile (+3/-3) doc/memcached.1 (+2/-2) doc/protocol.txt (+2/-0) extstore.c (+97/-33) extstore.h (+14/-2) items.c (+8/-16) linux_priv.c (+46/-4) logger.c (+12/-3) logger.h (+1/-0) memcached.c (+155/-87) memcached.h (+34/-2) memcached.spec (+4/-4) sasl_defs.c (+6/-2) scripts/memcached-automove (+1/-1) scripts/memcached-automove-extstore (+1/-2) scripts/memcached-tool (+72/-58) scripts/memcached-tool.1 (+5/-2) slabs.c (+76/-3) storage.c (+220/-28) storage.h (+7/-1) t/binary-extstore.t (+10/-2) t/binary-sasl.t (+20/-7) t/chunked-extstore.t (+35/-20) t/error-extstore.t (+130/-0) t/extstore-buckets.t (+1/-1) t/extstore-jbod.t (+69/-0) t/extstore.t (+37/-9) t/flush-all.t (+1/-1) t/issue_67.t (+19/-6) t/lib/MemcachedTest.pm (+21/-1) t/lru-crawler.t (+13/-1) t/lru-maintainer.t (+12/-0) t/misbehave.t (+10/-2) thread.c (+5/-3) timedrun.c (+6/-6) version.m4 (+1/-1) |
||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Christian Ehrhardt (community) | Approve | ||
Canonical Server | Pending | ||
Review via email: mp+352972@code.launchpad.net |
Commit message
Description of the change
This includes fixes for some issues which were causing build time test failures, preventing memcached from migrating to the release pocket. Thanks to upstream working closely with us, all the fixes are released upstream, so we just pull in the latest and it works.
After Debian takes this we should be able to sync.
Robie Basak (racb) wrote : | # |
Christian Ehrhardt (paelzer) wrote : | # |
Hi,
a few things inline - mostly suggestions for the changelog.
The bump itself seems correct to me - content change wise.
The md5sum of the upstream download matches:
$ md5sum memcached-
8462616b554183a
8462616b554183a
From:
http://
https:/
No other functional changes required.
The inline comments are all style-questions which you can decide either way.
I'll do some testing before a full +1
Christian Ehrhardt (paelzer) wrote : | # |
I copied from your ppa that was identical to the proposed changes.
Testing from that ...
qa tests:
- I ran against the old version in cosmic 1.5.6-0ubuntu1 to know which might be known issues and if the test is repeatable
- of these memcdump had a fail, but all others worked
- after upgrading to the ppa version (upgrade worked) I reran the tests
- all tests that worked still work (no change on memcdump)
Running test: './test-
Skipping private tests
test_add (__main_
Test adding data ... ok
test_cve_2011_4971 (__main_
Test CVE-2011-4971 ... ok
test_default_tcp (__main_
Test that memcached is listening on tcp port by default ... ok
test_default_udp (__main_
Test that memcached is listening on udp port by default ... ok
test_memccapable (__main_
Test memccapable ... ok
test_memccat (__main_
Test memccat ... ok
test_memcdump (__main_
Test memcdump ... FAIL
test_memcrm (__main_
Test memcrm ... ok
test_memcstat (__main_
Test memcstat ... ok
test_sample_data (__main_
Test sample data ... ok
test_set (__main_
Test setting data ... ok
Christian Ehrhardt (paelzer) wrote : | # |
Autopkgtests are enqueued, will check results after lunch.
Christian Ehrhardt (paelzer) wrote : | # |
Autopkgtests are also good:
flask-limiter/
gnocchi/
memcached/
pgmemcache/2.3.0-4
python-limits/1.3-1
That said, code good, changelog ok (up to you to adapt my suggestions) and tests good - +1 on the MP
Christian Ehrhardt (paelzer) wrote : | # |
FYI - test link https:/
Preview Diff
1 | diff --git a/INSTALL b/INSTALL | |||
2 | 0 | new file mode 100644 | 0 | new file mode 100644 |
3 | index 0000000..2099840 | |||
4 | --- /dev/null | |||
5 | +++ b/INSTALL | |||
6 | @@ -0,0 +1,370 @@ | |||
7 | 1 | Installation Instructions | ||
8 | 2 | ************************* | ||
9 | 3 | |||
10 | 4 | Copyright (C) 1994-1996, 1999-2002, 2004-2013 Free Software Foundation, | ||
11 | 5 | Inc. | ||
12 | 6 | |||
13 | 7 | Copying and distribution of this file, with or without modification, | ||
14 | 8 | are permitted in any medium without royalty provided the copyright | ||
15 | 9 | notice and this notice are preserved. This file is offered as-is, | ||
16 | 10 | without warranty of any kind. | ||
17 | 11 | |||
18 | 12 | Basic Installation | ||
19 | 13 | ================== | ||
20 | 14 | |||
21 | 15 | Briefly, the shell command `./configure && make && make install' | ||
22 | 16 | should configure, build, and install this package. The following | ||
23 | 17 | more-detailed instructions are generic; see the `README' file for | ||
24 | 18 | instructions specific to this package. Some packages provide this | ||
25 | 19 | `INSTALL' file but do not implement all of the features documented | ||
26 | 20 | below. The lack of an optional feature in a given package is not | ||
27 | 21 | necessarily a bug. More recommendations for GNU packages can be found | ||
28 | 22 | in *note Makefile Conventions: (standards)Makefile Conventions. | ||
29 | 23 | |||
30 | 24 | The `configure' shell script attempts to guess correct values for | ||
31 | 25 | various system-dependent variables used during compilation. It uses | ||
32 | 26 | those values to create a `Makefile' in each directory of the package. | ||
33 | 27 | It may also create one or more `.h' files containing system-dependent | ||
34 | 28 | definitions. Finally, it creates a shell script `config.status' that | ||
35 | 29 | you can run in the future to recreate the current configuration, and a | ||
36 | 30 | file `config.log' containing compiler output (useful mainly for | ||
37 | 31 | debugging `configure'). | ||
38 | 32 | |||
39 | 33 | It can also use an optional file (typically called `config.cache' | ||
40 | 34 | and enabled with `--cache-file=config.cache' or simply `-C') that saves | ||
41 | 35 | the results of its tests to speed up reconfiguring. Caching is | ||
42 | 36 | disabled by default to prevent problems with accidental use of stale | ||
43 | 37 | cache files. | ||
44 | 38 | |||
45 | 39 | If you need to do unusual things to compile the package, please try | ||
46 | 40 | to figure out how `configure' could check whether to do them, and mail | ||
47 | 41 | diffs or instructions to the address given in the `README' so they can | ||
48 | 42 | be considered for the next release. If you are using the cache, and at | ||
49 | 43 | some point `config.cache' contains results you don't want to keep, you | ||
50 | 44 | may remove or edit it. | ||
51 | 45 | |||
52 | 46 | The file `configure.ac' (or `configure.in') is used to create | ||
53 | 47 | `configure' by a program called `autoconf'. You need `configure.ac' if | ||
54 | 48 | you want to change it or regenerate `configure' using a newer version | ||
55 | 49 | of `autoconf'. | ||
56 | 50 | |||
57 | 51 | The simplest way to compile this package is: | ||
58 | 52 | |||
59 | 53 | 1. `cd' to the directory containing the package's source code and type | ||
60 | 54 | `./configure' to configure the package for your system. | ||
61 | 55 | |||
62 | 56 | Running `configure' might take a while. While running, it prints | ||
63 | 57 | some messages telling which features it is checking for. | ||
64 | 58 | |||
65 | 59 | 2. Type `make' to compile the package. | ||
66 | 60 | |||
67 | 61 | 3. Optionally, type `make check' to run any self-tests that come with | ||
68 | 62 | the package, generally using the just-built uninstalled binaries. | ||
69 | 63 | |||
70 | 64 | 4. Type `make install' to install the programs and any data files and | ||
71 | 65 | documentation. When installing into a prefix owned by root, it is | ||
72 | 66 | recommended that the package be configured and built as a regular | ||
73 | 67 | user, and only the `make install' phase executed with root | ||
74 | 68 | privileges. | ||
75 | 69 | |||
76 | 70 | 5. Optionally, type `make installcheck' to repeat any self-tests, but | ||
77 | 71 | this time using the binaries in their final installed location. | ||
78 | 72 | This target does not install anything. Running this target as a | ||
79 | 73 | regular user, particularly if the prior `make install' required | ||
80 | 74 | root privileges, verifies that the installation completed | ||
81 | 75 | correctly. | ||
82 | 76 | |||
83 | 77 | 6. You can remove the program binaries and object files from the | ||
84 | 78 | source code directory by typing `make clean'. To also remove the | ||
85 | 79 | files that `configure' created (so you can compile the package for | ||
86 | 80 | a different kind of computer), type `make distclean'. There is | ||
87 | 81 | also a `make maintainer-clean' target, but that is intended mainly | ||
88 | 82 | for the package's developers. If you use it, you may have to get | ||
89 | 83 | all sorts of other programs in order to regenerate files that came | ||
90 | 84 | with the distribution. | ||
91 | 85 | |||
92 | 86 | 7. Often, you can also type `make uninstall' to remove the installed | ||
93 | 87 | files again. In practice, not all packages have tested that | ||
94 | 88 | uninstallation works correctly, even though it is required by the | ||
95 | 89 | GNU Coding Standards. | ||
96 | 90 | |||
97 | 91 | 8. Some packages, particularly those that use Automake, provide `make | ||
98 | 92 | distcheck', which can by used by developers to test that all other | ||
99 | 93 | targets like `make install' and `make uninstall' work correctly. | ||
100 | 94 | This target is generally not run by end users. | ||
101 | 95 | |||
102 | 96 | Compilers and Options | ||
103 | 97 | ===================== | ||
104 | 98 | |||
105 | 99 | Some systems require unusual options for compilation or linking that | ||
106 | 100 | the `configure' script does not know about. Run `./configure --help' | ||
107 | 101 | for details on some of the pertinent environment variables. | ||
108 | 102 | |||
109 | 103 | You can give `configure' initial values for configuration parameters | ||
110 | 104 | by setting variables in the command line or in the environment. Here | ||
111 | 105 | is an example: | ||
112 | 106 | |||
113 | 107 | ./configure CC=c99 CFLAGS=-g LIBS=-lposix | ||
114 | 108 | |||
115 | 109 | *Note Defining Variables::, for more details. | ||
116 | 110 | |||
117 | 111 | Compiling For Multiple Architectures | ||
118 | 112 | ==================================== | ||
119 | 113 | |||
120 | 114 | You can compile the package for more than one kind of computer at the | ||
121 | 115 | same time, by placing the object files for each architecture in their | ||
122 | 116 | own directory. To do this, you can use GNU `make'. `cd' to the | ||
123 | 117 | directory where you want the object files and executables to go and run | ||
124 | 118 | the `configure' script. `configure' automatically checks for the | ||
125 | 119 | source code in the directory that `configure' is in and in `..'. This | ||
126 | 120 | is known as a "VPATH" build. | ||
127 | 121 | |||
128 | 122 | With a non-GNU `make', it is safer to compile the package for one | ||
129 | 123 | architecture at a time in the source code directory. After you have | ||
130 | 124 | installed the package for one architecture, use `make distclean' before | ||
131 | 125 | reconfiguring for another architecture. | ||
132 | 126 | |||
133 | 127 | On MacOS X 10.5 and later systems, you can create libraries and | ||
134 | 128 | executables that work on multiple system types--known as "fat" or | ||
135 | 129 | "universal" binaries--by specifying multiple `-arch' options to the | ||
136 | 130 | compiler but only a single `-arch' option to the preprocessor. Like | ||
137 | 131 | this: | ||
138 | 132 | |||
139 | 133 | ./configure CC="gcc -arch i386 -arch x86_64 -arch ppc -arch ppc64" \ | ||
140 | 134 | CXX="g++ -arch i386 -arch x86_64 -arch ppc -arch ppc64" \ | ||
141 | 135 | CPP="gcc -E" CXXCPP="g++ -E" | ||
142 | 136 | |||
143 | 137 | This is not guaranteed to produce working output in all cases, you | ||
144 | 138 | may have to build one architecture at a time and combine the results | ||
145 | 139 | using the `lipo' tool if you have problems. | ||
146 | 140 | |||
147 | 141 | Installation Names | ||
148 | 142 | ================== | ||
149 | 143 | |||
150 | 144 | By default, `make install' installs the package's commands under | ||
151 | 145 | `/usr/local/bin', include files under `/usr/local/include', etc. You | ||
152 | 146 | can specify an installation prefix other than `/usr/local' by giving | ||
153 | 147 | `configure' the option `--prefix=PREFIX', where PREFIX must be an | ||
154 | 148 | absolute file name. | ||
155 | 149 | |||
156 | 150 | You can specify separate installation prefixes for | ||
157 | 151 | architecture-specific files and architecture-independent files. If you | ||
158 | 152 | pass the option `--exec-prefix=PREFIX' to `configure', the package uses | ||
159 | 153 | PREFIX as the prefix for installing programs and libraries. | ||
160 | 154 | Documentation and other data files still use the regular prefix. | ||
161 | 155 | |||
162 | 156 | In addition, if you use an unusual directory layout you can give | ||
163 | 157 | options like `--bindir=DIR' to specify different values for particular | ||
164 | 158 | kinds of files. Run `configure --help' for a list of the directories | ||
165 | 159 | you can set and what kinds of files go in them. In general, the | ||
166 | 160 | default for these options is expressed in terms of `${prefix}', so that | ||
167 | 161 | specifying just `--prefix' will affect all of the other directory | ||
168 | 162 | specifications that were not explicitly provided. | ||
169 | 163 | |||
170 | 164 | The most portable way to affect installation locations is to pass the | ||
171 | 165 | correct locations to `configure'; however, many packages provide one or | ||
172 | 166 | both of the following shortcuts of passing variable assignments to the | ||
173 | 167 | `make install' command line to change installation locations without | ||
174 | 168 | having to reconfigure or recompile. | ||
175 | 169 | |||
176 | 170 | The first method involves providing an override variable for each | ||
177 | 171 | affected directory. For example, `make install | ||
178 | 172 | prefix=/alternate/directory' will choose an alternate location for all | ||
179 | 173 | directory configuration variables that were expressed in terms of | ||
180 | 174 | `${prefix}'. Any directories that were specified during `configure', | ||
181 | 175 | but not in terms of `${prefix}', must each be overridden at install | ||
182 | 176 | time for the entire installation to be relocated. The approach of | ||
183 | 177 | makefile variable overrides for each directory variable is required by | ||
184 | 178 | the GNU Coding Standards, and ideally causes no recompilation. | ||
185 | 179 | However, some platforms have known limitations with the semantics of | ||
186 | 180 | shared libraries that end up requiring recompilation when using this | ||
187 | 181 | method, particularly noticeable in packages that use GNU Libtool. | ||
188 | 182 | |||
189 | 183 | The second method involves providing the `DESTDIR' variable. For | ||
190 | 184 | example, `make install DESTDIR=/alternate/directory' will prepend | ||
191 | 185 | `/alternate/directory' before all installation names. The approach of | ||
192 | 186 | `DESTDIR' overrides is not required by the GNU Coding Standards, and | ||
193 | 187 | does not work on platforms that have drive letters. On the other hand, | ||
194 | 188 | it does better at avoiding recompilation issues, and works well even | ||
195 | 189 | when some directory options were not specified in terms of `${prefix}' | ||
196 | 190 | at `configure' time. | ||
197 | 191 | |||
198 | 192 | Optional Features | ||
199 | 193 | ================= | ||
200 | 194 | |||
201 | 195 | If the package supports it, you can cause programs to be installed | ||
202 | 196 | with an extra prefix or suffix on their names by giving `configure' the | ||
203 | 197 | option `--program-prefix=PREFIX' or `--program-suffix=SUFFIX'. | ||
204 | 198 | |||
205 | 199 | Some packages pay attention to `--enable-FEATURE' options to | ||
206 | 200 | `configure', where FEATURE indicates an optional part of the package. | ||
207 | 201 | They may also pay attention to `--with-PACKAGE' options, where PACKAGE | ||
208 | 202 | is something like `gnu-as' or `x' (for the X Window System). The | ||
209 | 203 | `README' should mention any `--enable-' and `--with-' options that the | ||
210 | 204 | package recognizes. | ||
211 | 205 | |||
212 | 206 | For packages that use the X Window System, `configure' can usually | ||
213 | 207 | find the X include and library files automatically, but if it doesn't, | ||
214 | 208 | you can use the `configure' options `--x-includes=DIR' and | ||
215 | 209 | `--x-libraries=DIR' to specify their locations. | ||
216 | 210 | |||
217 | 211 | Some packages offer the ability to configure how verbose the | ||
218 | 212 | execution of `make' will be. For these packages, running `./configure | ||
219 | 213 | --enable-silent-rules' sets the default to minimal output, which can be | ||
220 | 214 | overridden with `make V=1'; while running `./configure | ||
221 | 215 | --disable-silent-rules' sets the default to verbose, which can be | ||
222 | 216 | overridden with `make V=0'. | ||
223 | 217 | |||
224 | 218 | Particular systems | ||
225 | 219 | ================== | ||
226 | 220 | |||
227 | 221 | On HP-UX, the default C compiler is not ANSI C compatible. If GNU | ||
228 | 222 | CC is not installed, it is recommended to use the following options in | ||
229 | 223 | order to use an ANSI C compiler: | ||
230 | 224 | |||
231 | 225 | ./configure CC="cc -Ae -D_XOPEN_SOURCE=500" | ||
232 | 226 | |||
233 | 227 | and if that doesn't work, install pre-built binaries of GCC for HP-UX. | ||
234 | 228 | |||
235 | 229 | HP-UX `make' updates targets which have the same time stamps as | ||
236 | 230 | their prerequisites, which makes it generally unusable when shipped | ||
237 | 231 | generated files such as `configure' are involved. Use GNU `make' | ||
238 | 232 | instead. | ||
239 | 233 | |||
240 | 234 | On OSF/1 a.k.a. Tru64, some versions of the default C compiler cannot | ||
241 | 235 | parse its `<wchar.h>' header file. The option `-nodtk' can be used as | ||
242 | 236 | a workaround. If GNU CC is not installed, it is therefore recommended | ||
243 | 237 | to try | ||
244 | 238 | |||
245 | 239 | ./configure CC="cc" | ||
246 | 240 | |||
247 | 241 | and if that doesn't work, try | ||
248 | 242 | |||
249 | 243 | ./configure CC="cc -nodtk" | ||
250 | 244 | |||
251 | 245 | On Solaris, don't put `/usr/ucb' early in your `PATH'. This | ||
252 | 246 | directory contains several dysfunctional programs; working variants of | ||
253 | 247 | these programs are available in `/usr/bin'. So, if you need `/usr/ucb' | ||
254 | 248 | in your `PATH', put it _after_ `/usr/bin'. | ||
255 | 249 | |||
256 | 250 | On Haiku, software installed for all users goes in `/boot/common', | ||
257 | 251 | not `/usr/local'. It is recommended to use the following options: | ||
258 | 252 | |||
259 | 253 | ./configure --prefix=/boot/common | ||
260 | 254 | |||
261 | 255 | Specifying the System Type | ||
262 | 256 | ========================== | ||
263 | 257 | |||
264 | 258 | There may be some features `configure' cannot figure out | ||
265 | 259 | automatically, but needs to determine by the type of machine the package | ||
266 | 260 | will run on. Usually, assuming the package is built to be run on the | ||
267 | 261 | _same_ architectures, `configure' can figure that out, but if it prints | ||
268 | 262 | a message saying it cannot guess the machine type, give it the | ||
269 | 263 | `--build=TYPE' option. TYPE can either be a short name for the system | ||
270 | 264 | type, such as `sun4', or a canonical name which has the form: | ||
271 | 265 | |||
272 | 266 | CPU-COMPANY-SYSTEM | ||
273 | 267 | |||
274 | 268 | where SYSTEM can have one of these forms: | ||
275 | 269 | |||
276 | 270 | OS | ||
277 | 271 | KERNEL-OS | ||
278 | 272 | |||
279 | 273 | See the file `config.sub' for the possible values of each field. If | ||
280 | 274 | `config.sub' isn't included in this package, then this package doesn't | ||
281 | 275 | need to know the machine type. | ||
282 | 276 | |||
283 | 277 | If you are _building_ compiler tools for cross-compiling, you should | ||
284 | 278 | use the option `--target=TYPE' to select the type of system they will | ||
285 | 279 | produce code for. | ||
286 | 280 | |||
287 | 281 | If you want to _use_ a cross compiler, that generates code for a | ||
288 | 282 | platform different from the build platform, you should specify the | ||
289 | 283 | "host" platform (i.e., that on which the generated programs will | ||
290 | 284 | eventually be run) with `--host=TYPE'. | ||
291 | 285 | |||
292 | 286 | Sharing Defaults | ||
293 | 287 | ================ | ||
294 | 288 | |||
295 | 289 | If you want to set default values for `configure' scripts to share, | ||
296 | 290 | you can create a site shell script called `config.site' that gives | ||
297 | 291 | default values for variables like `CC', `cache_file', and `prefix'. | ||
298 | 292 | `configure' looks for `PREFIX/share/config.site' if it exists, then | ||
299 | 293 | `PREFIX/etc/config.site' if it exists. Or, you can set the | ||
300 | 294 | `CONFIG_SITE' environment variable to the location of the site script. | ||
301 | 295 | A warning: not all `configure' scripts look for a site script. | ||
302 | 296 | |||
303 | 297 | Defining Variables | ||
304 | 298 | ================== | ||
305 | 299 | |||
306 | 300 | Variables not defined in a site shell script can be set in the | ||
307 | 301 | environment passed to `configure'. However, some packages may run | ||
308 | 302 | configure again during the build, and the customized values of these | ||
309 | 303 | variables may be lost. In order to avoid this problem, you should set | ||
310 | 304 | them in the `configure' command line, using `VAR=value'. For example: | ||
311 | 305 | |||
312 | 306 | ./configure CC=/usr/local2/bin/gcc | ||
313 | 307 | |||
314 | 308 | causes the specified `gcc' to be used as the C compiler (unless it is | ||
315 | 309 | overridden in the site shell script). | ||
316 | 310 | |||
317 | 311 | Unfortunately, this technique does not work for `CONFIG_SHELL' due to | ||
318 | 312 | an Autoconf limitation. Until the limitation is lifted, you can use | ||
319 | 313 | this workaround: | ||
320 | 314 | |||
321 | 315 | CONFIG_SHELL=/bin/bash ./configure CONFIG_SHELL=/bin/bash | ||
322 | 316 | |||
323 | 317 | `configure' Invocation | ||
324 | 318 | ====================== | ||
325 | 319 | |||
326 | 320 | `configure' recognizes the following options to control how it | ||
327 | 321 | operates. | ||
328 | 322 | |||
329 | 323 | `--help' | ||
330 | 324 | `-h' | ||
331 | 325 | Print a summary of all of the options to `configure', and exit. | ||
332 | 326 | |||
333 | 327 | `--help=short' | ||
334 | 328 | `--help=recursive' | ||
335 | 329 | Print a summary of the options unique to this package's | ||
336 | 330 | `configure', and exit. The `short' variant lists options used | ||
337 | 331 | only in the top level, while the `recursive' variant lists options | ||
338 | 332 | also present in any nested packages. | ||
339 | 333 | |||
340 | 334 | `--version' | ||
341 | 335 | `-V' | ||
342 | 336 | Print the version of Autoconf used to generate the `configure' | ||
343 | 337 | script, and exit. | ||
344 | 338 | |||
345 | 339 | `--cache-file=FILE' | ||
346 | 340 | Enable the cache: use and save the results of the tests in FILE, | ||
347 | 341 | traditionally `config.cache'. FILE defaults to `/dev/null' to | ||
348 | 342 | disable caching. | ||
349 | 343 | |||
350 | 344 | `--config-cache' | ||
351 | 345 | `-C' | ||
352 | 346 | Alias for `--cache-file=config.cache'. | ||
353 | 347 | |||
354 | 348 | `--quiet' | ||
355 | 349 | `--silent' | ||
356 | 350 | `-q' | ||
357 | 351 | Do not print messages saying which checks are being made. To | ||
358 | 352 | suppress all normal output, redirect it to `/dev/null' (any error | ||
359 | 353 | messages will still be shown). | ||
360 | 354 | |||
361 | 355 | `--srcdir=DIR' | ||
362 | 356 | Look for the package's source code in directory DIR. Usually | ||
363 | 357 | `configure' can determine that directory automatically. | ||
364 | 358 | |||
365 | 359 | `--prefix=DIR' | ||
366 | 360 | Use DIR as the installation prefix. *note Installation Names:: | ||
367 | 361 | for more details, including other options available for fine-tuning | ||
368 | 362 | the installation locations. | ||
369 | 363 | |||
370 | 364 | `--no-create' | ||
371 | 365 | `-n' | ||
372 | 366 | Run the configure checks, but stop before creating any output | ||
373 | 367 | files. | ||
374 | 368 | |||
375 | 369 | `configure' also accepts some other, not widely useful, options. Run | ||
376 | 370 | `configure --help' for more details. | ||
377 | diff --git a/Makefile.in b/Makefile.in | |||
378 | index 966b4d7..c125ace 100644 | |||
379 | --- a/Makefile.in | |||
380 | +++ b/Makefile.in | |||
381 | @@ -327,8 +327,8 @@ CTAGS = ctags | |||
382 | 327 | CSCOPE = cscope | 327 | CSCOPE = cscope |
383 | 328 | DIST_SUBDIRS = $(SUBDIRS) | 328 | DIST_SUBDIRS = $(SUBDIRS) |
384 | 329 | am__DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/config.h.in AUTHORS \ | 329 | am__DIST_COMMON = $(srcdir)/Makefile.in $(srcdir)/config.h.in AUTHORS \ |
387 | 330 | COPYING ChangeLog NEWS compile config.guess config.sub depcomp \ | 330 | COPYING ChangeLog INSTALL NEWS compile config.guess config.sub \ |
388 | 331 | install-sh missing | 331 | depcomp install-sh missing |
389 | 332 | DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) | 332 | DISTFILES = $(DIST_COMMON) $(DIST_SOURCES) $(TEXINFOS) $(EXTRA_DIST) |
390 | 333 | distdir = $(PACKAGE)-$(VERSION) | 333 | distdir = $(PACKAGE)-$(VERSION) |
391 | 334 | top_distdir = $(distdir) | 334 | top_distdir = $(distdir) |
392 | diff --git a/README.md b/README.md | |||
393 | index 935e4cd..e330fbe 100644 | |||
394 | --- a/README.md | |||
395 | +++ b/README.md | |||
396 | @@ -18,8 +18,8 @@ list to ask questions, github issues aren't seen by everyone! | |||
397 | 18 | ## Dependencies | 18 | ## Dependencies |
398 | 19 | 19 | ||
399 | 20 | * libevent, http://www.monkey.org/~provos/libevent/ (libevent-dev) | 20 | * libevent, http://www.monkey.org/~provos/libevent/ (libevent-dev) |
402 | 21 | * libseccomp, (optional, linux) - enables process restrictions for better | 21 | * libseccomp, (optional, experimental, linux) - enables process restrictions for |
403 | 22 | security. | 22 | better security. Tested only on x86_64 architectures. |
404 | 23 | 23 | ||
405 | 24 | ## Environment | 24 | ## Environment |
406 | 25 | 25 | ||
407 | diff --git a/config.h.in b/config.h.in | |||
408 | index 9638dfa..3f3d42d 100644 | |||
409 | --- a/config.h.in | |||
410 | +++ b/config.h.in | |||
411 | @@ -1,5 +1,8 @@ | |||
412 | 1 | /* config.h.in. Generated from configure.ac by autoheader. */ | 1 | /* config.h.in. Generated from configure.ac by autoheader. */ |
413 | 2 | 2 | ||
414 | 3 | /* Set to nonzero if you want to enable ARMv8 crc32 */ | ||
415 | 4 | #undef ARM_CRC32 | ||
416 | 5 | |||
417 | 3 | /* Set to nonzero if you want to include DTRACE */ | 6 | /* Set to nonzero if you want to include DTRACE */ |
418 | 4 | #undef ENABLE_DTRACE | 7 | #undef ENABLE_DTRACE |
419 | 5 | 8 | ||
420 | @@ -15,7 +18,7 @@ | |||
421 | 15 | /* machine is littleendian */ | 18 | /* machine is littleendian */ |
422 | 16 | #undef ENDIAN_LITTLE | 19 | #undef ENDIAN_LITTLE |
423 | 17 | 20 | ||
425 | 18 | /* Set to nonzero if you want to enable extstorextstore */ | 21 | /* Set to nonzero if you want to enable extstore */ |
426 | 19 | #undef EXTSTORE | 22 | #undef EXTSTORE |
427 | 20 | 23 | ||
428 | 21 | /* Define to 1 if support accept4 */ | 24 | /* Define to 1 if support accept4 */ |
429 | @@ -66,6 +69,9 @@ | |||
430 | 66 | /* Set to nonzero if your SASL implementation supports SASL_CB_GETCONF */ | 69 | /* Set to nonzero if your SASL implementation supports SASL_CB_GETCONF */ |
431 | 67 | #undef HAVE_SASL_CB_GETCONF | 70 | #undef HAVE_SASL_CB_GETCONF |
432 | 68 | 71 | ||
433 | 72 | /* Set to nonzero if your SASL implementation supports SASL_CB_GETCONFPATH */ | ||
434 | 73 | #undef HAVE_SASL_CB_GETCONFPATH | ||
435 | 74 | |||
436 | 69 | /* Define to 1 if you have the <sasl/sasl.h> header file. */ | 75 | /* Define to 1 if you have the <sasl/sasl.h> header file. */ |
437 | 70 | #undef HAVE_SASL_SASL_H | 76 | #undef HAVE_SASL_SASL_H |
438 | 71 | 77 | ||
439 | diff --git a/configure b/configure | |||
440 | index e479e60..9ccb4dd 100755 | |||
441 | --- a/configure | |||
442 | +++ b/configure | |||
443 | @@ -1,6 +1,6 @@ | |||
444 | 1 | #! /bin/sh | 1 | #! /bin/sh |
445 | 2 | # Guess values for system-dependent variables and create Makefiles. | 2 | # Guess values for system-dependent variables and create Makefiles. |
447 | 3 | # Generated by GNU Autoconf 2.69 for memcached 1.5.6. | 3 | # Generated by GNU Autoconf 2.69 for memcached 1.5.10. |
448 | 4 | # | 4 | # |
449 | 5 | # Report bugs to <memcached@googlegroups.com>. | 5 | # Report bugs to <memcached@googlegroups.com>. |
450 | 6 | # | 6 | # |
451 | @@ -580,8 +580,8 @@ MAKEFLAGS= | |||
452 | 580 | # Identity of this package. | 580 | # Identity of this package. |
453 | 581 | PACKAGE_NAME='memcached' | 581 | PACKAGE_NAME='memcached' |
454 | 582 | PACKAGE_TARNAME='memcached' | 582 | PACKAGE_TARNAME='memcached' |
457 | 583 | PACKAGE_VERSION='1.5.6' | 583 | PACKAGE_VERSION='1.5.10' |
458 | 584 | PACKAGE_STRING='memcached 1.5.6' | 584 | PACKAGE_STRING='memcached 1.5.10' |
459 | 585 | PACKAGE_BUGREPORT='memcached@googlegroups.com' | 585 | PACKAGE_BUGREPORT='memcached@googlegroups.com' |
460 | 586 | PACKAGE_URL='' | 586 | PACKAGE_URL='' |
461 | 587 | 587 | ||
462 | @@ -643,6 +643,8 @@ PROFILER | |||
463 | 643 | PROFILER_LDFLAGS | 643 | PROFILER_LDFLAGS |
464 | 644 | ENABLE_SASL | 644 | ENABLE_SASL |
465 | 645 | DTRACEFLAGS | 645 | DTRACEFLAGS |
466 | 646 | ENABLE_ARM_CRC32_FALSE | ||
467 | 647 | ENABLE_ARM_CRC32_TRUE | ||
468 | 646 | ENABLE_EXTSTORE_FALSE | 648 | ENABLE_EXTSTORE_FALSE |
469 | 647 | ENABLE_EXTSTORE_TRUE | 649 | ENABLE_EXTSTORE_TRUE |
470 | 648 | ENABLE_SASL_FALSE | 650 | ENABLE_SASL_FALSE |
471 | @@ -751,6 +753,7 @@ ac_user_opts=' | |||
472 | 751 | enable_option_checking | 753 | enable_option_checking |
473 | 752 | enable_silent_rules | 754 | enable_silent_rules |
474 | 753 | enable_dependency_tracking | 755 | enable_dependency_tracking |
475 | 756 | enable_arm_crc32 | ||
476 | 754 | enable_extstore | 757 | enable_extstore |
477 | 755 | enable_seccomp | 758 | enable_seccomp |
478 | 756 | enable_sasl | 759 | enable_sasl |
479 | @@ -1320,7 +1323,7 @@ if test "$ac_init_help" = "long"; then | |||
480 | 1320 | # Omit some internal or obsolete options to make the list less imposing. | 1323 | # Omit some internal or obsolete options to make the list less imposing. |
481 | 1321 | # This message is too long to be a string in the A/UX 3.1 sh. | 1324 | # This message is too long to be a string in the A/UX 3.1 sh. |
482 | 1322 | cat <<_ACEOF | 1325 | cat <<_ACEOF |
484 | 1323 | \`configure' configures memcached 1.5.6 to adapt to many kinds of systems. | 1326 | \`configure' configures memcached 1.5.10 to adapt to many kinds of systems. |
485 | 1324 | 1327 | ||
486 | 1325 | Usage: $0 [OPTION]... [VAR=VALUE]... | 1328 | Usage: $0 [OPTION]... [VAR=VALUE]... |
487 | 1326 | 1329 | ||
488 | @@ -1391,7 +1394,7 @@ fi | |||
489 | 1391 | 1394 | ||
490 | 1392 | if test -n "$ac_init_help"; then | 1395 | if test -n "$ac_init_help"; then |
491 | 1393 | case $ac_init_help in | 1396 | case $ac_init_help in |
493 | 1394 | short | recursive ) echo "Configuration of memcached 1.5.6:";; | 1397 | short | recursive ) echo "Configuration of memcached 1.5.10:";; |
494 | 1395 | esac | 1398 | esac |
495 | 1396 | cat <<\_ACEOF | 1399 | cat <<\_ACEOF |
496 | 1397 | 1400 | ||
497 | @@ -1405,8 +1408,9 @@ Optional Features: | |||
498 | 1405 | do not reject slow dependency extractors | 1408 | do not reject slow dependency extractors |
499 | 1406 | --disable-dependency-tracking | 1409 | --disable-dependency-tracking |
500 | 1407 | speeds up one-time build | 1410 | speeds up one-time build |
501 | 1411 | --enable-arm-crc32 Enable ARMv8 CRC32 instructions | ||
502 | 1408 | --enable-extstore Enable external storage EXPERIMENTAL | 1412 | --enable-extstore Enable external storage EXPERIMENTAL |
504 | 1409 | --enable-seccomp Enable seccomp restrictions | 1413 | --enable-seccomp Enable seccomp restrictions EXPERIMENTAL |
505 | 1410 | --enable-sasl Enable SASL authentication | 1414 | --enable-sasl Enable SASL authentication |
506 | 1411 | --enable-sasl-pwdb Enable plaintext password db | 1415 | --enable-sasl-pwdb Enable plaintext password db |
507 | 1412 | --enable-dtrace Enable dtrace probes | 1416 | --enable-dtrace Enable dtrace probes |
508 | @@ -1495,7 +1499,7 @@ fi | |||
509 | 1495 | test -n "$ac_init_help" && exit $ac_status | 1499 | test -n "$ac_init_help" && exit $ac_status |
510 | 1496 | if $ac_init_version; then | 1500 | if $ac_init_version; then |
511 | 1497 | cat <<\_ACEOF | 1501 | cat <<\_ACEOF |
513 | 1498 | memcached configure 1.5.6 | 1502 | memcached configure 1.5.10 |
514 | 1499 | generated by GNU Autoconf 2.69 | 1503 | generated by GNU Autoconf 2.69 |
515 | 1500 | 1504 | ||
516 | 1501 | Copyright (C) 2012 Free Software Foundation, Inc. | 1505 | Copyright (C) 2012 Free Software Foundation, Inc. |
517 | @@ -1964,7 +1968,7 @@ cat >config.log <<_ACEOF | |||
518 | 1964 | This file contains any messages produced by compilers while | 1968 | This file contains any messages produced by compilers while |
519 | 1965 | running configure, to aid debugging if configure makes a mistake. | 1969 | running configure, to aid debugging if configure makes a mistake. |
520 | 1966 | 1970 | ||
522 | 1967 | It was created by memcached $as_me 1.5.6, which was | 1971 | It was created by memcached $as_me 1.5.10, which was |
523 | 1968 | generated by GNU Autoconf 2.69. Invocation command line was | 1972 | generated by GNU Autoconf 2.69. Invocation command line was |
524 | 1969 | 1973 | ||
525 | 1970 | $ $0 $@ | 1974 | $ $0 $@ |
526 | @@ -2899,7 +2903,7 @@ fi | |||
527 | 2899 | 2903 | ||
528 | 2900 | # Define the identity of the package. | 2904 | # Define the identity of the package. |
529 | 2901 | PACKAGE='memcached' | 2905 | PACKAGE='memcached' |
531 | 2902 | VERSION='1.5.6' | 2906 | VERSION='1.5.10' |
532 | 2903 | 2907 | ||
533 | 2904 | 2908 | ||
534 | 2905 | cat >>confdefs.h <<_ACEOF | 2909 | cat >>confdefs.h <<_ACEOF |
535 | @@ -4701,6 +4705,12 @@ fi | |||
536 | 4701 | 4705 | ||
537 | 4702 | 4706 | ||
538 | 4703 | 4707 | ||
539 | 4708 | # Check whether --enable-arm_crc32 was given. | ||
540 | 4709 | if test "${enable_arm_crc32+set}" = set; then : | ||
541 | 4710 | enableval=$enable_arm_crc32; | ||
542 | 4711 | fi | ||
543 | 4712 | |||
544 | 4713 | |||
545 | 4704 | # Check whether --enable-extstore was given. | 4714 | # Check whether --enable-extstore was given. |
546 | 4705 | if test "${enable_extstore+set}" = set; then : | 4715 | if test "${enable_extstore+set}" = set; then : |
547 | 4706 | enableval=$enable_extstore; | 4716 | enableval=$enable_extstore; |
548 | @@ -4733,6 +4743,8 @@ fi | |||
549 | 4733 | 4743 | ||
550 | 4734 | 4744 | ||
551 | 4735 | 4745 | ||
552 | 4746 | |||
553 | 4747 | |||
554 | 4736 | for ac_header in sasl/sasl.h | 4748 | for ac_header in sasl/sasl.h |
555 | 4737 | do : | 4749 | do : |
556 | 4738 | ac_fn_c_check_header_mongrel "$LINENO" "sasl/sasl.h" "ac_cv_header_sasl_sasl_h" "$ac_includes_default" | 4750 | ac_fn_c_check_header_mongrel "$LINENO" "sasl/sasl.h" "ac_cv_header_sasl_sasl_h" "$ac_includes_default" |
557 | @@ -4784,6 +4796,43 @@ $as_echo "#define HAVE_SASL_CB_GETCONF 1" >>confdefs.h | |||
558 | 4784 | fi | 4796 | fi |
559 | 4785 | 4797 | ||
560 | 4786 | 4798 | ||
561 | 4799 | { $as_echo "$as_me:${as_lineno-$LINENO}: checking for SASL_CB_GETCONFPATH" >&5 | ||
562 | 4800 | $as_echo_n "checking for SASL_CB_GETCONFPATH... " >&6; } | ||
563 | 4801 | if ${ac_cv_c_sasl_cb_getconfpath+:} false; then : | ||
564 | 4802 | $as_echo_n "(cached) " >&6 | ||
565 | 4803 | else | ||
566 | 4804 | cat confdefs.h - <<_ACEOF >conftest.$ac_ext | ||
567 | 4805 | /* end confdefs.h. */ | ||
568 | 4806 | |||
569 | 4807 | #include <sasl/sasl.h> | ||
570 | 4808 | |||
571 | 4809 | int | ||
572 | 4810 | main () | ||
573 | 4811 | { | ||
574 | 4812 | |||
575 | 4813 | unsigned long val = SASL_CB_GETCONFPATH; | ||
576 | 4814 | |||
577 | 4815 | ; | ||
578 | 4816 | return 0; | ||
579 | 4817 | } | ||
580 | 4818 | _ACEOF | ||
581 | 4819 | if ac_fn_c_try_compile "$LINENO"; then : | ||
582 | 4820 | ac_cv_c_sasl_cb_getconfpath=yes | ||
583 | 4821 | else | ||
584 | 4822 | ac_cv_c_sasl_cb_getconfpath=no | ||
585 | 4823 | fi | ||
586 | 4824 | rm -f core conftest.err conftest.$ac_objext conftest.$ac_ext | ||
587 | 4825 | |||
588 | 4826 | fi | ||
589 | 4827 | { $as_echo "$as_me:${as_lineno-$LINENO}: result: $ac_cv_c_sasl_cb_getconfpath" >&5 | ||
590 | 4828 | $as_echo "$ac_cv_c_sasl_cb_getconfpath" >&6; } | ||
591 | 4829 | if test "$ac_cv_c_sasl_cb_getconfpath" = "yes"; then : | ||
592 | 4830 | |||
593 | 4831 | $as_echo "#define HAVE_SASL_CB_GETCONFPATH 1" >>confdefs.h | ||
594 | 4832 | |||
595 | 4833 | fi | ||
596 | 4834 | |||
597 | 4835 | |||
598 | 4787 | $as_echo "#define ENABLE_SASL 1" >>confdefs.h | 4836 | $as_echo "#define ENABLE_SASL 1" >>confdefs.h |
599 | 4788 | 4837 | ||
600 | 4789 | { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sasl_server_init" >&5 | 4838 | { $as_echo "$as_me:${as_lineno-$LINENO}: checking for library containing sasl_server_init" >&5 |
601 | @@ -4929,6 +4978,12 @@ $as_echo "#define EXTSTORE 1" >>confdefs.h | |||
602 | 4929 | 4978 | ||
603 | 4930 | fi | 4979 | fi |
604 | 4931 | 4980 | ||
605 | 4981 | if test "x$enable_arm_crc32" = "xyes"; then | ||
606 | 4982 | |||
607 | 4983 | $as_echo "#define ARM_CRC32 1" >>confdefs.h | ||
608 | 4984 | |||
609 | 4985 | fi | ||
610 | 4986 | |||
611 | 4932 | if test "$build_dtrace" = "yes"; then | 4987 | if test "$build_dtrace" = "yes"; then |
612 | 4933 | BUILD_DTRACE_TRUE= | 4988 | BUILD_DTRACE_TRUE= |
613 | 4934 | BUILD_DTRACE_FALSE='#' | 4989 | BUILD_DTRACE_FALSE='#' |
614 | @@ -4961,6 +5016,14 @@ else | |||
615 | 4961 | ENABLE_EXTSTORE_FALSE= | 5016 | ENABLE_EXTSTORE_FALSE= |
616 | 4962 | fi | 5017 | fi |
617 | 4963 | 5018 | ||
618 | 5019 | if test "$enable_arm_crc32" = "yes"; then | ||
619 | 5020 | ENABLE_ARM_CRC32_TRUE= | ||
620 | 5021 | ENABLE_ARM_CRC32_FALSE='#' | ||
621 | 5022 | else | ||
622 | 5023 | ENABLE_ARM_CRC32_TRUE='#' | ||
623 | 5024 | ENABLE_ARM_CRC32_FALSE= | ||
624 | 5025 | fi | ||
625 | 5026 | |||
626 | 4964 | 5027 | ||
627 | 4965 | 5028 | ||
628 | 4966 | 5029 | ||
629 | @@ -6203,6 +6266,7 @@ else | |||
630 | 6203 | 6266 | ||
631 | 6204 | #include <stdlib.h> | 6267 | #include <stdlib.h> |
632 | 6205 | #include <inttypes.h> | 6268 | #include <inttypes.h> |
633 | 6269 | #pragma GCC optimize ("O0") | ||
634 | 6206 | 6270 | ||
635 | 6207 | int | 6271 | int |
636 | 6208 | main () | 6272 | main () |
637 | @@ -6292,6 +6356,7 @@ have_gcc_64atomics=no | |||
638 | 6292 | $as_echo_n "checking for GCC 64bit atomics... " >&6; } | 6356 | $as_echo_n "checking for GCC 64bit atomics... " >&6; } |
639 | 6293 | cat confdefs.h - <<_ACEOF >conftest.$ac_ext | 6357 | cat confdefs.h - <<_ACEOF >conftest.$ac_ext |
640 | 6294 | /* end confdefs.h. */ | 6358 | /* end confdefs.h. */ |
641 | 6359 | #include <inttypes.h> | ||
642 | 6295 | 6360 | ||
643 | 6296 | int | 6361 | int |
644 | 6297 | main () | 6362 | main () |
645 | @@ -6739,6 +6804,10 @@ if test -z "${ENABLE_EXTSTORE_TRUE}" && test -z "${ENABLE_EXTSTORE_FALSE}"; then | |||
646 | 6739 | as_fn_error $? "conditional \"ENABLE_EXTSTORE\" was never defined. | 6804 | as_fn_error $? "conditional \"ENABLE_EXTSTORE\" was never defined. |
647 | 6740 | Usually this means the macro was only invoked conditionally." "$LINENO" 5 | 6805 | Usually this means the macro was only invoked conditionally." "$LINENO" 5 |
648 | 6741 | fi | 6806 | fi |
649 | 6807 | if test -z "${ENABLE_ARM_CRC32_TRUE}" && test -z "${ENABLE_ARM_CRC32_FALSE}"; then | ||
650 | 6808 | as_fn_error $? "conditional \"ENABLE_ARM_CRC32\" was never defined. | ||
651 | 6809 | Usually this means the macro was only invoked conditionally." "$LINENO" 5 | ||
652 | 6810 | fi | ||
653 | 6742 | if test -z "${BUILD_SOLARIS_PRIVS_TRUE}" && test -z "${BUILD_SOLARIS_PRIVS_FALSE}"; then | 6811 | if test -z "${BUILD_SOLARIS_PRIVS_TRUE}" && test -z "${BUILD_SOLARIS_PRIVS_FALSE}"; then |
654 | 6743 | as_fn_error $? "conditional \"BUILD_SOLARIS_PRIVS\" was never defined. | 6812 | as_fn_error $? "conditional \"BUILD_SOLARIS_PRIVS\" was never defined. |
655 | 6744 | Usually this means the macro was only invoked conditionally." "$LINENO" 5 | 6813 | Usually this means the macro was only invoked conditionally." "$LINENO" 5 |
656 | @@ -7156,7 +7225,7 @@ cat >>$CONFIG_STATUS <<\_ACEOF || ac_write_fail=1 | |||
657 | 7156 | # report actual input values of CONFIG_FILES etc. instead of their | 7225 | # report actual input values of CONFIG_FILES etc. instead of their |
658 | 7157 | # values after options handling. | 7226 | # values after options handling. |
659 | 7158 | ac_log=" | 7227 | ac_log=" |
661 | 7159 | This file was extended by memcached $as_me 1.5.6, which was | 7228 | This file was extended by memcached $as_me 1.5.10, which was |
662 | 7160 | generated by GNU Autoconf 2.69. Invocation command line was | 7229 | generated by GNU Autoconf 2.69. Invocation command line was |
663 | 7161 | 7230 | ||
664 | 7162 | CONFIG_FILES = $CONFIG_FILES | 7231 | CONFIG_FILES = $CONFIG_FILES |
665 | @@ -7222,7 +7291,7 @@ _ACEOF | |||
666 | 7222 | cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 | 7291 | cat >>$CONFIG_STATUS <<_ACEOF || ac_write_fail=1 |
667 | 7223 | ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" | 7292 | ac_cs_config="`$as_echo "$ac_configure_args" | sed 's/^ //; s/[\\""\`\$]/\\\\&/g'`" |
668 | 7224 | ac_cs_version="\\ | 7293 | ac_cs_version="\\ |
670 | 7225 | memcached config.status 1.5.6 | 7294 | memcached config.status 1.5.10 |
671 | 7226 | configured by $0, generated by GNU Autoconf 2.69, | 7295 | configured by $0, generated by GNU Autoconf 2.69, |
672 | 7227 | with options \\"\$ac_cs_config\\" | 7296 | with options \\"\$ac_cs_config\\" |
673 | 7228 | 7297 | ||
674 | diff --git a/configure.ac b/configure.ac | |||
675 | index 9f05cfa..abcb8ae 100644 | |||
676 | --- a/configure.ac | |||
677 | +++ b/configure.ac | |||
678 | @@ -82,11 +82,14 @@ fi | |||
679 | 82 | AM_PROG_CC_C_O | 82 | AM_PROG_CC_C_O |
680 | 83 | AC_PROG_INSTALL | 83 | AC_PROG_INSTALL |
681 | 84 | 84 | ||
682 | 85 | AC_ARG_ENABLE(arm_crc32, | ||
683 | 86 | [AS_HELP_STRING([--enable-arm-crc32], [Enable ARMv8 CRC32 instructions])]) | ||
684 | 87 | |||
685 | 85 | AC_ARG_ENABLE(extstore, | 88 | AC_ARG_ENABLE(extstore, |
686 | 86 | [AS_HELP_STRING([--enable-extstore], [Enable external storage EXPERIMENTAL ])]) | 89 | [AS_HELP_STRING([--enable-extstore], [Enable external storage EXPERIMENTAL ])]) |
687 | 87 | 90 | ||
688 | 88 | AC_ARG_ENABLE(seccomp, | 91 | AC_ARG_ENABLE(seccomp, |
690 | 89 | [AS_HELP_STRING([--enable-seccomp],[Enable seccomp restrictions])]) | 92 | [AS_HELP_STRING([--enable-seccomp],[Enable seccomp restrictions EXPERIMENTAL])]) |
691 | 90 | 93 | ||
692 | 91 | AC_ARG_ENABLE(sasl, | 94 | AC_ARG_ENABLE(sasl, |
693 | 92 | [AS_HELP_STRING([--enable-sasl],[Enable SASL authentication])]) | 95 | [AS_HELP_STRING([--enable-sasl],[Enable SASL authentication])]) |
694 | @@ -122,9 +125,33 @@ unsigned long val = SASL_CB_GETCONF; | |||
695 | 122 | [Set to nonzero if your SASL implementation supports SASL_CB_GETCONF])]) | 125 | [Set to nonzero if your SASL implementation supports SASL_CB_GETCONF])]) |
696 | 123 | ]) | 126 | ]) |
697 | 124 | 127 | ||
698 | 128 | dnl ********************************************************************** | ||
699 | 129 | dnl DETECT_SASL_CB_GETCONFPATH | ||
700 | 130 | dnl | ||
701 | 131 | dnl check if we can use SASL_CB_GETCONFPATH | ||
702 | 132 | dnl ********************************************************************** | ||
703 | 133 | AC_DEFUN([AC_C_DETECT_SASL_CB_GETCONFPATH], | ||
704 | 134 | [ | ||
705 | 135 | AC_CACHE_CHECK([for SASL_CB_GETCONFPATH], | ||
706 | 136 | [ac_cv_c_sasl_cb_getconfpath], | ||
707 | 137 | [AC_TRY_COMPILE( | ||
708 | 138 | [ | ||
709 | 139 | #include <sasl/sasl.h> | ||
710 | 140 | ], [ | ||
711 | 141 | unsigned long val = SASL_CB_GETCONFPATH; | ||
712 | 142 | ], | ||
713 | 143 | [ ac_cv_c_sasl_cb_getconfpath=yes ], | ||
714 | 144 | [ ac_cv_c_sasl_cb_getconfpath=no ]) | ||
715 | 145 | ]) | ||
716 | 146 | AS_IF([test "$ac_cv_c_sasl_cb_getconfpath" = "yes"], | ||
717 | 147 | [AC_DEFINE([HAVE_SASL_CB_GETCONFPATH], 1, | ||
718 | 148 | [Set to nonzero if your SASL implementation supports SASL_CB_GETCONFPATH])]) | ||
719 | 149 | ]) | ||
720 | 150 | |||
721 | 125 | AC_CHECK_HEADERS([sasl/sasl.h]) | 151 | AC_CHECK_HEADERS([sasl/sasl.h]) |
722 | 126 | if test "x$enable_sasl" = "xyes"; then | 152 | if test "x$enable_sasl" = "xyes"; then |
723 | 127 | AC_C_DETECT_SASL_CB_GETCONF | 153 | AC_C_DETECT_SASL_CB_GETCONF |
724 | 154 | AC_C_DETECT_SASL_CB_GETCONFPATH | ||
725 | 128 | AC_DEFINE([ENABLE_SASL],1,[Set to nonzero if you want to include SASL]) | 155 | AC_DEFINE([ENABLE_SASL],1,[Set to nonzero if you want to include SASL]) |
726 | 129 | AC_SEARCH_LIBS([sasl_server_init], [sasl2 sasl], [], | 156 | AC_SEARCH_LIBS([sasl_server_init], [sasl2 sasl], [], |
727 | 130 | [ | 157 | [ |
728 | @@ -160,13 +187,18 @@ if test "x$enable_dtrace" = "xyes"; then | |||
729 | 160 | fi | 187 | fi |
730 | 161 | 188 | ||
731 | 162 | if test "x$enable_extstore" = "xyes"; then | 189 | if test "x$enable_extstore" = "xyes"; then |
733 | 163 | AC_DEFINE([EXTSTORE],1,[Set to nonzero if you want to enable extstorextstore]) | 190 | AC_DEFINE([EXTSTORE],1,[Set to nonzero if you want to enable extstore]) |
734 | 191 | fi | ||
735 | 192 | |||
736 | 193 | if test "x$enable_arm_crc32" = "xyes"; then | ||
737 | 194 | AC_DEFINE([ARM_CRC32],1,[Set to nonzero if you want to enable ARMv8 crc32]) | ||
738 | 164 | fi | 195 | fi |
739 | 165 | 196 | ||
740 | 166 | AM_CONDITIONAL([BUILD_DTRACE],[test "$build_dtrace" = "yes"]) | 197 | AM_CONDITIONAL([BUILD_DTRACE],[test "$build_dtrace" = "yes"]) |
741 | 167 | AM_CONDITIONAL([DTRACE_INSTRUMENT_OBJ],[test "$dtrace_instrument_obj" = "yes"]) | 198 | AM_CONDITIONAL([DTRACE_INSTRUMENT_OBJ],[test "$dtrace_instrument_obj" = "yes"]) |
742 | 168 | AM_CONDITIONAL([ENABLE_SASL],[test "$enable_sasl" = "yes"]) | 199 | AM_CONDITIONAL([ENABLE_SASL],[test "$enable_sasl" = "yes"]) |
743 | 169 | AM_CONDITIONAL([ENABLE_EXTSTORE],[test "$enable_extstore" = "yes"]) | 200 | AM_CONDITIONAL([ENABLE_EXTSTORE],[test "$enable_extstore" = "yes"]) |
744 | 201 | AM_CONDITIONAL([ENABLE_ARM_CRC32],[test "$enable_arm_crc32" = "yes"]) | ||
745 | 170 | 202 | ||
746 | 171 | AC_SUBST(DTRACE) | 203 | AC_SUBST(DTRACE) |
747 | 172 | AC_SUBST(DTRACEFLAGS) | 204 | AC_SUBST(DTRACEFLAGS) |
748 | @@ -490,6 +522,8 @@ AC_CHECK_FUNCS(clock_gettime) | |||
749 | 490 | AC_CHECK_FUNCS([accept4], [AC_DEFINE(HAVE_ACCEPT4, 1, [Define to 1 if support accept4])]) | 522 | AC_CHECK_FUNCS([accept4], [AC_DEFINE(HAVE_ACCEPT4, 1, [Define to 1 if support accept4])]) |
750 | 491 | AC_CHECK_FUNCS([getopt_long], [AC_DEFINE(HAVE_GETOPT_LONG, 1, [Define to 1 if support getopt_long])]) | 523 | AC_CHECK_FUNCS([getopt_long], [AC_DEFINE(HAVE_GETOPT_LONG, 1, [Define to 1 if support getopt_long])]) |
751 | 492 | 524 | ||
752 | 525 | dnl Need to disable opt for alignment check. GCC is too clever and turns this | ||
753 | 526 | dnl into wide stores and no cmp under O2. | ||
754 | 493 | AC_DEFUN([AC_C_ALIGNMENT], | 527 | AC_DEFUN([AC_C_ALIGNMENT], |
755 | 494 | [AC_CACHE_CHECK(for alignment, ac_cv_c_alignment, | 528 | [AC_CACHE_CHECK(for alignment, ac_cv_c_alignment, |
756 | 495 | [ | 529 | [ |
757 | @@ -497,6 +531,7 @@ AC_DEFUN([AC_C_ALIGNMENT], | |||
758 | 497 | [AC_LANG_PROGRAM([ | 531 | [AC_LANG_PROGRAM([ |
759 | 498 | #include <stdlib.h> | 532 | #include <stdlib.h> |
760 | 499 | #include <inttypes.h> | 533 | #include <inttypes.h> |
761 | 534 | #pragma GCC optimize ("O0") | ||
762 | 500 | ], [ | 535 | ], [ |
763 | 501 | char *buf = malloc(32); | 536 | char *buf = malloc(32); |
764 | 502 | 537 | ||
765 | @@ -551,7 +586,8 @@ dnl Check for usage of 64bit atomics | |||
766 | 551 | dnl 32bit systems shouldn't have these. | 586 | dnl 32bit systems shouldn't have these. |
767 | 552 | have_gcc_64atomics=no | 587 | have_gcc_64atomics=no |
768 | 553 | AC_MSG_CHECKING(for GCC 64bit atomics) | 588 | AC_MSG_CHECKING(for GCC 64bit atomics) |
770 | 554 | AC_TRY_LINK([],[ | 589 | AC_TRY_LINK([#include <inttypes.h> |
771 | 590 | ],[ | ||
772 | 555 | uint64_t a; | 591 | uint64_t a; |
773 | 556 | uint64_t b; | 592 | uint64_t b; |
774 | 557 | b = __sync_add_and_fetch(&a, 1); | 593 | b = __sync_add_and_fetch(&a, 1); |
775 | diff --git a/crawler.c b/crawler.c | |||
776 | index 9c81bba..a1a5ead 100644 | |||
777 | --- a/crawler.c | |||
778 | +++ b/crawler.c | |||
779 | @@ -218,7 +218,6 @@ static void crawler_expired_eval(crawler_module_t *cm, item *search, uint32_t hv | |||
780 | 218 | #endif | 218 | #endif |
781 | 219 | do_item_unlink_nolock(search, hv); | 219 | do_item_unlink_nolock(search, hv); |
782 | 220 | do_item_remove(search); | 220 | do_item_remove(search); |
783 | 221 | assert(search->slabs_clsid == 0); | ||
784 | 222 | } else { | 221 | } else { |
785 | 223 | s->seen++; | 222 | s->seen++; |
786 | 224 | refcount_decr(search); | 223 | refcount_decr(search); |
787 | @@ -533,6 +532,14 @@ static int do_lru_crawler_start(uint32_t id, uint32_t remaining) { | |||
788 | 533 | if (remaining == LRU_CRAWLER_CAP_REMAINING) { | 532 | if (remaining == LRU_CRAWLER_CAP_REMAINING) { |
789 | 534 | remaining = do_get_lru_size(sid); | 533 | remaining = do_get_lru_size(sid); |
790 | 535 | } | 534 | } |
791 | 535 | /* Values for remaining: | ||
792 | 536 | * remaining = 0 | ||
793 | 537 | * - scan all elements, until a NULL is reached | ||
794 | 538 | * - if empty, NULL is reached right away | ||
795 | 539 | * remaining = n + 1 | ||
796 | 540 | * - first n elements are parsed (or until a NULL is reached) | ||
797 | 541 | */ | ||
798 | 542 | if (remaining) remaining++; | ||
799 | 536 | crawlers[sid].remaining = remaining; | 543 | crawlers[sid].remaining = remaining; |
800 | 537 | crawlers[sid].slabs_clsid = sid; | 544 | crawlers[sid].slabs_clsid = sid; |
801 | 538 | crawlers[sid].reclaimed = 0; | 545 | crawlers[sid].reclaimed = 0; |
802 | diff --git a/crc32c.c b/crc32c.c | |||
803 | index 13deee2..a4296a7 100644 | |||
804 | --- a/crc32c.c | |||
805 | +++ b/crc32c.c | |||
806 | @@ -40,6 +40,10 @@ | |||
807 | 40 | #include <stdint.h> | 40 | #include <stdint.h> |
808 | 41 | #include <unistd.h> | 41 | #include <unistd.h> |
809 | 42 | #include <pthread.h> | 42 | #include <pthread.h> |
810 | 43 | #include "config.h" | ||
811 | 44 | #if defined(__linux__) && defined(__aarch64__) | ||
812 | 45 | #include <sys/auxv.h> | ||
813 | 46 | #endif | ||
814 | 43 | #include "crc32c.h" | 47 | #include "crc32c.h" |
815 | 44 | 48 | ||
816 | 45 | /* CRC-32C (iSCSI) polynomial in reversed bit order. */ | 49 | /* CRC-32C (iSCSI) polynomial in reversed bit order. */ |
817 | @@ -109,6 +113,60 @@ static uint32_t crc32c_sw(uint32_t crci, const void *buf, size_t len) | |||
818 | 109 | return (uint32_t)crc ^ 0xffffffff; | 113 | return (uint32_t)crc ^ 0xffffffff; |
819 | 110 | } | 114 | } |
820 | 111 | 115 | ||
821 | 116 | /* Hardware CRC support for aarch64 platform */ | ||
822 | 117 | #if defined(__linux__) && defined(__aarch64__) && defined(ARM_CRC32) | ||
823 | 118 | |||
824 | 119 | #define CRC32CX(crc, value) __asm__("crc32cx %w[c], %w[c], %x[v]":[c]"+r"(crc):[v]"r"(+value)) | ||
825 | 120 | #define CRC32CW(crc, value) __asm__("crc32cw %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value)) | ||
826 | 121 | #define CRC32CH(crc, value) __asm__("crc32ch %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value)) | ||
827 | 122 | #define CRC32CB(crc, value) __asm__("crc32cb %w[c], %w[c], %w[v]":[c]"+r"(crc):[v]"r"(+value)) | ||
828 | 123 | |||
829 | 124 | #ifndef HWCAP_CRC32 | ||
830 | 125 | #define HWCAP_CRC32 (1 << 7) | ||
831 | 126 | #endif /* HWCAP for crc32 */ | ||
832 | 127 | |||
833 | 128 | static uint32_t crc32c_hw_aarch64(uint32_t crc, const void* buf, size_t len) | ||
834 | 129 | { | ||
835 | 130 | const uint8_t* p_buf = buf; | ||
836 | 131 | uint64_t crc64bit = crc; | ||
837 | 132 | for (size_t i = 0; i < len / sizeof(uint64_t); i++) { | ||
838 | 133 | CRC32CX(crc64bit, *(uint64_t*) p_buf); | ||
839 | 134 | p_buf += sizeof(uint64_t); | ||
840 | 135 | } | ||
841 | 136 | |||
842 | 137 | uint32_t crc32bit = (uint32_t) crc64bit; | ||
843 | 138 | len &= sizeof(uint64_t) - 1; | ||
844 | 139 | switch (len) { | ||
845 | 140 | case 7: | ||
846 | 141 | CRC32CB(crc32bit, *p_buf++); | ||
847 | 142 | case 6: | ||
848 | 143 | CRC32CH(crc32bit, *(uint16_t*) p_buf); | ||
849 | 144 | p_buf += 2; | ||
850 | 145 | case 4: | ||
851 | 146 | CRC32CW(crc32bit, *(uint32_t*) p_buf); | ||
852 | 147 | break; | ||
853 | 148 | case 3: | ||
854 | 149 | CRC32CB(crc32bit, *p_buf++); | ||
855 | 150 | case 2: | ||
856 | 151 | CRC32CH(crc32bit, *(uint16_t*) p_buf); | ||
857 | 152 | break; | ||
858 | 153 | case 5: | ||
859 | 154 | CRC32CW(crc32bit, *(uint32_t*) p_buf); | ||
860 | 155 | p_buf += 4; | ||
861 | 156 | case 1: | ||
862 | 157 | CRC32CB(crc32bit, *p_buf); | ||
863 | 158 | break; | ||
864 | 159 | case 0: | ||
865 | 160 | break; | ||
866 | 161 | } | ||
867 | 162 | |||
868 | 163 | return crc32bit; | ||
869 | 164 | } | ||
870 | 165 | #endif | ||
871 | 166 | |||
872 | 167 | /* Apply if the platform is intel */ | ||
873 | 168 | #if defined(__X86_64__)||defined(__x86_64__)||defined(__ia64__) | ||
874 | 169 | |||
875 | 112 | /* Multiply a matrix times a vector over the Galois field of two elements, | 170 | /* Multiply a matrix times a vector over the Galois field of two elements, |
876 | 113 | GF(2). Each element is a bit in an unsigned integer. mat must have at | 171 | GF(2). Each element is a bit in an unsigned integer. mat must have at |
877 | 114 | least as many entries as the power of two for most significant one bit in | 172 | least as many entries as the power of two for most significant one bit in |
878 | @@ -329,15 +387,27 @@ static uint32_t crc32c_hw(uint32_t crc, const void *buf, size_t len) | |||
879 | 329 | (have) = (ecx >> 20) & 1; \ | 387 | (have) = (ecx >> 20) & 1; \ |
880 | 330 | } while (0) | 388 | } while (0) |
881 | 331 | 389 | ||
882 | 390 | #endif | ||
883 | 332 | /* Compute a CRC-32C. If the crc32 instruction is available, use the hardware | 391 | /* Compute a CRC-32C. If the crc32 instruction is available, use the hardware |
884 | 333 | version. Otherwise, use the software version. */ | 392 | version. Otherwise, use the software version. */ |
885 | 334 | void crc32c_init(void) { | 393 | void crc32c_init(void) { |
886 | 394 | #if defined(__X86_64__)||defined(__x86_64__)||defined(__ia64__) | ||
887 | 335 | int sse42; | 395 | int sse42; |
888 | 336 | |||
889 | 337 | SSE42(sse42); | 396 | SSE42(sse42); |
890 | 397 | |||
891 | 338 | if (sse42) { | 398 | if (sse42) { |
892 | 339 | crc32c = crc32c_hw; | 399 | crc32c = crc32c_hw; |
894 | 340 | } else { | 400 | } else |
895 | 401 | #endif | ||
896 | 402 | /* Check if CRC instructions supported by aarch64 */ | ||
897 | 403 | #if defined(__linux__) && defined(__aarch64__) && defined(ARM_CRC32) | ||
898 | 404 | unsigned long hwcap = getauxval(AT_HWCAP); | ||
899 | 405 | |||
900 | 406 | if (hwcap & HWCAP_CRC32) { | ||
901 | 407 | crc32c = crc32c_hw_aarch64; | ||
902 | 408 | } else | ||
903 | 409 | #endif | ||
904 | 410 | { | ||
905 | 341 | crc32c = crc32c_sw; | 411 | crc32c = crc32c_sw; |
906 | 342 | } | 412 | } |
907 | 343 | } | 413 | } |
908 | diff --git a/debian/changelog b/debian/changelog | |||
909 | index 43c90a2..1397f89 100644 | |||
910 | --- a/debian/changelog | |||
911 | +++ b/debian/changelog | |||
912 | @@ -1,3 +1,11 @@ | |||
913 | 1 | memcached (1.5.10-0ubuntu1) cosmic; urgency=medium | ||
914 | 2 | |||
915 | 3 | * New upstream release. | ||
916 | 4 | * Includes fixes for various failures on various architectures | ||
917 | 5 | (LP: #1780838). | ||
918 | 6 | |||
919 | 7 | -- Robie Basak <robie.basak@ubuntu.com> Mon, 13 Aug 2018 13:10:37 +0100 | ||
920 | 8 | |||
921 | 1 | memcached (1.5.6-1) unstable; urgency=medium | 9 | memcached (1.5.6-1) unstable; urgency=medium |
922 | 2 | 10 | ||
923 | 3 | * New upstream release | 11 | * New upstream release |
924 | diff --git a/debian/control b/debian/control | |||
925 | index 72e948f..2c63ea0 100644 | |||
926 | --- a/debian/control | |||
927 | +++ b/debian/control | |||
928 | @@ -1,7 +1,8 @@ | |||
929 | 1 | Source: memcached | 1 | Source: memcached |
930 | 2 | Section: web | 2 | Section: web |
931 | 3 | Priority: optional | 3 | Priority: optional |
933 | 4 | Maintainer: Guillaume Delacour <gui@iroqwa.org> | 4 | Maintainer: Ubuntu Developers <ubuntu-devel-discuss@lists.ubuntu.com> |
934 | 5 | XSBC-Original-Maintainer: Guillaume Delacour <gui@iroqwa.org> | ||
935 | 5 | Build-Depends: debhelper (>> 10), libevent-dev, libsasl2-dev, adduser, | 6 | Build-Depends: debhelper (>> 10), libevent-dev, libsasl2-dev, adduser, |
936 | 6 | autotools-dev, dh-autoreconf | 7 | autotools-dev, dh-autoreconf |
937 | 7 | Homepage: http://www.memcached.org/ | 8 | Homepage: http://www.memcached.org/ |
938 | diff --git a/doc/Makefile b/doc/Makefile | |||
939 | index 0337680..49cce71 100644 | |||
940 | --- a/doc/Makefile | |||
941 | +++ b/doc/Makefile | |||
942 | @@ -191,10 +191,10 @@ OBJEXT = o | |||
943 | 191 | PACKAGE = memcached | 191 | PACKAGE = memcached |
944 | 192 | PACKAGE_BUGREPORT = memcached@googlegroups.com | 192 | PACKAGE_BUGREPORT = memcached@googlegroups.com |
945 | 193 | PACKAGE_NAME = memcached | 193 | PACKAGE_NAME = memcached |
947 | 194 | PACKAGE_STRING = memcached 1.5.6 | 194 | PACKAGE_STRING = memcached 1.5.10 |
948 | 195 | PACKAGE_TARNAME = memcached | 195 | PACKAGE_TARNAME = memcached |
949 | 196 | PACKAGE_URL = | 196 | PACKAGE_URL = |
951 | 197 | PACKAGE_VERSION = 1.5.6 | 197 | PACKAGE_VERSION = 1.5.10 |
952 | 198 | PATH_SEPARATOR = : | 198 | PATH_SEPARATOR = : |
953 | 199 | PROFILER = /usr/bin/gcov | 199 | PROFILER = /usr/bin/gcov |
954 | 200 | PROFILER_FLAGS = -fprofile-arcs -ftest-coverage | 200 | PROFILER_FLAGS = -fprofile-arcs -ftest-coverage |
955 | @@ -202,7 +202,7 @@ PROFILER_LDFLAGS = -lgcov | |||
956 | 202 | SET_MAKE = | 202 | SET_MAKE = |
957 | 203 | SHELL = /bin/bash | 203 | SHELL = /bin/bash |
958 | 204 | STRIP = | 204 | STRIP = |
960 | 205 | VERSION = 1.5.6 | 205 | VERSION = 1.5.10 |
961 | 206 | XML2RFC = /usr/bin/xml2rfc | 206 | XML2RFC = /usr/bin/xml2rfc |
962 | 207 | XSLTPROC = no | 207 | XSLTPROC = no |
963 | 208 | abs_builddir = /home/dormando/d/p/danga/git/memcached/doc | 208 | abs_builddir = /home/dormando/d/p/danga/git/memcached/doc |
964 | diff --git a/doc/memcached.1 b/doc/memcached.1 | |||
965 | index caa53b7..04aca5f 100644 | |||
966 | --- a/doc/memcached.1 | |||
967 | +++ b/doc/memcached.1 | |||
968 | @@ -62,10 +62,10 @@ caches, so consult the README and memcached homepage for configuration | |||
969 | 62 | suggestions. | 62 | suggestions. |
970 | 63 | .TP | 63 | .TP |
971 | 64 | .B \-p, --port=<num> | 64 | .B \-p, --port=<num> |
973 | 65 | Listen on TCP port <num>, the default is port 11211. | 65 | Listen on TCP port <num>, the default is port 11211. 0 means off. |
974 | 66 | .TP | 66 | .TP |
975 | 67 | .B \-U, --udp-port=<num> | 67 | .B \-U, --udp-port=<num> |
977 | 68 | Listen on UDP port <num>, the default is port 11211, 0 is off. | 68 | Listen on UDP port <num>, the default is port 0, which is off. |
978 | 69 | .TP | 69 | .TP |
979 | 70 | .B \-M, --disable-evictions | 70 | .B \-M, --disable-evictions |
980 | 71 | Disable automatic removal of items from the cache when out of memory. | 71 | Disable automatic removal of items from the cache when out of memory. |
981 | diff --git a/doc/protocol.txt b/doc/protocol.txt | |||
982 | index 9c8e8bc..0984594 100644 | |||
983 | --- a/doc/protocol.txt | |||
984 | +++ b/doc/protocol.txt | |||
985 | @@ -849,6 +849,8 @@ other stats command. | |||
986 | 849 | | | bool | If yes, stores numbers from VALUE response | | 849 | | | bool | If yes, stores numbers from VALUE response | |
987 | 850 | | | | inside an item, using up to 24 bytes. | | 850 | | | | inside an item, using up to 24 bytes. | |
988 | 851 | | | | Small slowdown for ASCII get, faster sets. | | 851 | | | | Small slowdown for ASCII get, faster sets. | |
989 | 852 | | drop_privileges | bool | If yes, and available, drop unused syscalls | | ||
990 | 853 | | | | (see seccomp on Linux, pledge on OpenBSD) | | ||
991 | 852 | |-------------------+----------+----------------------------------------------| | 854 | |-------------------+----------+----------------------------------------------| |
992 | 853 | 855 | ||
993 | 854 | 856 | ||
994 | diff --git a/extstore.c b/extstore.c | |||
995 | index 02558c0..726435c 100644 | |||
996 | --- a/extstore.c | |||
997 | +++ b/extstore.c | |||
998 | @@ -62,6 +62,7 @@ typedef struct _store_page { | |||
999 | 62 | unsigned int allocated; | 62 | unsigned int allocated; |
1000 | 63 | unsigned int written; /* item offsets can be past written if wbuf not flushed */ | 63 | unsigned int written; /* item offsets can be past written if wbuf not flushed */ |
1001 | 64 | unsigned int bucket; /* which bucket the page is linked into */ | 64 | unsigned int bucket; /* which bucket the page is linked into */ |
1002 | 65 | unsigned int free_bucket; /* which bucket this page returns to when freed */ | ||
1003 | 65 | int fd; | 66 | int fd; |
1004 | 66 | unsigned short id; | 67 | unsigned short id; |
1005 | 67 | bool active; /* actively being written to */ | 68 | bool active; /* actively being written to */ |
1006 | @@ -95,6 +96,7 @@ struct store_engine { | |||
1007 | 95 | store_maint_thread *maint_thread; | 96 | store_maint_thread *maint_thread; |
1008 | 96 | store_page *page_freelist; | 97 | store_page *page_freelist; |
1009 | 97 | store_page **page_buckets; /* stack of pages currently allocated to each bucket */ | 98 | store_page **page_buckets; /* stack of pages currently allocated to each bucket */ |
1010 | 99 | store_page **free_page_buckets; /* stack of use-case isolated free pages */ | ||
1011 | 98 | size_t page_size; | 100 | size_t page_size; |
1012 | 99 | unsigned int version; /* global version counter */ | 101 | unsigned int version; /* global version counter */ |
1013 | 100 | unsigned int last_io_thread; /* round robin the IO threads */ | 102 | unsigned int last_io_thread; /* round robin the IO threads */ |
1014 | @@ -102,6 +104,7 @@ struct store_engine { | |||
1015 | 102 | unsigned int page_count; | 104 | unsigned int page_count; |
1016 | 103 | unsigned int page_free; /* unallocated pages */ | 105 | unsigned int page_free; /* unallocated pages */ |
1017 | 104 | unsigned int page_bucketcount; /* count of potential page buckets */ | 106 | unsigned int page_bucketcount; /* count of potential page buckets */ |
1018 | 107 | unsigned int free_page_bucketcount; /* count of free page buckets */ | ||
1019 | 105 | unsigned int io_depth; /* FIXME: Might cache into thr struct */ | 108 | unsigned int io_depth; /* FIXME: Might cache into thr struct */ |
1020 | 106 | pthread_mutex_t stats_mutex; | 109 | pthread_mutex_t stats_mutex; |
1021 | 107 | struct extstore_stats stats; | 110 | struct extstore_stats stats; |
1022 | @@ -192,11 +195,11 @@ const char *extstore_err(enum extstore_res res) { | |||
1023 | 192 | return rv; | 195 | return rv; |
1024 | 193 | } | 196 | } |
1025 | 194 | 197 | ||
1027 | 195 | void *extstore_init(char *fn, struct extstore_conf *cf, | 198 | // TODO: #define's for DEFAULT_BUCKET, FREE_VERSION, etc |
1028 | 199 | void *extstore_init(struct extstore_conf_file *fh, struct extstore_conf *cf, | ||
1029 | 196 | enum extstore_res *res) { | 200 | enum extstore_res *res) { |
1030 | 197 | int i; | 201 | int i; |
1033 | 198 | int fd; | 202 | struct extstore_conf_file *f = NULL; |
1032 | 199 | uint64_t offset = 0; | ||
1034 | 200 | pthread_t thread; | 203 | pthread_t thread; |
1035 | 201 | 204 | ||
1036 | 202 | if (cf->page_size % cf->wbuf_size != 0) { | 205 | if (cf->page_size % cf->wbuf_size != 0) { |
1037 | @@ -227,43 +230,72 @@ void *extstore_init(char *fn, struct extstore_conf *cf, | |||
1038 | 227 | } | 230 | } |
1039 | 228 | 231 | ||
1040 | 229 | e->page_size = cf->page_size; | 232 | e->page_size = cf->page_size; |
1044 | 230 | fd = open(fn, O_RDWR | O_CREAT | O_TRUNC, 0644); | 233 | for (f = fh; f != NULL; f = f->next) { |
1045 | 231 | if (fd < 0) { | 234 | f->fd = open(f->file, O_RDWR | O_CREAT | O_TRUNC, 0644); |
1046 | 232 | *res = EXTSTORE_INIT_OPEN_FAIL; | 235 | if (f->fd < 0) { |
1047 | 236 | *res = EXTSTORE_INIT_OPEN_FAIL; | ||
1048 | 233 | #ifdef EXTSTORE_DEBUG | 237 | #ifdef EXTSTORE_DEBUG |
1050 | 234 | perror("open"); | 238 | perror("open"); |
1051 | 235 | #endif | 239 | #endif |
1054 | 236 | free(e); | 240 | free(e); |
1055 | 237 | return NULL; | 241 | return NULL; |
1056 | 242 | } | ||
1057 | 243 | e->page_count += f->page_count; | ||
1058 | 244 | f->offset = 0; | ||
1059 | 238 | } | 245 | } |
1060 | 239 | 246 | ||
1062 | 240 | e->pages = calloc(cf->page_count, sizeof(store_page)); | 247 | e->pages = calloc(e->page_count, sizeof(store_page)); |
1063 | 241 | if (e->pages == NULL) { | 248 | if (e->pages == NULL) { |
1064 | 242 | *res = EXTSTORE_INIT_OOM; | 249 | *res = EXTSTORE_INIT_OOM; |
1066 | 243 | close(fd); | 250 | // FIXME: loop-close. make error label |
1067 | 244 | free(e); | 251 | free(e); |
1068 | 245 | return NULL; | 252 | return NULL; |
1069 | 246 | } | 253 | } |
1070 | 247 | 254 | ||
1072 | 248 | for (i = 0; i < cf->page_count; i++) { | 255 | // interleave the pages between devices |
1073 | 256 | f = NULL; // start at the first device. | ||
1074 | 257 | for (i = 0; i < e->page_count; i++) { | ||
1075 | 258 | // find next device with available pages | ||
1076 | 259 | while (1) { | ||
1077 | 260 | // restart the loop | ||
1078 | 261 | if (f == NULL || f->next == NULL) { | ||
1079 | 262 | f = fh; | ||
1080 | 263 | } else { | ||
1081 | 264 | f = f->next; | ||
1082 | 265 | } | ||
1083 | 266 | if (f->page_count) { | ||
1084 | 267 | f->page_count--; | ||
1085 | 268 | break; | ||
1086 | 269 | } | ||
1087 | 270 | } | ||
1088 | 249 | pthread_mutex_init(&e->pages[i].mutex, NULL); | 271 | pthread_mutex_init(&e->pages[i].mutex, NULL); |
1089 | 250 | e->pages[i].id = i; | 272 | e->pages[i].id = i; |
1092 | 251 | e->pages[i].fd = fd; | 273 | e->pages[i].fd = f->fd; |
1093 | 252 | e->pages[i].offset = offset; | 274 | e->pages[i].free_bucket = f->free_bucket; |
1094 | 275 | e->pages[i].offset = f->offset; | ||
1095 | 253 | e->pages[i].free = true; | 276 | e->pages[i].free = true; |
1097 | 254 | offset += e->page_size; | 277 | f->offset += e->page_size; |
1098 | 255 | } | 278 | } |
1099 | 256 | 279 | ||
1103 | 257 | for (i = cf->page_count-1; i > 0; i--) { | 280 | // free page buckets allows the app to organize devices by use case |
1104 | 258 | e->pages[i].next = e->page_freelist; | 281 | e->free_page_buckets = calloc(cf->page_buckets, sizeof(store_page *)); |
1105 | 259 | e->page_freelist = &e->pages[i]; | 282 | e->page_bucketcount = cf->page_buckets; |
1106 | 283 | |||
1107 | 284 | for (i = e->page_count-1; i > 0; i--) { | ||
1108 | 260 | e->page_free++; | 285 | e->page_free++; |
1109 | 286 | if (e->pages[i].free_bucket == 0) { | ||
1110 | 287 | e->pages[i].next = e->page_freelist; | ||
1111 | 288 | e->page_freelist = &e->pages[i]; | ||
1112 | 289 | } else { | ||
1113 | 290 | int fb = e->pages[i].free_bucket; | ||
1114 | 291 | e->pages[i].next = e->free_page_buckets[fb]; | ||
1115 | 292 | e->free_page_buckets[fb] = &e->pages[i]; | ||
1116 | 293 | } | ||
1117 | 261 | } | 294 | } |
1118 | 262 | 295 | ||
1119 | 263 | // 0 is magic "page is freed" version | 296 | // 0 is magic "page is freed" version |
1120 | 264 | e->version = 1; | 297 | e->version = 1; |
1121 | 265 | 298 | ||
1122 | 266 | e->page_count = cf->page_count; | ||
1123 | 267 | // scratch data for stats. TODO: malloc failure handle | 299 | // scratch data for stats. TODO: malloc failure handle |
1124 | 268 | e->stats.page_data = | 300 | e->stats.page_data = |
1125 | 269 | calloc(e->page_count, sizeof(struct extstore_page_data)); | 301 | calloc(e->page_count, sizeof(struct extstore_page_data)); |
1126 | @@ -305,8 +337,12 @@ void *extstore_init(char *fn, struct extstore_conf *cf, | |||
1127 | 305 | e->maint_thread = calloc(1, sizeof(store_maint_thread)); | 337 | e->maint_thread = calloc(1, sizeof(store_maint_thread)); |
1128 | 306 | e->maint_thread->e = e; | 338 | e->maint_thread->e = e; |
1129 | 307 | // FIXME: error handling | 339 | // FIXME: error handling |
1130 | 340 | pthread_mutex_init(&e->maint_thread->mutex, NULL); | ||
1131 | 341 | pthread_cond_init(&e->maint_thread->cond, NULL); | ||
1132 | 308 | pthread_create(&thread, NULL, extstore_maint_thread, e->maint_thread); | 342 | pthread_create(&thread, NULL, extstore_maint_thread, e->maint_thread); |
1133 | 309 | 343 | ||
1134 | 344 | extstore_run_maint(e); | ||
1135 | 345 | |||
1136 | 310 | return (void *)e; | 346 | return (void *)e; |
1137 | 311 | } | 347 | } |
1138 | 312 | 348 | ||
1139 | @@ -316,13 +352,25 @@ void extstore_run_maint(void *ptr) { | |||
1140 | 316 | } | 352 | } |
1141 | 317 | 353 | ||
1142 | 318 | // call with *e locked | 354 | // call with *e locked |
1144 | 319 | static store_page *_allocate_page(store_engine *e, unsigned int bucket) { | 355 | static store_page *_allocate_page(store_engine *e, unsigned int bucket, |
1145 | 356 | unsigned int free_bucket) { | ||
1146 | 320 | assert(!e->page_buckets[bucket] || e->page_buckets[bucket]->allocated == e->page_size); | 357 | assert(!e->page_buckets[bucket] || e->page_buckets[bucket]->allocated == e->page_size); |
1151 | 321 | store_page *tmp = e->page_freelist; | 358 | store_page *tmp = NULL; |
1152 | 322 | E_DEBUG("EXTSTORE: allocating new page\n"); | 359 | // if a specific free bucket was requested, check there first |
1153 | 323 | if (e->page_free > 0) { | 360 | if (free_bucket != 0 && e->free_page_buckets[free_bucket] != NULL) { |
1154 | 324 | assert(e->page_freelist != NULL); | 361 | assert(e->page_free > 0); |
1155 | 362 | tmp = e->free_page_buckets[free_bucket]; | ||
1156 | 363 | e->free_page_buckets[free_bucket] = tmp->next; | ||
1157 | 364 | } | ||
1158 | 365 | // failing that, try the global list. | ||
1159 | 366 | if (tmp == NULL && e->page_freelist != NULL) { | ||
1160 | 367 | tmp = e->page_freelist; | ||
1161 | 325 | e->page_freelist = tmp->next; | 368 | e->page_freelist = tmp->next; |
1162 | 369 | } | ||
1163 | 370 | E_DEBUG("EXTSTORE: allocating new page\n"); | ||
1164 | 371 | // page_freelist can be empty if the only free pages are specialized and | ||
1165 | 372 | // we didn't just request one. | ||
1166 | 373 | if (e->page_free > 0 && tmp != NULL) { | ||
1167 | 326 | tmp->next = e->page_buckets[bucket]; | 374 | tmp->next = e->page_buckets[bucket]; |
1168 | 327 | e->page_buckets[bucket] = tmp; | 375 | e->page_buckets[bucket] = tmp; |
1169 | 328 | tmp->active = true; | 376 | tmp->active = true; |
1170 | @@ -432,7 +480,8 @@ static void _submit_wbuf(store_engine *e, store_page *p) { | |||
1171 | 432 | * new page. best if used from a background thread that can harmlessly retry. | 480 | * new page. best if used from a background thread that can harmlessly retry. |
1172 | 433 | */ | 481 | */ |
1173 | 434 | 482 | ||
1175 | 435 | int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) { | 483 | int extstore_write_request(void *ptr, unsigned int bucket, |
1176 | 484 | unsigned int free_bucket, obj_io *io) { | ||
1177 | 436 | store_engine *e = (store_engine *)ptr; | 485 | store_engine *e = (store_engine *)ptr; |
1178 | 437 | store_page *p; | 486 | store_page *p; |
1179 | 438 | int ret = -1; | 487 | int ret = -1; |
1180 | @@ -442,7 +491,7 @@ int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) { | |||
1181 | 442 | pthread_mutex_lock(&e->mutex); | 491 | pthread_mutex_lock(&e->mutex); |
1182 | 443 | p = e->page_buckets[bucket]; | 492 | p = e->page_buckets[bucket]; |
1183 | 444 | if (!p) { | 493 | if (!p) { |
1185 | 445 | p = _allocate_page(e, bucket); | 494 | p = _allocate_page(e, bucket, free_bucket); |
1186 | 446 | } | 495 | } |
1187 | 447 | pthread_mutex_unlock(&e->mutex); | 496 | pthread_mutex_unlock(&e->mutex); |
1188 | 448 | if (!p) | 497 | if (!p) |
1189 | @@ -456,7 +505,7 @@ int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io) { | |||
1190 | 456 | ((!p->wbuf || p->wbuf->full) && p->allocated >= e->page_size)) { | 505 | ((!p->wbuf || p->wbuf->full) && p->allocated >= e->page_size)) { |
1191 | 457 | pthread_mutex_unlock(&p->mutex); | 506 | pthread_mutex_unlock(&p->mutex); |
1192 | 458 | pthread_mutex_lock(&e->mutex); | 507 | pthread_mutex_lock(&e->mutex); |
1194 | 459 | _allocate_page(e, bucket); | 508 | _allocate_page(e, bucket, free_bucket); |
1195 | 460 | pthread_mutex_unlock(&e->mutex); | 509 | pthread_mutex_unlock(&e->mutex); |
1196 | 461 | return ret; | 510 | return ret; |
1197 | 462 | } | 511 | } |
1198 | @@ -510,7 +559,7 @@ void extstore_write(void *ptr, obj_io *io) { | |||
1199 | 510 | /* engine submit function; takes engine, item_io stack. | 559 | /* engine submit function; takes engine, item_io stack. |
1200 | 511 | * lock io_thread context and add stack? | 560 | * lock io_thread context and add stack? |
1201 | 512 | * signal io thread to wake. | 561 | * signal io thread to wake. |
1203 | 513 | * return sucess. | 562 | * return success. |
1204 | 514 | */ | 563 | */ |
1205 | 515 | int extstore_submit(void *ptr, obj_io *io) { | 564 | int extstore_submit(void *ptr, obj_io *io) { |
1206 | 516 | store_engine *e = (store_engine *)ptr; | 565 | store_engine *e = (store_engine *)ptr; |
1207 | @@ -603,7 +652,7 @@ void extstore_close_page(void *ptr, unsigned int page_id, uint64_t page_version) | |||
1208 | 603 | 652 | ||
1209 | 604 | /* Finds an attached wbuf that can satisfy the read. | 653 | /* Finds an attached wbuf that can satisfy the read. |
1210 | 605 | * Since wbufs can potentially be flushed to disk out of order, they are only | 654 | * Since wbufs can potentially be flushed to disk out of order, they are only |
1212 | 606 | * removed as the head of the list successfuly flushes to disk. | 655 | * removed as the head of the list successfully flushes to disk. |
1213 | 607 | */ | 656 | */ |
1214 | 608 | // call with *p locked | 657 | // call with *p locked |
1215 | 609 | // FIXME: protect from reading past wbuf | 658 | // FIXME: protect from reading past wbuf |
1216 | @@ -762,8 +811,14 @@ static void _free_page(store_engine *e, store_page *p) { | |||
1217 | 762 | p->closed = false; | 811 | p->closed = false; |
1218 | 763 | p->free = true; | 812 | p->free = true; |
1219 | 764 | // add to page stack | 813 | // add to page stack |
1222 | 765 | p->next = e->page_freelist; | 814 | // TODO: free_page_buckets first class and remove redundancy? |
1223 | 766 | e->page_freelist = p; | 815 | if (p->free_bucket != 0) { |
1224 | 816 | p->next = e->free_page_buckets[p->free_bucket]; | ||
1225 | 817 | e->free_page_buckets[p->free_bucket] = p; | ||
1226 | 818 | } else { | ||
1227 | 819 | p->next = e->page_freelist; | ||
1228 | 820 | e->page_freelist = p; | ||
1229 | 821 | } | ||
1230 | 767 | e->page_free++; | 822 | e->page_free++; |
1231 | 768 | pthread_mutex_unlock(&e->mutex); | 823 | pthread_mutex_unlock(&e->mutex); |
1232 | 769 | } | 824 | } |
1233 | @@ -795,7 +850,9 @@ static void *extstore_maint_thread(void *arg) { | |||
1234 | 795 | 850 | ||
1235 | 796 | pthread_cond_wait(&me->cond, &me->mutex); | 851 | pthread_cond_wait(&me->cond, &me->mutex); |
1236 | 797 | pthread_mutex_lock(&e->mutex); | 852 | pthread_mutex_lock(&e->mutex); |
1238 | 798 | if (e->page_free == 0) { | 853 | // default freelist requires at least one page free. |
1239 | 854 | // specialized freelists fall back to default once full. | ||
1240 | 855 | if (e->page_free == 0 || e->page_freelist == NULL) { | ||
1241 | 799 | do_evict = true; | 856 | do_evict = true; |
1242 | 800 | } | 857 | } |
1243 | 801 | pthread_mutex_unlock(&e->mutex); | 858 | pthread_mutex_unlock(&e->mutex); |
1244 | @@ -804,6 +861,7 @@ static void *extstore_maint_thread(void *arg) { | |||
1245 | 804 | for (i = 0; i < e->page_count; i++) { | 861 | for (i = 0; i < e->page_count; i++) { |
1246 | 805 | store_page *p = &e->pages[i]; | 862 | store_page *p = &e->pages[i]; |
1247 | 806 | pthread_mutex_lock(&p->mutex); | 863 | pthread_mutex_lock(&p->mutex); |
1248 | 864 | pd[p->id].free_bucket = p->free_bucket; | ||
1249 | 807 | if (p->active || p->free) { | 865 | if (p->active || p->free) { |
1250 | 808 | pthread_mutex_unlock(&p->mutex); | 866 | pthread_mutex_unlock(&p->mutex); |
1251 | 809 | continue; | 867 | continue; |
1252 | @@ -812,7 +870,13 @@ static void *extstore_maint_thread(void *arg) { | |||
1253 | 812 | pd[p->id].version = p->version; | 870 | pd[p->id].version = p->version; |
1254 | 813 | pd[p->id].bytes_used = p->bytes_used; | 871 | pd[p->id].bytes_used = p->bytes_used; |
1255 | 814 | pd[p->id].bucket = p->bucket; | 872 | pd[p->id].bucket = p->bucket; |
1257 | 815 | if (p->version < low_version) { | 873 | // low_version/low_page are only used in the eviction |
1258 | 874 | // scenario. when we evict, it's only to fill the default page | ||
1259 | 875 | // bucket again. | ||
1260 | 876 | // TODO: experiment with allowing evicting up to a single page | ||
1261 | 877 | // for any specific free bucket. this is *probably* required | ||
1262 | 878 | // since it could cause a load bias on default-only devices? | ||
1263 | 879 | if (p->free_bucket == 0 && p->version < low_version) { | ||
1264 | 816 | low_version = p->version; | 880 | low_version = p->version; |
1265 | 817 | low_page = i; | 881 | low_page = i; |
1266 | 818 | } | 882 | } |
1267 | diff --git a/extstore.h b/extstore.h | |||
1268 | index a466562..6814415 100644 | |||
1269 | --- a/extstore.h | |||
1270 | +++ b/extstore.h | |||
1271 | @@ -8,6 +8,7 @@ struct extstore_page_data { | |||
1272 | 8 | uint64_t version; | 8 | uint64_t version; |
1273 | 9 | uint64_t bytes_used; | 9 | uint64_t bytes_used; |
1274 | 10 | unsigned int bucket; | 10 | unsigned int bucket; |
1275 | 11 | unsigned int free_bucket; | ||
1276 | 11 | }; | 12 | }; |
1277 | 12 | 13 | ||
1278 | 13 | /* Pages can have objects deleted from them at any time. This creates holes | 14 | /* Pages can have objects deleted from them at any time. This creates holes |
1279 | @@ -43,12 +44,23 @@ struct extstore_conf { | |||
1280 | 43 | unsigned int page_size; // ideally 64-256M in size | 44 | unsigned int page_size; // ideally 64-256M in size |
1281 | 44 | unsigned int page_count; | 45 | unsigned int page_count; |
1282 | 45 | unsigned int page_buckets; // number of different writeable pages | 46 | unsigned int page_buckets; // number of different writeable pages |
1283 | 47 | unsigned int free_page_buckets; // buckets of dedicated pages (see code) | ||
1284 | 46 | unsigned int wbuf_size; // must divide cleanly into page_size | 48 | unsigned int wbuf_size; // must divide cleanly into page_size |
1285 | 47 | unsigned int wbuf_count; // this might get locked to "2 per active page" | 49 | unsigned int wbuf_count; // this might get locked to "2 per active page" |
1286 | 48 | unsigned int io_threadcount; | 50 | unsigned int io_threadcount; |
1287 | 49 | unsigned int io_depth; // with normal I/O, hits locks less. req'd for AIO | 51 | unsigned int io_depth; // with normal I/O, hits locks less. req'd for AIO |
1288 | 50 | }; | 52 | }; |
1289 | 51 | 53 | ||
1290 | 54 | struct extstore_conf_file { | ||
1291 | 55 | unsigned int page_count; | ||
1292 | 56 | char *file; | ||
1293 | 57 | int fd; // internal usage | ||
1294 | 58 | uint64_t offset; // internal usage | ||
1295 | 59 | unsigned int bucket; // free page bucket | ||
1296 | 60 | unsigned int free_bucket; // specialized free bucket | ||
1297 | 61 | struct extstore_conf_file *next; | ||
1298 | 62 | }; | ||
1299 | 63 | |||
1300 | 52 | enum obj_io_mode { | 64 | enum obj_io_mode { |
1301 | 53 | OBJ_IO_READ = 0, | 65 | OBJ_IO_READ = 0, |
1302 | 54 | OBJ_IO_WRITE, | 66 | OBJ_IO_WRITE, |
1303 | @@ -87,8 +99,8 @@ enum extstore_res { | |||
1304 | 87 | }; | 99 | }; |
1305 | 88 | 100 | ||
1306 | 89 | const char *extstore_err(enum extstore_res res); | 101 | const char *extstore_err(enum extstore_res res); |
1309 | 90 | void *extstore_init(char *fn, struct extstore_conf *cf, enum extstore_res *res); | 102 | void *extstore_init(struct extstore_conf_file *fh, struct extstore_conf *cf, enum extstore_res *res); |
1310 | 91 | int extstore_write_request(void *ptr, unsigned int bucket, obj_io *io); | 103 | int extstore_write_request(void *ptr, unsigned int bucket, unsigned int free_bucket, obj_io *io); |
1311 | 92 | void extstore_write(void *ptr, obj_io *io); | 104 | void extstore_write(void *ptr, obj_io *io); |
1312 | 93 | int extstore_submit(void *ptr, obj_io *io); | 105 | int extstore_submit(void *ptr, obj_io *io); |
1313 | 94 | /* count are the number of objects being removed, bytes are the original | 106 | /* count are the number of objects being removed, bytes are the original |
1314 | diff --git a/items.c b/items.c | |||
1315 | index 0aefaf0..d4ce5a1 100644 | |||
1316 | --- a/items.c | |||
1317 | +++ b/items.c | |||
1318 | @@ -285,6 +285,13 @@ item *do_item_alloc(char *key, const size_t nkey, const unsigned int flags, | |||
1319 | 285 | if (settings.use_cas) { | 285 | if (settings.use_cas) { |
1320 | 286 | htotal += sizeof(uint64_t); | 286 | htotal += sizeof(uint64_t); |
1321 | 287 | } | 287 | } |
1322 | 288 | #ifdef NEED_ALIGN | ||
1323 | 289 | // header chunk needs to be padded on some systems | ||
1324 | 290 | int remain = htotal % 8; | ||
1325 | 291 | if (remain != 0) { | ||
1326 | 292 | htotal += 8 - remain; | ||
1327 | 293 | } | ||
1328 | 294 | #endif | ||
1329 | 288 | hdr_id = slabs_clsid(htotal); | 295 | hdr_id = slabs_clsid(htotal); |
1330 | 289 | it = do_item_alloc_pull(htotal, hdr_id); | 296 | it = do_item_alloc_pull(htotal, hdr_id); |
1331 | 290 | /* setting ITEM_CHUNKED is fine here because we aren't LINKED yet. */ | 297 | /* setting ITEM_CHUNKED is fine here because we aren't LINKED yet. */ |
1332 | @@ -336,7 +343,7 @@ item *do_item_alloc(char *key, const size_t nkey, const unsigned int flags, | |||
1333 | 336 | 343 | ||
1334 | 337 | /* Initialize internal chunk. */ | 344 | /* Initialize internal chunk. */ |
1335 | 338 | if (it->it_flags & ITEM_CHUNKED) { | 345 | if (it->it_flags & ITEM_CHUNKED) { |
1337 | 339 | item_chunk *chunk = (item_chunk *) ITEM_data(it); | 346 | item_chunk *chunk = (item_chunk *) ITEM_schunk(it); |
1338 | 340 | 347 | ||
1339 | 341 | chunk->next = 0; | 348 | chunk->next = 0; |
1340 | 342 | chunk->prev = 0; | 349 | chunk->prev = 0; |
1341 | @@ -1538,7 +1545,6 @@ static void *lru_maintainer_thread(void *arg) { | |||
1342 | 1538 | void *storage = arg; | 1545 | void *storage = arg; |
1343 | 1539 | if (storage != NULL) | 1546 | if (storage != NULL) |
1344 | 1540 | sam = &slab_automove_extstore; | 1547 | sam = &slab_automove_extstore; |
1345 | 1541 | int x; | ||
1346 | 1542 | #endif | 1548 | #endif |
1347 | 1543 | int i; | 1549 | int i; |
1348 | 1544 | useconds_t to_sleep = MIN_LRU_MAINTAINER_SLEEP; | 1550 | useconds_t to_sleep = MIN_LRU_MAINTAINER_SLEEP; |
1349 | @@ -1592,20 +1598,6 @@ static void *lru_maintainer_thread(void *arg) { | |||
1350 | 1592 | } | 1598 | } |
1351 | 1593 | 1599 | ||
1352 | 1594 | int did_moves = lru_maintainer_juggle(i); | 1600 | int did_moves = lru_maintainer_juggle(i); |
1353 | 1595 | #ifdef EXTSTORE | ||
1354 | 1596 | // Deeper loop to speed up pushing to storage. | ||
1355 | 1597 | if (storage) { | ||
1356 | 1598 | for (x = 0; x < 500; x++) { | ||
1357 | 1599 | int found; | ||
1358 | 1600 | found = lru_maintainer_store(storage, i); | ||
1359 | 1601 | if (found) { | ||
1360 | 1602 | did_moves += found; | ||
1361 | 1603 | } else { | ||
1362 | 1604 | break; | ||
1363 | 1605 | } | ||
1364 | 1606 | } | ||
1365 | 1607 | } | ||
1366 | 1608 | #endif | ||
1367 | 1609 | if (did_moves == 0) { | 1601 | if (did_moves == 0) { |
1368 | 1610 | if (backoff_juggles[i] != 0) { | 1602 | if (backoff_juggles[i] != 0) { |
1369 | 1611 | backoff_juggles[i] += backoff_juggles[i] / 8; | 1603 | backoff_juggles[i] += backoff_juggles[i] / 8; |
1370 | diff --git a/linux_priv.c b/linux_priv.c | |||
1371 | index 04155dd..cc9aef3 100644 | |||
1372 | --- a/linux_priv.c | |||
1373 | +++ b/linux_priv.c | |||
1374 | @@ -2,11 +2,11 @@ | |||
1375 | 2 | #include <seccomp.h> | 2 | #include <seccomp.h> |
1376 | 3 | #include <errno.h> | 3 | #include <errno.h> |
1377 | 4 | #include <stdlib.h> | 4 | #include <stdlib.h> |
1378 | 5 | #include <sys/ioctl.h> | ||
1379 | 5 | #include "memcached.h" | 6 | #include "memcached.h" |
1380 | 6 | 7 | ||
1384 | 7 | // In the future when the system is more tested this could be switched | 8 | // If anything crosses the policy, kill the process. |
1385 | 8 | // to SCMP_ACT_KILL instead. | 9 | #define DENY_ACTION SCMP_ACT_KILL |
1383 | 9 | #define DENY_ACTION SCMP_ACT_ERRNO(EACCES) | ||
1386 | 10 | 10 | ||
1387 | 11 | void drop_privileges(void) { | 11 | void drop_privileges(void) { |
1388 | 12 | scmp_filter_ctx ctx = seccomp_init(DENY_ACTION); | 12 | scmp_filter_ctx ctx = seccomp_init(DENY_ACTION); |
1389 | @@ -16,24 +16,41 @@ void drop_privileges(void) { | |||
1390 | 16 | 16 | ||
1391 | 17 | int rc = 0; | 17 | int rc = 0; |
1392 | 18 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0); | 18 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0); |
1393 | 19 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0); | ||
1394 | 19 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0); | 20 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0); |
1395 | 20 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0); | 21 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0); |
1396 | 22 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_pwait), 0); | ||
1397 | 21 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept4), 0); | 23 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept4), 0); |
1398 | 22 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept), 0); | 24 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(accept), 0); |
1399 | 23 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0); | 25 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0); |
1400 | 26 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 0); | ||
1401 | 24 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0); | 27 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0); |
1402 | 25 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mmap), 0); | 28 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mmap), 0); |
1403 | 26 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0); | 29 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0); |
1404 | 27 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(shmctl), 0); | 30 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(shmctl), 0); |
1405 | 28 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0); | 31 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0); |
1406 | 32 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0); | ||
1407 | 33 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TIOCGWINSZ)); | ||
1408 | 34 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TCGETS)); | ||
1409 | 35 | |||
1410 | 36 | #if defined(HAVE_CLOCK_GETTIME) && defined(CLOCK_MONOTONIC) | ||
1411 | 37 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clock_gettime), 0); | ||
1412 | 38 | #endif | ||
1413 | 29 | 39 | ||
1414 | 30 | #ifdef MEMCACHED_DEBUG | 40 | #ifdef MEMCACHED_DEBUG |
1415 | 31 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0); | 41 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0); |
1416 | 32 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0); | 42 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0); |
1417 | 33 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0); | 43 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0); |
1418 | 44 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(readv), 0); | ||
1419 | 34 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0); | 45 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0); |
1420 | 35 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0); | 46 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0); |
1421 | 36 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0); | 47 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0); |
1422 | 48 | |||
1423 | 49 | if (settings.relaxed_privileges) { | ||
1424 | 50 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(openat), 0); | ||
1425 | 51 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mkdir), 0); | ||
1426 | 52 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(access), 0); | ||
1427 | 53 | } | ||
1428 | 37 | #endif | 54 | #endif |
1429 | 38 | 55 | ||
1430 | 39 | if (rc != 0) { | 56 | if (rc != 0) { |
1431 | @@ -45,8 +62,13 @@ void drop_privileges(void) { | |||
1432 | 45 | goto fail; | 62 | goto fail; |
1433 | 46 | } | 63 | } |
1434 | 47 | 64 | ||
1435 | 65 | seccomp_release(ctx); | ||
1436 | 66 | return; | ||
1437 | 67 | |||
1438 | 48 | fail: | 68 | fail: |
1439 | 49 | seccomp_release(ctx); | 69 | seccomp_release(ctx); |
1440 | 70 | fprintf(stderr, "Failed to set a seccomp profile on the main thread\n"); | ||
1441 | 71 | exit(EXIT_FAILURE); | ||
1442 | 50 | } | 72 | } |
1443 | 51 | 73 | ||
1444 | 52 | void drop_worker_privileges(void) { | 74 | void drop_worker_privileges(void) { |
1445 | @@ -57,10 +79,14 @@ void drop_worker_privileges(void) { | |||
1446 | 57 | 79 | ||
1447 | 58 | int rc = 0; | 80 | int rc = 0; |
1448 | 59 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0); | 81 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(sigreturn), 0); |
1449 | 82 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0); | ||
1450 | 60 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0); | 83 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(futex), 0); |
1451 | 61 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0); | 84 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_wait), 0); |
1452 | 85 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_pwait), 0); | ||
1453 | 62 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_ctl), 0); | 86 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(epoll_ctl), 0); |
1454 | 87 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(poll), 0); | ||
1455 | 63 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0); | 88 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(read), 0); |
1456 | 89 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(readv), 0); | ||
1457 | 64 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mprotect), 0); | 90 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mprotect), 0); |
1458 | 65 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpeername), 0); | 91 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpeername), 0); |
1459 | 66 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0); | 92 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(close), 0); |
1460 | @@ -70,6 +96,8 @@ void drop_worker_privileges(void) { | |||
1461 | 70 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mremap), 0); | 96 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mremap), 0); |
1462 | 71 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0); | 97 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(munmap), 0); |
1463 | 72 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(recvfrom), 0); | 98 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(recvfrom), 0); |
1464 | 99 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(brk), 0); | ||
1465 | 100 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(ioctl), 1, SCMP_A1(SCMP_CMP_EQ, TIOCGWINSZ)); | ||
1466 | 73 | 101 | ||
1467 | 74 | // for spawning the LRU crawler | 102 | // for spawning the LRU crawler |
1468 | 75 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clone), 0); | 103 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(clone), 0); |
1469 | @@ -83,20 +111,29 @@ void drop_worker_privileges(void) { | |||
1470 | 83 | 111 | ||
1471 | 84 | if (settings.shutdown_command) { | 112 | if (settings.shutdown_command) { |
1472 | 85 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tgkill), 0); | 113 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tgkill), 0); |
1473 | 114 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(tkill), 0); | ||
1474 | 86 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0); | 115 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(exit_group), 0); |
1475 | 87 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0); | 116 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fstat), 0); |
1476 | 88 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(rt_sigprocmask), 0); | ||
1477 | 89 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0); | 117 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(getpid), 0); |
1478 | 90 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(gettid), 0); | 118 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(gettid), 0); |
1479 | 91 | } | 119 | } |
1480 | 92 | 120 | ||
1481 | 93 | if (settings.relaxed_privileges) { | 121 | if (settings.relaxed_privileges) { |
1482 | 122 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(openat), 0); | ||
1483 | 123 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(mkdir), 0); | ||
1484 | 124 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(access), 0); | ||
1485 | 94 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0); | 125 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(open), 0); |
1486 | 95 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0); | 126 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(fcntl), 0); |
1487 | 96 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0); | 127 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(lseek), 0); |
1488 | 97 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0); | 128 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 0); |
1489 | 129 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 0); | ||
1490 | 98 | } else { | 130 | } else { |
1491 | 131 | // stdout | ||
1492 | 99 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 1)); | 132 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 1)); |
1493 | 133 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 1, SCMP_A0(SCMP_CMP_EQ, 1)); | ||
1494 | 134 | // stderr | ||
1495 | 135 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(write), 1, SCMP_A0(SCMP_CMP_EQ, 2)); | ||
1496 | 136 | rc |= seccomp_rule_add(ctx, SCMP_ACT_ALLOW, SCMP_SYS(writev), 1, SCMP_A0(SCMP_CMP_EQ, 2)); | ||
1497 | 100 | } | 137 | } |
1498 | 101 | 138 | ||
1499 | 102 | if (rc != 0) { | 139 | if (rc != 0) { |
1500 | @@ -108,6 +145,11 @@ void drop_worker_privileges(void) { | |||
1501 | 108 | goto fail; | 145 | goto fail; |
1502 | 109 | } | 146 | } |
1503 | 110 | 147 | ||
1504 | 148 | seccomp_release(ctx); | ||
1505 | 149 | return; | ||
1506 | 150 | |||
1507 | 111 | fail: | 151 | fail: |
1508 | 112 | seccomp_release(ctx); | 152 | seccomp_release(ctx); |
1509 | 153 | fprintf(stderr, "Failed to set a seccomp profile on a worker thread\n"); | ||
1510 | 154 | exit(EXIT_FAILURE); | ||
1511 | 113 | } | 155 | } |
1512 | diff --git a/logger.c b/logger.c | |||
1513 | index 7af9917..1322d7d 100644 | |||
1514 | --- a/logger.c | |||
1515 | +++ b/logger.c | |||
1516 | @@ -363,7 +363,7 @@ static int logger_thread_read(logger *l, struct logger_stats *ls) { | |||
1517 | 363 | } else { | 363 | } else { |
1518 | 364 | logger_thread_write_entry(e, ls, scratch, scratch_len); | 364 | logger_thread_write_entry(e, ls, scratch, scratch_len); |
1519 | 365 | } | 365 | } |
1521 | 366 | pos += sizeof(logentry) + e->size; | 366 | pos += sizeof(logentry) + e->size + e->pad; |
1522 | 367 | } | 367 | } |
1523 | 368 | assert(pos <= size); | 368 | assert(pos <= size); |
1524 | 369 | 369 | ||
1525 | @@ -699,8 +699,9 @@ enum logger_ret_type logger_log(logger *l, const enum log_entry_type event, cons | |||
1526 | 699 | l->dropped++; | 699 | l->dropped++; |
1527 | 700 | return LOGGER_RET_NOSPACE; | 700 | return LOGGER_RET_NOSPACE; |
1528 | 701 | } | 701 | } |
1529 | 702 | e->gid = logger_get_gid(); | ||
1530 | 703 | e->event = d->subtype; | 702 | e->event = d->subtype; |
1531 | 703 | e->pad = 0; | ||
1532 | 704 | e->gid = logger_get_gid(); | ||
1533 | 704 | /* TODO: Could pass this down as an argument now that we're using | 705 | /* TODO: Could pass this down as an argument now that we're using |
1534 | 705 | * LOGGER_LOG() macro. | 706 | * LOGGER_LOG() macro. |
1535 | 706 | */ | 707 | */ |
1536 | @@ -751,11 +752,19 @@ enum logger_ret_type logger_log(logger *l, const enum log_entry_type event, cons | |||
1537 | 751 | rel_time_t sttl = va_arg(ap, rel_time_t); | 752 | rel_time_t sttl = va_arg(ap, rel_time_t); |
1538 | 752 | uint8_t sclsid = va_arg(ap, int); | 753 | uint8_t sclsid = va_arg(ap, int); |
1539 | 753 | _logger_log_item_store(e, status, comm, skey, snkey, sttl, sclsid); | 754 | _logger_log_item_store(e, status, comm, skey, snkey, sttl, sclsid); |
1540 | 755 | va_end(ap); | ||
1541 | 754 | break; | 756 | break; |
1542 | 755 | } | 757 | } |
1543 | 756 | 758 | ||
1544 | 759 | #ifdef NEED_ALIGN | ||
1545 | 760 | /* Need to ensure *next* request is aligned. */ | ||
1546 | 761 | if (sizeof(logentry) + e->size % 8 != 0) { | ||
1547 | 762 | e->pad = 8 - (sizeof(logentry) + e->size % 8); | ||
1548 | 763 | } | ||
1549 | 764 | #endif | ||
1550 | 765 | |||
1551 | 757 | /* Push pointer forward by the actual amount required */ | 766 | /* Push pointer forward by the actual amount required */ |
1553 | 758 | if (bipbuf_push(buf, (sizeof(logentry) + e->size)) == 0) { | 767 | if (bipbuf_push(buf, (sizeof(logentry) + e->size + e->pad)) == 0) { |
1554 | 759 | fprintf(stderr, "LOGGER: Failed to bipbuf push a text entry\n"); | 768 | fprintf(stderr, "LOGGER: Failed to bipbuf push a text entry\n"); |
1555 | 760 | pthread_mutex_unlock(&l->mutex); | 769 | pthread_mutex_unlock(&l->mutex); |
1556 | 761 | return LOGGER_RET_ERR; | 770 | return LOGGER_RET_ERR; |
1557 | diff --git a/logger.h b/logger.h | |||
1558 | index 3d4c44c..730c86f 100644 | |||
1559 | --- a/logger.h | |||
1560 | +++ b/logger.h | |||
1561 | @@ -100,6 +100,7 @@ struct logentry_item_store { | |||
1562 | 100 | 100 | ||
1563 | 101 | typedef struct _logentry { | 101 | typedef struct _logentry { |
1564 | 102 | enum log_entry_subtype event; | 102 | enum log_entry_subtype event; |
1565 | 103 | uint8_t pad; | ||
1566 | 103 | uint16_t eflags; | 104 | uint16_t eflags; |
1567 | 104 | uint64_t gid; | 105 | uint64_t gid; |
1568 | 105 | struct timeval tv; /* not monotonic! */ | 106 | struct timeval tv; /* not monotonic! */ |
1569 | diff --git a/memcached.c b/memcached.c | |||
1570 | index 7178666..621b317 100644 | |||
1571 | --- a/memcached.c | |||
1572 | +++ b/memcached.c | |||
1573 | @@ -271,7 +271,7 @@ static void settings_init(void) { | |||
1574 | 271 | settings.crawls_persleep = 1000; | 271 | settings.crawls_persleep = 1000; |
1575 | 272 | settings.logger_watcher_buf_size = LOGGER_WATCHER_BUF_SIZE; | 272 | settings.logger_watcher_buf_size = LOGGER_WATCHER_BUF_SIZE; |
1576 | 273 | settings.logger_buf_size = LOGGER_BUF_SIZE; | 273 | settings.logger_buf_size = LOGGER_BUF_SIZE; |
1578 | 274 | settings.drop_privileges = true; | 274 | settings.drop_privileges = false; |
1579 | 275 | #ifdef MEMCACHED_DEBUG | 275 | #ifdef MEMCACHED_DEBUG |
1580 | 276 | settings.relaxed_privileges = false; | 276 | settings.relaxed_privileges = false; |
1581 | 277 | #endif | 277 | #endif |
1582 | @@ -612,6 +612,7 @@ conn *conn_new(const int sfd, enum conn_states init_state, | |||
1583 | 612 | c->iovused = 0; | 612 | c->iovused = 0; |
1584 | 613 | c->msgcurr = 0; | 613 | c->msgcurr = 0; |
1585 | 614 | c->msgused = 0; | 614 | c->msgused = 0; |
1586 | 615 | c->sasl_started = false; | ||
1587 | 615 | c->authenticated = false; | 616 | c->authenticated = false; |
1588 | 616 | c->last_cmd_time = current_time; /* initialize for idle kicker */ | 617 | c->last_cmd_time = current_time; /* initialize for idle kicker */ |
1589 | 617 | #ifdef EXTSTORE | 618 | #ifdef EXTSTORE |
1590 | @@ -648,8 +649,19 @@ static void recache_or_free(conn *c, io_wrap *wrap) { | |||
1591 | 648 | item *it; | 649 | item *it; |
1592 | 649 | it = (item *)wrap->io.buf; | 650 | it = (item *)wrap->io.buf; |
1593 | 650 | bool do_free = true; | 651 | bool do_free = true; |
1596 | 651 | // If request was ultimately a miss, unlink the header. | 652 | if (wrap->active) { |
1597 | 652 | if (wrap->miss) { | 653 | // If request never dispatched, free the read buffer but leave the |
1598 | 654 | // item header alone. | ||
1599 | 655 | do_free = false; | ||
1600 | 656 | size_t ntotal = ITEM_ntotal(wrap->hdr_it); | ||
1601 | 657 | slabs_free(it, ntotal, slabs_clsid(ntotal)); | ||
1602 | 658 | c->io_wrapleft--; | ||
1603 | 659 | assert(c->io_wrapleft >= 0); | ||
1604 | 660 | pthread_mutex_lock(&c->thread->stats.mutex); | ||
1605 | 661 | c->thread->stats.get_aborted_extstore++; | ||
1606 | 662 | pthread_mutex_unlock(&c->thread->stats.mutex); | ||
1607 | 663 | } else if (wrap->miss) { | ||
1608 | 664 | // If request was ultimately a miss, unlink the header. | ||
1609 | 653 | do_free = false; | 665 | do_free = false; |
1610 | 654 | size_t ntotal = ITEM_ntotal(wrap->hdr_it); | 666 | size_t ntotal = ITEM_ntotal(wrap->hdr_it); |
1611 | 655 | item_unlink(wrap->hdr_it); | 667 | item_unlink(wrap->hdr_it); |
1612 | @@ -1020,7 +1032,7 @@ static int add_iov(conn *c, const void *buf, int len) { | |||
1613 | 1020 | 1032 | ||
1614 | 1021 | static int add_chunked_item_iovs(conn *c, item *it, int len) { | 1033 | static int add_chunked_item_iovs(conn *c, item *it, int len) { |
1615 | 1022 | assert(it->it_flags & ITEM_CHUNKED); | 1034 | assert(it->it_flags & ITEM_CHUNKED); |
1617 | 1023 | item_chunk *ch = (item_chunk *) ITEM_data(it); | 1035 | item_chunk *ch = (item_chunk *) ITEM_schunk(it); |
1618 | 1024 | while (ch) { | 1036 | while (ch) { |
1619 | 1025 | int todo = (len > ch->used) ? ch->used : len; | 1037 | int todo = (len > ch->used) ? ch->used : len; |
1620 | 1026 | if (add_iov(c, ch->data, todo) != 0) { | 1038 | if (add_iov(c, ch->data, todo) != 0) { |
1621 | @@ -1640,13 +1652,8 @@ static void process_bin_get_or_touch(conn *c) { | |||
1622 | 1640 | rsp->message.header.response.cas = htonll(ITEM_get_cas(it)); | 1652 | rsp->message.header.response.cas = htonll(ITEM_get_cas(it)); |
1623 | 1641 | 1653 | ||
1624 | 1642 | // add the flags | 1654 | // add the flags |
1632 | 1643 | if (settings.inline_ascii_response) { | 1655 | FLAGS_CONV(settings.inline_ascii_response, it, rsp->message.body.flags); |
1633 | 1644 | rsp->message.body.flags = htonl(strtoul(ITEM_suffix(it), NULL, 10)); | 1656 | rsp->message.body.flags = htonl(rsp->message.body.flags); |
1627 | 1645 | } else if (it->nsuffix > 0) { | ||
1628 | 1646 | rsp->message.body.flags = htonl(*((uint32_t *)ITEM_suffix(it))); | ||
1629 | 1647 | } else { | ||
1630 | 1648 | rsp->message.body.flags = 0; | ||
1631 | 1649 | } | ||
1634 | 1650 | add_iov(c, &rsp->message.body, sizeof(rsp->message.body)); | 1657 | add_iov(c, &rsp->message.body, sizeof(rsp->message.body)); |
1635 | 1651 | 1658 | ||
1636 | 1652 | if (should_return_key) { | 1659 | if (should_return_key) { |
1637 | @@ -1663,10 +1670,14 @@ static void process_bin_get_or_touch(conn *c) { | |||
1638 | 1663 | iovcnt = 3; | 1670 | iovcnt = 3; |
1639 | 1664 | iovst = c->iovused - 2; | 1671 | iovst = c->iovused - 2; |
1640 | 1665 | } | 1672 | } |
1644 | 1666 | // FIXME: this can return an error, but code flow doesn't | 1673 | |
1645 | 1667 | // allow bailing here. | 1674 | if (_get_extstore(c, it, iovst, iovcnt) != 0) { |
1646 | 1668 | if (_get_extstore(c, it, iovst, iovcnt) != 0) | 1675 | pthread_mutex_lock(&c->thread->stats.mutex); |
1647 | 1676 | c->thread->stats.get_oom_extstore++; | ||
1648 | 1677 | pthread_mutex_unlock(&c->thread->stats.mutex); | ||
1649 | 1678 | |||
1650 | 1669 | failed = true; | 1679 | failed = true; |
1651 | 1680 | } | ||
1652 | 1670 | } else if ((it->it_flags & ITEM_CHUNKED) == 0) { | 1681 | } else if ((it->it_flags & ITEM_CHUNKED) == 0) { |
1653 | 1671 | add_iov(c, ITEM_data(it), it->nbytes - 2); | 1682 | add_iov(c, ITEM_data(it), it->nbytes - 2); |
1654 | 1672 | } else { | 1683 | } else { |
1655 | @@ -1686,10 +1697,11 @@ static void process_bin_get_or_touch(conn *c) { | |||
1656 | 1686 | c->write_and_go = conn_new_cmd; | 1697 | c->write_and_go = conn_new_cmd; |
1657 | 1687 | /* Remember this command so we can garbage collect it later */ | 1698 | /* Remember this command so we can garbage collect it later */ |
1658 | 1688 | #ifdef EXTSTORE | 1699 | #ifdef EXTSTORE |
1662 | 1689 | if ((it->it_flags & ITEM_HDR) == 0) { | 1700 | if ((it->it_flags & ITEM_HDR) != 0 && should_return_value) { |
1663 | 1690 | c->item = it; | 1701 | // Only have extstore clean if header and returning value. |
1661 | 1691 | } else { | ||
1664 | 1692 | c->item = NULL; | 1702 | c->item = NULL; |
1665 | 1703 | } else { | ||
1666 | 1704 | c->item = it; | ||
1667 | 1693 | } | 1705 | } |
1668 | 1694 | #else | 1706 | #else |
1669 | 1695 | c->item = it; | 1707 | c->item = it; |
1670 | @@ -2105,8 +2117,16 @@ static void process_bin_complete_sasl_auth(conn *c) { | |||
1671 | 2105 | result = sasl_server_start(c->sasl_conn, mech, | 2117 | result = sasl_server_start(c->sasl_conn, mech, |
1672 | 2106 | challenge, vlen, | 2118 | challenge, vlen, |
1673 | 2107 | &out, &outlen); | 2119 | &out, &outlen); |
1674 | 2120 | c->sasl_started = (result == SASL_OK || result == SASL_CONTINUE); | ||
1675 | 2108 | break; | 2121 | break; |
1676 | 2109 | case PROTOCOL_BINARY_CMD_SASL_STEP: | 2122 | case PROTOCOL_BINARY_CMD_SASL_STEP: |
1677 | 2123 | if (!c->sasl_started) { | ||
1678 | 2124 | if (settings.verbose) { | ||
1679 | 2125 | fprintf(stderr, "%d: SASL_STEP called but sasl_server_start " | ||
1680 | 2126 | "not called for this connection!\n", c->sfd); | ||
1681 | 2127 | } | ||
1682 | 2128 | break; | ||
1683 | 2129 | } | ||
1684 | 2110 | result = sasl_server_step(c->sasl_conn, | 2130 | result = sasl_server_step(c->sasl_conn, |
1685 | 2111 | challenge, vlen, | 2131 | challenge, vlen, |
1686 | 2112 | &out, &outlen); | 2132 | &out, &outlen); |
1687 | @@ -2465,7 +2485,15 @@ static void process_bin_update(conn *c) { | |||
1688 | 2465 | } | 2485 | } |
1689 | 2466 | 2486 | ||
1690 | 2467 | c->item = it; | 2487 | c->item = it; |
1691 | 2488 | #ifdef NEED_ALIGN | ||
1692 | 2489 | if (it->it_flags & ITEM_CHUNKED) { | ||
1693 | 2490 | c->ritem = ITEM_schunk(it); | ||
1694 | 2491 | } else { | ||
1695 | 2492 | c->ritem = ITEM_data(it); | ||
1696 | 2493 | } | ||
1697 | 2494 | #else | ||
1698 | 2468 | c->ritem = ITEM_data(it); | 2495 | c->ritem = ITEM_data(it); |
1699 | 2496 | #endif | ||
1700 | 2469 | c->rlbytes = vlen; | 2497 | c->rlbytes = vlen; |
1701 | 2470 | conn_set_state(c, conn_nread); | 2498 | conn_set_state(c, conn_nread); |
1702 | 2471 | c->substate = bin_read_set_value; | 2499 | c->substate = bin_read_set_value; |
1703 | @@ -2520,7 +2548,15 @@ static void process_bin_append_prepend(conn *c) { | |||
1704 | 2520 | } | 2548 | } |
1705 | 2521 | 2549 | ||
1706 | 2522 | c->item = it; | 2550 | c->item = it; |
1707 | 2551 | #ifdef NEED_ALIGN | ||
1708 | 2552 | if (it->it_flags & ITEM_CHUNKED) { | ||
1709 | 2553 | c->ritem = ITEM_schunk(it); | ||
1710 | 2554 | } else { | ||
1711 | 2555 | c->ritem = ITEM_data(it); | ||
1712 | 2556 | } | ||
1713 | 2557 | #else | ||
1714 | 2523 | c->ritem = ITEM_data(it); | 2558 | c->ritem = ITEM_data(it); |
1715 | 2559 | #endif | ||
1716 | 2524 | c->rlbytes = vlen; | 2560 | c->rlbytes = vlen; |
1717 | 2525 | conn_set_state(c, conn_nread); | 2561 | conn_set_state(c, conn_nread); |
1718 | 2526 | c->substate = bin_read_set_value; | 2562 | c->substate = bin_read_set_value; |
1719 | @@ -2681,7 +2717,7 @@ static void complete_nread(conn *c) { | |||
1720 | 2681 | /* Destination must always be chunked */ | 2717 | /* Destination must always be chunked */ |
1721 | 2682 | /* This should be part of item.c */ | 2718 | /* This should be part of item.c */ |
1722 | 2683 | static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) { | 2719 | static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) { |
1724 | 2684 | item_chunk *dch = (item_chunk *) ITEM_data(d_it); | 2720 | item_chunk *dch = (item_chunk *) ITEM_schunk(d_it); |
1725 | 2685 | /* Advance dch until we find free space */ | 2721 | /* Advance dch until we find free space */ |
1726 | 2686 | while (dch->size == dch->used) { | 2722 | while (dch->size == dch->used) { |
1727 | 2687 | if (dch->next) { | 2723 | if (dch->next) { |
1728 | @@ -2693,7 +2729,7 @@ static int _store_item_copy_chunks(item *d_it, item *s_it, const int len) { | |||
1729 | 2693 | 2729 | ||
1730 | 2694 | if (s_it->it_flags & ITEM_CHUNKED) { | 2730 | if (s_it->it_flags & ITEM_CHUNKED) { |
1731 | 2695 | int remain = len; | 2731 | int remain = len; |
1733 | 2696 | item_chunk *sch = (item_chunk *) ITEM_data(s_it); | 2732 | item_chunk *sch = (item_chunk *) ITEM_schunk(s_it); |
1734 | 2697 | int copied = 0; | 2733 | int copied = 0; |
1735 | 2698 | /* Fills dch's to capacity, not straight copy sch in case data is | 2734 | /* Fills dch's to capacity, not straight copy sch in case data is |
1736 | 2699 | * being added or removed (ie append/prepend) | 2735 | * being added or removed (ie append/prepend) |
1737 | @@ -2857,15 +2893,7 @@ enum store_item_type do_store_item(item *it, int comm, conn *c, const uint32_t h | |||
1738 | 2857 | if (stored == NOT_STORED) { | 2893 | if (stored == NOT_STORED) { |
1739 | 2858 | /* we have it and old_it here - alloc memory to hold both */ | 2894 | /* we have it and old_it here - alloc memory to hold both */ |
1740 | 2859 | /* flags was already lost - so recover them from ITEM_suffix(it) */ | 2895 | /* flags was already lost - so recover them from ITEM_suffix(it) */ |
1750 | 2860 | 2896 | FLAGS_CONV(settings.inline_ascii_response, old_it, flags); | |
1742 | 2861 | if (settings.inline_ascii_response) { | ||
1743 | 2862 | flags = (uint32_t) strtoul(ITEM_suffix(old_it), (char **) NULL, 10); | ||
1744 | 2863 | } else if (old_it->nsuffix > 0) { | ||
1745 | 2864 | flags = *((uint32_t *)ITEM_suffix(old_it)); | ||
1746 | 2865 | } else { | ||
1747 | 2866 | flags = 0; | ||
1748 | 2867 | } | ||
1749 | 2868 | |||
1751 | 2869 | new_it = do_item_alloc(key, it->nkey, flags, old_it->exptime, it->nbytes + old_it->nbytes - 2 /* CRLF */); | 2897 | new_it = do_item_alloc(key, it->nkey, flags, old_it->exptime, it->nbytes + old_it->nbytes - 2 /* CRLF */); |
1752 | 2870 | 2898 | ||
1753 | 2871 | /* copy data from it and old_it to new_it */ | 2899 | /* copy data from it and old_it to new_it */ |
1754 | @@ -3105,6 +3133,8 @@ static void server_stats(ADD_STAT add_stats, conn *c) { | |||
1755 | 3105 | #ifdef EXTSTORE | 3133 | #ifdef EXTSTORE |
1756 | 3106 | if (c->thread->storage) { | 3134 | if (c->thread->storage) { |
1757 | 3107 | APPEND_STAT("get_extstore", "%llu", (unsigned long long)thread_stats.get_extstore); | 3135 | APPEND_STAT("get_extstore", "%llu", (unsigned long long)thread_stats.get_extstore); |
1758 | 3136 | APPEND_STAT("get_aborted_extstore", "%llu", (unsigned long long)thread_stats.get_aborted_extstore); | ||
1759 | 3137 | APPEND_STAT("get_oom_extstore", "%llu", (unsigned long long)thread_stats.get_oom_extstore); | ||
1760 | 3108 | APPEND_STAT("recache_from_extstore", "%llu", (unsigned long long)thread_stats.recache_from_extstore); | 3138 | APPEND_STAT("recache_from_extstore", "%llu", (unsigned long long)thread_stats.recache_from_extstore); |
1761 | 3109 | APPEND_STAT("miss_from_extstore", "%llu", (unsigned long long)thread_stats.miss_from_extstore); | 3139 | APPEND_STAT("miss_from_extstore", "%llu", (unsigned long long)thread_stats.miss_from_extstore); |
1762 | 3110 | APPEND_STAT("badcrc_from_extstore", "%llu", (unsigned long long)thread_stats.badcrc_from_extstore); | 3140 | APPEND_STAT("badcrc_from_extstore", "%llu", (unsigned long long)thread_stats.badcrc_from_extstore); |
1763 | @@ -3242,6 +3272,9 @@ static void process_stat_settings(ADD_STAT add_stats, void *c) { | |||
1764 | 3242 | APPEND_STAT("worker_logbuf_size", "%u", settings.logger_buf_size); | 3272 | APPEND_STAT("worker_logbuf_size", "%u", settings.logger_buf_size); |
1765 | 3243 | APPEND_STAT("track_sizes", "%s", item_stats_sizes_status() ? "yes" : "no"); | 3273 | APPEND_STAT("track_sizes", "%s", item_stats_sizes_status() ? "yes" : "no"); |
1766 | 3244 | APPEND_STAT("inline_ascii_response", "%s", settings.inline_ascii_response ? "yes" : "no"); | 3274 | APPEND_STAT("inline_ascii_response", "%s", settings.inline_ascii_response ? "yes" : "no"); |
1767 | 3275 | #ifdef HAVE_DROP_PRIVILEGES | ||
1768 | 3276 | APPEND_STAT("drop_privileges", "%s", settings.drop_privileges ? "yes" : "no"); | ||
1769 | 3277 | #endif | ||
1770 | 3245 | #ifdef EXTSTORE | 3278 | #ifdef EXTSTORE |
1771 | 3246 | APPEND_STAT("ext_item_size", "%u", settings.ext_item_size); | 3279 | APPEND_STAT("ext_item_size", "%u", settings.ext_item_size); |
1772 | 3247 | APPEND_STAT("ext_item_age", "%u", settings.ext_item_age); | 3280 | APPEND_STAT("ext_item_age", "%u", settings.ext_item_age); |
1773 | @@ -3384,6 +3417,8 @@ static void process_extstore_stats(ADD_STAT add_stats, conn *c) { | |||
1774 | 3384 | (unsigned long long) st.page_data[i].bytes_used); | 3417 | (unsigned long long) st.page_data[i].bytes_used); |
1775 | 3385 | APPEND_NUM_STAT(i, "bucket", "%u", | 3418 | APPEND_NUM_STAT(i, "bucket", "%u", |
1776 | 3386 | st.page_data[i].bucket); | 3419 | st.page_data[i].bucket); |
1777 | 3420 | APPEND_NUM_STAT(i, "free_bucket", "%u", | ||
1778 | 3421 | st.page_data[i].free_bucket); | ||
1779 | 3387 | } | 3422 | } |
1780 | 3388 | } | 3423 | } |
1781 | 3389 | #endif | 3424 | #endif |
1782 | @@ -3583,14 +3618,14 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) { | |||
1783 | 3583 | // item is chunked, crc the iov's | 3618 | // item is chunked, crc the iov's |
1784 | 3584 | if (io->iov != NULL) { | 3619 | if (io->iov != NULL) { |
1785 | 3585 | // first iov is the header, which we don't use beyond crc | 3620 | // first iov is the header, which we don't use beyond crc |
1787 | 3586 | crc2 = crc32c(0, (char *)io->iov[0].iov_base+32, io->iov[0].iov_len-32); | 3621 | crc2 = crc32c(0, (char *)io->iov[0].iov_base+STORE_OFFSET, io->iov[0].iov_len-STORE_OFFSET); |
1788 | 3587 | // make sure it's not sent. hack :( | 3622 | // make sure it's not sent. hack :( |
1789 | 3588 | io->iov[0].iov_len = 0; | 3623 | io->iov[0].iov_len = 0; |
1790 | 3589 | for (x = 1; x < io->iovcnt; x++) { | 3624 | for (x = 1; x < io->iovcnt; x++) { |
1791 | 3590 | crc2 = crc32c(crc2, (char *)io->iov[x].iov_base, io->iov[x].iov_len); | 3625 | crc2 = crc32c(crc2, (char *)io->iov[x].iov_base, io->iov[x].iov_len); |
1792 | 3591 | } | 3626 | } |
1793 | 3592 | } else { | 3627 | } else { |
1795 | 3593 | crc2 = crc32c(0, (char *)read_it+32, io->len-32); | 3628 | crc2 = crc32c(0, (char *)read_it+STORE_OFFSET, io->len-STORE_OFFSET); |
1796 | 3594 | } | 3629 | } |
1797 | 3595 | 3630 | ||
1798 | 3596 | if (crc != crc2) { | 3631 | if (crc != crc2) { |
1799 | @@ -3639,6 +3674,7 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) { | |||
1800 | 3639 | } | 3674 | } |
1801 | 3640 | } | 3675 | } |
1802 | 3641 | } | 3676 | } |
1803 | 3677 | wrap->miss = false; | ||
1804 | 3642 | // iov_len is already set | 3678 | // iov_len is already set |
1805 | 3643 | // TODO: Should do that here instead and cuddle in the wrap object | 3679 | // TODO: Should do that here instead and cuddle in the wrap object |
1806 | 3644 | } | 3680 | } |
1807 | @@ -3657,23 +3693,20 @@ static void _get_extstore_cb(void *e, obj_io *io, int ret) { | |||
1808 | 3657 | 3693 | ||
1809 | 3658 | // FIXME: This completely breaks UDP support. | 3694 | // FIXME: This completely breaks UDP support. |
1810 | 3659 | static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) { | 3695 | static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) { |
1811 | 3696 | #ifdef NEED_ALIGN | ||
1812 | 3697 | item_hdr hdr; | ||
1813 | 3698 | memcpy(&hdr, ITEM_data(it), sizeof(hdr)); | ||
1814 | 3699 | #else | ||
1815 | 3660 | item_hdr *hdr = (item_hdr *)ITEM_data(it); | 3700 | item_hdr *hdr = (item_hdr *)ITEM_data(it); |
1816 | 3701 | #endif | ||
1817 | 3661 | size_t ntotal = ITEM_ntotal(it); | 3702 | size_t ntotal = ITEM_ntotal(it); |
1818 | 3662 | unsigned int clsid = slabs_clsid(ntotal); | 3703 | unsigned int clsid = slabs_clsid(ntotal); |
1819 | 3663 | item *new_it; | 3704 | item *new_it; |
1820 | 3664 | bool chunked = false; | 3705 | bool chunked = false; |
1821 | 3665 | if (ntotal > settings.slab_chunk_size_max) { | 3706 | if (ntotal > settings.slab_chunk_size_max) { |
1822 | 3666 | // Pull a chunked item header. | 3707 | // Pull a chunked item header. |
1823 | 3667 | // FIXME: make a func. used in several places. | ||
1824 | 3668 | uint32_t flags; | 3708 | uint32_t flags; |
1833 | 3669 | if (settings.inline_ascii_response) { | 3709 | FLAGS_CONV(settings.inline_ascii_response, it, flags); |
1826 | 3670 | flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10); | ||
1827 | 3671 | } else if (it->nsuffix > 0) { | ||
1828 | 3672 | flags = *((uint32_t *)ITEM_suffix(it)); | ||
1829 | 3673 | } else { | ||
1830 | 3674 | flags = 0; | ||
1831 | 3675 | } | ||
1832 | 3676 | |||
1834 | 3677 | new_it = item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, it->nbytes); | 3710 | new_it = item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, it->nbytes); |
1835 | 3678 | assert(new_it == NULL || (new_it->it_flags & ITEM_CHUNKED)); | 3711 | assert(new_it == NULL || (new_it->it_flags & ITEM_CHUNKED)); |
1836 | 3679 | chunked = true; | 3712 | chunked = true; |
1837 | @@ -3702,13 +3735,12 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) { | |||
1838 | 3702 | if (chunked) { | 3735 | if (chunked) { |
1839 | 3703 | unsigned int ciovcnt = 1; | 3736 | unsigned int ciovcnt = 1; |
1840 | 3704 | size_t remain = new_it->nbytes; | 3737 | size_t remain = new_it->nbytes; |
1842 | 3705 | item_chunk *chunk = (item_chunk *) ITEM_data(new_it); | 3738 | item_chunk *chunk = (item_chunk *) ITEM_schunk(new_it); |
1843 | 3706 | io->io.iov = &c->iov[c->iovused]; | 3739 | io->io.iov = &c->iov[c->iovused]; |
1844 | 3707 | // fill the header so we can get the full data + crc back. | 3740 | // fill the header so we can get the full data + crc back. |
1845 | 3708 | add_iov(c, new_it, ITEM_ntotal(new_it) - new_it->nbytes); | 3741 | add_iov(c, new_it, ITEM_ntotal(new_it) - new_it->nbytes); |
1846 | 3709 | while (remain > 0) { | 3742 | while (remain > 0) { |
1847 | 3710 | chunk = do_item_alloc_chunk(chunk, remain); | 3743 | chunk = do_item_alloc_chunk(chunk, remain); |
1848 | 3711 | // TODO: counter bump | ||
1849 | 3712 | if (chunk == NULL) { | 3744 | if (chunk == NULL) { |
1850 | 3713 | item_remove(new_it); | 3745 | item_remove(new_it); |
1851 | 3714 | do_cache_free(c->thread->io_cache, io); | 3746 | do_cache_free(c->thread->io_cache, io); |
1852 | @@ -3745,9 +3777,15 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) { | |||
1853 | 3745 | io->io.data = (void *)io; | 3777 | io->io.data = (void *)io; |
1854 | 3746 | 3778 | ||
1855 | 3747 | // Now, fill in io->io based on what was in our header. | 3779 | // Now, fill in io->io based on what was in our header. |
1856 | 3780 | #ifdef NEED_ALIGN | ||
1857 | 3781 | io->io.page_version = hdr.page_version; | ||
1858 | 3782 | io->io.page_id = hdr.page_id; | ||
1859 | 3783 | io->io.offset = hdr.offset; | ||
1860 | 3784 | #else | ||
1861 | 3748 | io->io.page_version = hdr->page_version; | 3785 | io->io.page_version = hdr->page_version; |
1862 | 3749 | io->io.page_id = hdr->page_id; | 3786 | io->io.page_id = hdr->page_id; |
1863 | 3750 | io->io.offset = hdr->offset; | 3787 | io->io.offset = hdr->offset; |
1864 | 3788 | #endif | ||
1865 | 3751 | io->io.len = ntotal; | 3789 | io->io.len = ntotal; |
1866 | 3752 | io->io.mode = OBJ_IO_READ; | 3790 | io->io.mode = OBJ_IO_READ; |
1867 | 3753 | io->io.cb = _get_extstore_cb; | 3791 | io->io.cb = _get_extstore_cb; |
1868 | @@ -3763,8 +3801,6 @@ static inline int _get_extstore(conn *c, item *it, int iovst, int iovcnt) { | |||
1869 | 3763 | return 0; | 3801 | return 0; |
1870 | 3764 | } | 3802 | } |
1871 | 3765 | #endif | 3803 | #endif |
1872 | 3766 | // FIXME: the 'breaks' around memory malloc's should break all the way down, | ||
1873 | 3767 | // fill ileft/suffixleft, then run conn_releaseitems() | ||
1874 | 3768 | /* ntokens is overwritten here... shrug.. */ | 3804 | /* ntokens is overwritten here... shrug.. */ |
1875 | 3769 | static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, bool return_cas, bool should_touch) { | 3805 | static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, bool return_cas, bool should_touch) { |
1876 | 3770 | char *key; | 3806 | char *key; |
1877 | @@ -3776,6 +3812,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1878 | 3776 | char *suffix; | 3812 | char *suffix; |
1879 | 3777 | int32_t exptime_int = 0; | 3813 | int32_t exptime_int = 0; |
1880 | 3778 | rel_time_t exptime = 0; | 3814 | rel_time_t exptime = 0; |
1881 | 3815 | bool fail_length = false; | ||
1882 | 3779 | assert(c != NULL); | 3816 | assert(c != NULL); |
1883 | 3780 | 3817 | ||
1884 | 3781 | if (should_touch) { | 3818 | if (should_touch) { |
1885 | @@ -3795,14 +3832,8 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1886 | 3795 | nkey = key_token->length; | 3832 | nkey = key_token->length; |
1887 | 3796 | 3833 | ||
1888 | 3797 | if (nkey > KEY_MAX_LENGTH) { | 3834 | if (nkey > KEY_MAX_LENGTH) { |
1897 | 3798 | out_string(c, "CLIENT_ERROR bad command line format"); | 3835 | fail_length = true; |
1898 | 3799 | while (i-- > 0) { | 3836 | goto stop; |
1891 | 3800 | item_remove(*(c->ilist + i)); | ||
1892 | 3801 | if (return_cas || !settings.inline_ascii_response) { | ||
1893 | 3802 | do_cache_free(c->thread->suffix_cache, *(c->suffixlist + i)); | ||
1894 | 3803 | } | ||
1895 | 3804 | } | ||
1896 | 3805 | return; | ||
1899 | 3806 | } | 3837 | } |
1900 | 3807 | 3838 | ||
1901 | 3808 | it = limited_get(key, nkey, c, exptime, should_touch); | 3839 | it = limited_get(key, nkey, c, exptime, should_touch); |
1902 | @@ -3812,7 +3843,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1903 | 3812 | if (it) { | 3843 | if (it) { |
1904 | 3813 | if (_ascii_get_expand_ilist(c, i) != 0) { | 3844 | if (_ascii_get_expand_ilist(c, i) != 0) { |
1905 | 3814 | item_remove(it); | 3845 | item_remove(it); |
1907 | 3815 | break; // FIXME: Should bail down to error. | 3846 | goto stop; |
1908 | 3816 | } | 3847 | } |
1909 | 3817 | 3848 | ||
1910 | 3818 | /* | 3849 | /* |
1911 | @@ -3831,7 +3862,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1912 | 3831 | suffix = _ascii_get_suffix_buf(c, si); | 3862 | suffix = _ascii_get_suffix_buf(c, si); |
1913 | 3832 | if (suffix == NULL) { | 3863 | if (suffix == NULL) { |
1914 | 3833 | item_remove(it); | 3864 | item_remove(it); |
1916 | 3834 | break; | 3865 | goto stop; |
1917 | 3835 | } | 3866 | } |
1918 | 3836 | si++; | 3867 | si++; |
1919 | 3837 | nbytes = it->nbytes; | 3868 | nbytes = it->nbytes; |
1920 | @@ -3842,13 +3873,17 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1921 | 3842 | add_iov(c, suffix, suffix_len) != 0) | 3873 | add_iov(c, suffix, suffix_len) != 0) |
1922 | 3843 | { | 3874 | { |
1923 | 3844 | item_remove(it); | 3875 | item_remove(it); |
1925 | 3845 | break; | 3876 | goto stop; |
1926 | 3846 | } | 3877 | } |
1927 | 3847 | #ifdef EXTSTORE | 3878 | #ifdef EXTSTORE |
1928 | 3848 | if (it->it_flags & ITEM_HDR) { | 3879 | if (it->it_flags & ITEM_HDR) { |
1929 | 3849 | if (_get_extstore(c, it, c->iovused-3, 4) != 0) { | 3880 | if (_get_extstore(c, it, c->iovused-3, 4) != 0) { |
1930 | 3881 | pthread_mutex_lock(&c->thread->stats.mutex); | ||
1931 | 3882 | c->thread->stats.get_oom_extstore++; | ||
1932 | 3883 | pthread_mutex_unlock(&c->thread->stats.mutex); | ||
1933 | 3884 | |||
1934 | 3850 | item_remove(it); | 3885 | item_remove(it); |
1936 | 3851 | break; | 3886 | goto stop; |
1937 | 3852 | } | 3887 | } |
1938 | 3853 | } else if ((it->it_flags & ITEM_CHUNKED) == 0) { | 3888 | } else if ((it->it_flags & ITEM_CHUNKED) == 0) { |
1939 | 3854 | #else | 3889 | #else |
1940 | @@ -3857,7 +3892,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1941 | 3857 | add_iov(c, ITEM_data(it), it->nbytes); | 3892 | add_iov(c, ITEM_data(it), it->nbytes); |
1942 | 3858 | } else if (add_chunked_item_iovs(c, it, it->nbytes) != 0) { | 3893 | } else if (add_chunked_item_iovs(c, it, it->nbytes) != 0) { |
1943 | 3859 | item_remove(it); | 3894 | item_remove(it); |
1945 | 3860 | break; | 3895 | goto stop; |
1946 | 3861 | } | 3896 | } |
1947 | 3862 | } | 3897 | } |
1948 | 3863 | else | 3898 | else |
1949 | @@ -3868,19 +3903,19 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1950 | 3868 | add_iov(c, ITEM_key(it), it->nkey) != 0) | 3903 | add_iov(c, ITEM_key(it), it->nkey) != 0) |
1951 | 3869 | { | 3904 | { |
1952 | 3870 | item_remove(it); | 3905 | item_remove(it); |
1954 | 3871 | break; | 3906 | goto stop; |
1955 | 3872 | } | 3907 | } |
1956 | 3873 | if ((it->it_flags & ITEM_CHUNKED) == 0) | 3908 | if ((it->it_flags & ITEM_CHUNKED) == 0) |
1957 | 3874 | { | 3909 | { |
1958 | 3875 | if (add_iov(c, ITEM_suffix(it), it->nsuffix + it->nbytes) != 0) | 3910 | if (add_iov(c, ITEM_suffix(it), it->nsuffix + it->nbytes) != 0) |
1959 | 3876 | { | 3911 | { |
1960 | 3877 | item_remove(it); | 3912 | item_remove(it); |
1962 | 3878 | break; | 3913 | goto stop; |
1963 | 3879 | } | 3914 | } |
1964 | 3880 | } else if (add_iov(c, ITEM_suffix(it), it->nsuffix) != 0 || | 3915 | } else if (add_iov(c, ITEM_suffix(it), it->nsuffix) != 0 || |
1965 | 3881 | add_chunked_item_iovs(c, it, it->nbytes) != 0) { | 3916 | add_chunked_item_iovs(c, it, it->nbytes) != 0) { |
1966 | 3882 | item_remove(it); | 3917 | item_remove(it); |
1968 | 3883 | break; | 3918 | goto stop; |
1969 | 3884 | } | 3919 | } |
1970 | 3885 | } | 3920 | } |
1971 | 3886 | 3921 | ||
1972 | @@ -3940,6 +3975,7 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1973 | 3940 | } | 3975 | } |
1974 | 3941 | 3976 | ||
1975 | 3942 | } while(key_token->value != NULL); | 3977 | } while(key_token->value != NULL); |
1976 | 3978 | stop: | ||
1977 | 3943 | 3979 | ||
1978 | 3944 | c->icurr = c->ilist; | 3980 | c->icurr = c->ilist; |
1979 | 3945 | c->ileft = i; | 3981 | c->ileft = i; |
1980 | @@ -3958,7 +3994,12 @@ static inline void process_get_command(conn *c, token_t *tokens, size_t ntokens, | |||
1981 | 3958 | */ | 3994 | */ |
1982 | 3959 | if (key_token->value != NULL || add_iov(c, "END\r\n", 5) != 0 | 3995 | if (key_token->value != NULL || add_iov(c, "END\r\n", 5) != 0 |
1983 | 3960 | || (IS_UDP(c->transport) && build_udp_headers(c) != 0)) { | 3996 | || (IS_UDP(c->transport) && build_udp_headers(c) != 0)) { |
1985 | 3961 | out_of_memory(c, "SERVER_ERROR out of memory writing get response"); | 3997 | if (fail_length) { |
1986 | 3998 | out_string(c, "CLIENT_ERROR bad command line format"); | ||
1987 | 3999 | } else { | ||
1988 | 4000 | out_of_memory(c, "SERVER_ERROR out of memory writing get response"); | ||
1989 | 4001 | } | ||
1990 | 4002 | conn_release_items(c); | ||
1991 | 3962 | } | 4003 | } |
1992 | 3963 | else { | 4004 | else { |
1993 | 3964 | conn_set_state(c, conn_mwrite); | 4005 | conn_set_state(c, conn_mwrite); |
1994 | @@ -4055,7 +4096,15 @@ static void process_update_command(conn *c, token_t *tokens, const size_t ntoken | |||
1995 | 4055 | ITEM_set_cas(it, req_cas_id); | 4096 | ITEM_set_cas(it, req_cas_id); |
1996 | 4056 | 4097 | ||
1997 | 4057 | c->item = it; | 4098 | c->item = it; |
1998 | 4099 | #ifdef NEED_ALIGN | ||
1999 | 4100 | if (it->it_flags & ITEM_CHUNKED) { | ||
2000 | 4101 | c->ritem = ITEM_schunk(it); | ||
2001 | 4102 | } else { | ||
2002 | 4103 | c->ritem = ITEM_data(it); | ||
2003 | 4104 | } | ||
2004 | 4105 | #else | ||
2005 | 4058 | c->ritem = ITEM_data(it); | 4106 | c->ritem = ITEM_data(it); |
2006 | 4107 | #endif | ||
2007 | 4059 | c->rlbytes = it->nbytes; | 4108 | c->rlbytes = it->nbytes; |
2008 | 4060 | c->cmd = comm; | 4109 | c->cmd = comm; |
2009 | 4061 | conn_set_state(c, conn_nread); | 4110 | conn_set_state(c, conn_nread); |
2010 | @@ -4239,13 +4288,7 @@ enum delta_result_type do_add_delta(conn *c, const char *key, const size_t nkey, | |||
2011 | 4239 | } else if (it->refcount > 1) { | 4288 | } else if (it->refcount > 1) { |
2012 | 4240 | item *new_it; | 4289 | item *new_it; |
2013 | 4241 | uint32_t flags; | 4290 | uint32_t flags; |
2021 | 4242 | if (settings.inline_ascii_response) { | 4291 | FLAGS_CONV(settings.inline_ascii_response, it, flags); |
2015 | 4243 | flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10); | ||
2016 | 4244 | } else if (it->nsuffix > 0) { | ||
2017 | 4245 | flags = *((uint32_t *)ITEM_suffix(it)); | ||
2018 | 4246 | } else { | ||
2019 | 4247 | flags = 0; | ||
2020 | 4248 | } | ||
2022 | 4249 | new_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, res + 2); | 4292 | new_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, res + 2); |
2023 | 4250 | if (new_it == 0) { | 4293 | if (new_it == 0) { |
2024 | 4251 | do_item_remove(it); | 4294 | do_item_remove(it); |
2025 | @@ -5274,7 +5317,6 @@ static int read_into_chunked_item(conn *c) { | |||
2026 | 5274 | 5317 | ||
2027 | 5275 | while (c->rlbytes > 0) { | 5318 | while (c->rlbytes > 0) { |
2028 | 5276 | item_chunk *ch = (item_chunk *)c->ritem; | 5319 | item_chunk *ch = (item_chunk *)c->ritem; |
2029 | 5277 | assert(ch->used <= ch->size); | ||
2030 | 5278 | if (ch->size == ch->used) { | 5320 | if (ch->size == ch->used) { |
2031 | 5279 | // FIXME: ch->next is currently always 0. remove this? | 5321 | // FIXME: ch->next is currently always 0. remove this? |
2032 | 5280 | if (ch->next) { | 5322 | if (ch->next) { |
2033 | @@ -6156,7 +6198,7 @@ static void clock_handler(const int fd, const short which, void *arg) { | |||
2034 | 6156 | static void usage(void) { | 6198 | static void usage(void) { |
2035 | 6157 | printf(PACKAGE " " VERSION "\n"); | 6199 | printf(PACKAGE " " VERSION "\n"); |
2036 | 6158 | printf("-p, --port=<num> TCP port to listen on (default: 11211)\n" | 6200 | printf("-p, --port=<num> TCP port to listen on (default: 11211)\n" |
2038 | 6159 | "-U, --udp-port=<num> UDP port to listen on (default: 11211, 0 is off)\n" | 6201 | "-U, --udp-port=<num> UDP port to listen on (default: 0, off)\n" |
2039 | 6160 | "-s, --unix-socket=<file> UNIX socket to listen on (disables network support)\n" | 6202 | "-s, --unix-socket=<file> UNIX socket to listen on (disables network support)\n" |
2040 | 6161 | "-A, --enable-shutdown enable ascii \"shutdown\" command\n" | 6203 | "-A, --enable-shutdown enable ascii \"shutdown\" command\n" |
2041 | 6162 | "-a, --unix-mask=<mask> access mask for UNIX socket, in octal (default: 0700)\n" | 6204 | "-a, --unix-mask=<mask> access mask for UNIX socket, in octal (default: 0700)\n" |
2042 | @@ -6236,7 +6278,8 @@ static void usage(void) { | |||
2043 | 6236 | " currently: nothing\n" | 6278 | " currently: nothing\n" |
2044 | 6237 | " - no_modern: uses defaults of previous major version (1.4.x)\n" | 6279 | " - no_modern: uses defaults of previous major version (1.4.x)\n" |
2045 | 6238 | #ifdef HAVE_DROP_PRIVILEGES | 6280 | #ifdef HAVE_DROP_PRIVILEGES |
2047 | 6239 | " - no_drop_privileges: Disable drop_privileges in case it causes issues with\n" | 6281 | " - drop_privileges: enable dropping extra syscall privileges\n" |
2048 | 6282 | " - no_drop_privileges: disable drop_privileges in case it causes issues with\n" | ||
2049 | 6240 | " some customisation.\n" | 6283 | " some customisation.\n" |
2050 | 6241 | #ifdef MEMCACHED_DEBUG | 6284 | #ifdef MEMCACHED_DEBUG |
2051 | 6242 | " - relaxed_privileges: Running tests requires extra privileges.\n" | 6285 | " - relaxed_privileges: Running tests requires extra privileges.\n" |
2052 | @@ -6244,6 +6287,7 @@ static void usage(void) { | |||
2053 | 6244 | #endif | 6287 | #endif |
2054 | 6245 | #ifdef EXTSTORE | 6288 | #ifdef EXTSTORE |
2055 | 6246 | " - ext_path: file to write to for external storage.\n" | 6289 | " - ext_path: file to write to for external storage.\n" |
2056 | 6290 | " ie: ext_path=/mnt/d1/extstore:1G\n" | ||
2057 | 6247 | " - ext_page_size: size in megabytes of storage pages.\n" | 6291 | " - ext_page_size: size in megabytes of storage pages.\n" |
2058 | 6248 | " - ext_wbuf_size: size in megabytes of page write buffers.\n" | 6292 | " - ext_wbuf_size: size in megabytes of page write buffers.\n" |
2059 | 6249 | " - ext_threads: number of IO threads to run.\n" | 6293 | " - ext_threads: number of IO threads to run.\n" |
2060 | @@ -6255,6 +6299,7 @@ static void usage(void) { | |||
2061 | 6255 | " - ext_compact_under: compact when fewer than this many free pages\n" | 6299 | " - ext_compact_under: compact when fewer than this many free pages\n" |
2062 | 6256 | " - ext_drop_under: drop COLD items when fewer than this many free pages\n" | 6300 | " - ext_drop_under: drop COLD items when fewer than this many free pages\n" |
2063 | 6257 | " - ext_max_frag: max page fragmentation to tolerage\n" | 6301 | " - ext_max_frag: max page fragmentation to tolerage\n" |
2064 | 6302 | " - slab_automove_freeratio: ratio of memory to hold free as buffer.\n" | ||
2065 | 6258 | " (see doc/storage.txt for more info)\n" | 6303 | " (see doc/storage.txt for more info)\n" |
2066 | 6259 | #endif | 6304 | #endif |
2067 | 6260 | ); | 6305 | ); |
2068 | @@ -6436,6 +6481,16 @@ static int enable_large_pages(void) { | |||
2069 | 6436 | } | 6481 | } |
2070 | 6437 | 6482 | ||
2071 | 6438 | return ret; | 6483 | return ret; |
2072 | 6484 | #elif defined(__linux__) && defined(MADV_HUGEPAGE) | ||
2073 | 6485 | /* check if transparent hugepages is compiled into the kernel */ | ||
2074 | 6486 | struct stat st; | ||
2075 | 6487 | int ret = stat("/sys/kernel/mm/transparent_hugepage/enabled", &st); | ||
2076 | 6488 | if (ret || !(st.st_mode & S_IFREG)) { | ||
2077 | 6489 | fprintf(stderr, "Transparent huge pages support not detected.\n"); | ||
2078 | 6490 | fprintf(stderr, "Will use default page size.\n"); | ||
2079 | 6491 | return -1; | ||
2080 | 6492 | } | ||
2081 | 6493 | return 0; | ||
2082 | 6439 | #else | 6494 | #else |
2083 | 6440 | return -1; | 6495 | return -1; |
2084 | 6441 | #endif | 6496 | #endif |
2085 | @@ -6534,7 +6589,7 @@ int main (int argc, char **argv) { | |||
2086 | 6534 | bool slab_chunk_size_changed = false; | 6589 | bool slab_chunk_size_changed = false; |
2087 | 6535 | #ifdef EXTSTORE | 6590 | #ifdef EXTSTORE |
2088 | 6536 | void *storage = NULL; | 6591 | void *storage = NULL; |
2090 | 6537 | char *storage_file = NULL; | 6592 | struct extstore_conf_file *storage_file = NULL; |
2091 | 6538 | struct extstore_conf ext_cf; | 6593 | struct extstore_conf ext_cf; |
2092 | 6539 | #endif | 6594 | #endif |
2093 | 6540 | char *subopts, *subopts_orig; | 6595 | char *subopts, *subopts_orig; |
2094 | @@ -6575,12 +6630,12 @@ int main (int argc, char **argv) { | |||
2095 | 6575 | NO_LRU_CRAWLER, | 6630 | NO_LRU_CRAWLER, |
2096 | 6576 | NO_LRU_MAINTAINER, | 6631 | NO_LRU_MAINTAINER, |
2097 | 6577 | NO_DROP_PRIVILEGES, | 6632 | NO_DROP_PRIVILEGES, |
2098 | 6633 | DROP_PRIVILEGES, | ||
2099 | 6578 | #ifdef MEMCACHED_DEBUG | 6634 | #ifdef MEMCACHED_DEBUG |
2100 | 6579 | RELAXED_PRIVILEGES, | 6635 | RELAXED_PRIVILEGES, |
2101 | 6580 | #endif | 6636 | #endif |
2102 | 6581 | #ifdef EXTSTORE | 6637 | #ifdef EXTSTORE |
2103 | 6582 | EXT_PAGE_SIZE, | 6638 | EXT_PAGE_SIZE, |
2104 | 6583 | EXT_PAGE_COUNT, | ||
2105 | 6584 | EXT_WBUF_SIZE, | 6639 | EXT_WBUF_SIZE, |
2106 | 6585 | EXT_THREADS, | 6640 | EXT_THREADS, |
2107 | 6586 | EXT_IO_DEPTH, | 6641 | EXT_IO_DEPTH, |
2108 | @@ -6632,12 +6687,12 @@ int main (int argc, char **argv) { | |||
2109 | 6632 | [NO_LRU_CRAWLER] = "no_lru_crawler", | 6687 | [NO_LRU_CRAWLER] = "no_lru_crawler", |
2110 | 6633 | [NO_LRU_MAINTAINER] = "no_lru_maintainer", | 6688 | [NO_LRU_MAINTAINER] = "no_lru_maintainer", |
2111 | 6634 | [NO_DROP_PRIVILEGES] = "no_drop_privileges", | 6689 | [NO_DROP_PRIVILEGES] = "no_drop_privileges", |
2112 | 6690 | [DROP_PRIVILEGES] = "drop_privileges", | ||
2113 | 6635 | #ifdef MEMCACHED_DEBUG | 6691 | #ifdef MEMCACHED_DEBUG |
2114 | 6636 | [RELAXED_PRIVILEGES] = "relaxed_privileges", | 6692 | [RELAXED_PRIVILEGES] = "relaxed_privileges", |
2115 | 6637 | #endif | 6693 | #endif |
2116 | 6638 | #ifdef EXTSTORE | 6694 | #ifdef EXTSTORE |
2117 | 6639 | [EXT_PAGE_SIZE] = "ext_page_size", | 6695 | [EXT_PAGE_SIZE] = "ext_page_size", |
2118 | 6640 | [EXT_PAGE_COUNT] = "ext_page_count", | ||
2119 | 6641 | [EXT_WBUF_SIZE] = "ext_wbuf_size", | 6696 | [EXT_WBUF_SIZE] = "ext_wbuf_size", |
2120 | 6642 | [EXT_THREADS] = "ext_threads", | 6697 | [EXT_THREADS] = "ext_threads", |
2121 | 6643 | [EXT_IO_DEPTH] = "ext_io_depth", | 6698 | [EXT_IO_DEPTH] = "ext_io_depth", |
2122 | @@ -6677,7 +6732,6 @@ int main (int argc, char **argv) { | |||
2123 | 6677 | settings.ext_drop_under = 0; | 6732 | settings.ext_drop_under = 0; |
2124 | 6678 | settings.slab_automove_freeratio = 0.01; | 6733 | settings.slab_automove_freeratio = 0.01; |
2125 | 6679 | ext_cf.page_size = 1024 * 1024 * 64; | 6734 | ext_cf.page_size = 1024 * 1024 * 64; |
2126 | 6680 | ext_cf.page_count = 64; | ||
2127 | 6681 | ext_cf.wbuf_size = settings.ext_wbuf_size; | 6735 | ext_cf.wbuf_size = settings.ext_wbuf_size; |
2128 | 6682 | ext_cf.io_threadcount = 1; | 6736 | ext_cf.io_threadcount = 1; |
2129 | 6683 | ext_cf.io_depth = 1; | 6737 | ext_cf.io_depth = 1; |
2130 | @@ -7203,16 +7257,6 @@ int main (int argc, char **argv) { | |||
2131 | 7203 | } | 7257 | } |
2132 | 7204 | ext_cf.page_size *= 1024 * 1024; /* megabytes */ | 7258 | ext_cf.page_size *= 1024 * 1024; /* megabytes */ |
2133 | 7205 | break; | 7259 | break; |
2134 | 7206 | case EXT_PAGE_COUNT: | ||
2135 | 7207 | if (subopts_value == NULL) { | ||
2136 | 7208 | fprintf(stderr, "Missing ext_page_count argument\n"); | ||
2137 | 7209 | return 1; | ||
2138 | 7210 | } | ||
2139 | 7211 | if (!safe_strtoul(subopts_value, &ext_cf.page_count)) { | ||
2140 | 7212 | fprintf(stderr, "could not parse argument to ext_page_count\n"); | ||
2141 | 7213 | return 1; | ||
2142 | 7214 | } | ||
2143 | 7215 | break; | ||
2144 | 7216 | case EXT_WBUF_SIZE: | 7260 | case EXT_WBUF_SIZE: |
2145 | 7217 | if (subopts_value == NULL) { | 7261 | if (subopts_value == NULL) { |
2146 | 7218 | fprintf(stderr, "Missing ext_wbuf_size argument\n"); | 7262 | fprintf(stderr, "Missing ext_wbuf_size argument\n"); |
2147 | @@ -7329,7 +7373,20 @@ int main (int argc, char **argv) { | |||
2148 | 7329 | settings.ext_drop_unread = true; | 7373 | settings.ext_drop_unread = true; |
2149 | 7330 | break; | 7374 | break; |
2150 | 7331 | case EXT_PATH: | 7375 | case EXT_PATH: |
2152 | 7332 | storage_file = strdup(subopts_value); | 7376 | if (subopts_value) { |
2153 | 7377 | struct extstore_conf_file *tmp = storage_conf_parse(subopts_value, ext_cf.page_size); | ||
2154 | 7378 | if (tmp == NULL) { | ||
2155 | 7379 | fprintf(stderr, "failed to parse ext_path argument\n"); | ||
2156 | 7380 | return 1; | ||
2157 | 7381 | } | ||
2158 | 7382 | if (storage_file != NULL) { | ||
2159 | 7383 | tmp->next = storage_file; | ||
2160 | 7384 | } | ||
2161 | 7385 | storage_file = tmp; | ||
2162 | 7386 | } else { | ||
2163 | 7387 | fprintf(stderr, "missing argument to ext_path, ie: ext_path=/d/file:5G\n"); | ||
2164 | 7388 | return 1; | ||
2165 | 7389 | } | ||
2166 | 7333 | break; | 7390 | break; |
2167 | 7334 | #endif | 7391 | #endif |
2168 | 7335 | case MODERN: | 7392 | case MODERN: |
2169 | @@ -7351,6 +7408,9 @@ int main (int argc, char **argv) { | |||
2170 | 7351 | case NO_DROP_PRIVILEGES: | 7408 | case NO_DROP_PRIVILEGES: |
2171 | 7352 | settings.drop_privileges = false; | 7409 | settings.drop_privileges = false; |
2172 | 7353 | break; | 7410 | break; |
2173 | 7411 | case DROP_PRIVILEGES: | ||
2174 | 7412 | settings.drop_privileges = true; | ||
2175 | 7413 | break; | ||
2176 | 7354 | #ifdef MEMCACHED_DEBUG | 7414 | #ifdef MEMCACHED_DEBUG |
2177 | 7355 | case RELAXED_PRIVILEGES: | 7415 | case RELAXED_PRIVILEGES: |
2178 | 7356 | settings.relaxed_privileges = true; | 7416 | settings.relaxed_privileges = true; |
2179 | @@ -7536,6 +7596,10 @@ int main (int argc, char **argv) { | |||
2180 | 7536 | fprintf(stderr, "can't find the user %s to switch to\n", username); | 7596 | fprintf(stderr, "can't find the user %s to switch to\n", username); |
2181 | 7537 | exit(EX_NOUSER); | 7597 | exit(EX_NOUSER); |
2182 | 7538 | } | 7598 | } |
2183 | 7599 | if (setgroups(0, NULL) < 0) { | ||
2184 | 7600 | fprintf(stderr, "failed to drop supplementary groups\n"); | ||
2185 | 7601 | exit(EX_OSERR); | ||
2186 | 7602 | } | ||
2187 | 7539 | if (setgid(pw->pw_gid) < 0 || setuid(pw->pw_uid) < 0) { | 7603 | if (setgid(pw->pw_gid) < 0 || setuid(pw->pw_uid) < 0) { |
2188 | 7540 | fprintf(stderr, "failed to assume identity of user %s\n", username); | 7604 | fprintf(stderr, "failed to assume identity of user %s\n", username); |
2189 | 7541 | exit(EX_OSERR); | 7605 | exit(EX_OSERR); |
2190 | @@ -7596,9 +7660,9 @@ int main (int argc, char **argv) { | |||
2191 | 7596 | if (storage_file) { | 7660 | if (storage_file) { |
2192 | 7597 | enum extstore_res eres; | 7661 | enum extstore_res eres; |
2193 | 7598 | if (settings.ext_compact_under == 0) { | 7662 | if (settings.ext_compact_under == 0) { |
2195 | 7599 | settings.ext_compact_under = ext_cf.page_count / 4; | 7663 | settings.ext_compact_under = storage_file->page_count / 4; |
2196 | 7600 | /* Only rescues non-COLD items if below this threshold */ | 7664 | /* Only rescues non-COLD items if below this threshold */ |
2198 | 7601 | settings.ext_drop_under = ext_cf.page_count / 4; | 7665 | settings.ext_drop_under = storage_file->page_count / 4; |
2199 | 7602 | } | 7666 | } |
2200 | 7603 | crc32c_init(); | 7667 | crc32c_init(); |
2201 | 7604 | /* Init free chunks to zero. */ | 7668 | /* Init free chunks to zero. */ |
2202 | @@ -7649,6 +7713,10 @@ int main (int argc, char **argv) { | |||
2203 | 7649 | fprintf(stderr, "Failed to start storage compaction thread\n"); | 7713 | fprintf(stderr, "Failed to start storage compaction thread\n"); |
2204 | 7650 | exit(EXIT_FAILURE); | 7714 | exit(EXIT_FAILURE); |
2205 | 7651 | } | 7715 | } |
2206 | 7716 | if (storage && start_storage_write_thread(storage) != 0) { | ||
2207 | 7717 | fprintf(stderr, "Failed to start storage writer thread\n"); | ||
2208 | 7718 | exit(EXIT_FAILURE); | ||
2209 | 7719 | } | ||
2210 | 7652 | 7720 | ||
2211 | 7653 | if (start_lru_maintainer && start_lru_maintainer_thread(storage) != 0) { | 7721 | if (start_lru_maintainer && start_lru_maintainer_thread(storage) != 0) { |
2212 | 7654 | #else | 7722 | #else |
2213 | diff --git a/memcached.h b/memcached.h | |||
2214 | index 22893f5..a8dd59d 100644 | |||
2215 | --- a/memcached.h | |||
2216 | +++ b/memcached.h | |||
2217 | @@ -18,6 +18,7 @@ | |||
2218 | 18 | #include <pthread.h> | 18 | #include <pthread.h> |
2219 | 19 | #include <unistd.h> | 19 | #include <unistd.h> |
2220 | 20 | #include <assert.h> | 20 | #include <assert.h> |
2221 | 21 | #include <grp.h> | ||
2222 | 21 | 22 | ||
2223 | 22 | #include "itoa_ljust.h" | 23 | #include "itoa_ljust.h" |
2224 | 23 | #include "protocol_binary.h" | 24 | #include "protocol_binary.h" |
2225 | @@ -140,6 +141,17 @@ | |||
2226 | 140 | #define APPEND_NUM_STAT(num, name, fmt, val) \ | 141 | #define APPEND_NUM_STAT(num, name, fmt, val) \ |
2227 | 141 | APPEND_NUM_FMT_STAT("%d:%s", num, name, fmt, val) | 142 | APPEND_NUM_FMT_STAT("%d:%s", num, name, fmt, val) |
2228 | 142 | 143 | ||
2229 | 144 | /** Item client flag conversion */ | ||
2230 | 145 | #define FLAGS_CONV(iar, it, flag) { \ | ||
2231 | 146 | if ((iar)) { \ | ||
2232 | 147 | flag = (uint32_t) strtoul(ITEM_suffix((it)), (char **) NULL, 10); \ | ||
2233 | 148 | } else if ((it)->nsuffix > 0) { \ | ||
2234 | 149 | flag = *((uint32_t *)ITEM_suffix((it))); \ | ||
2235 | 150 | } else { \ | ||
2236 | 151 | flag = 0; \ | ||
2237 | 152 | } \ | ||
2238 | 153 | } | ||
2239 | 154 | |||
2240 | 143 | /** | 155 | /** |
2241 | 144 | * Callback for any function producing stats. | 156 | * Callback for any function producing stats. |
2242 | 145 | * | 157 | * |
2243 | @@ -274,6 +286,8 @@ struct slab_stats { | |||
2244 | 274 | #ifdef EXTSTORE | 286 | #ifdef EXTSTORE |
2245 | 275 | #define EXTSTORE_THREAD_STATS_FIELDS \ | 287 | #define EXTSTORE_THREAD_STATS_FIELDS \ |
2246 | 276 | X(get_extstore) \ | 288 | X(get_extstore) \ |
2247 | 289 | X(get_aborted_extstore) \ | ||
2248 | 290 | X(get_oom_extstore) \ | ||
2249 | 277 | X(recache_from_extstore) \ | 291 | X(recache_from_extstore) \ |
2250 | 278 | X(miss_from_extstore) \ | 292 | X(miss_from_extstore) \ |
2251 | 279 | X(badcrc_from_extstore) | 293 | X(badcrc_from_extstore) |
2252 | @@ -510,6 +524,23 @@ typedef struct _strchunk { | |||
2253 | 510 | uint8_t slabs_clsid; /* Same as above. */ | 524 | uint8_t slabs_clsid; /* Same as above. */ |
2254 | 511 | char data[]; | 525 | char data[]; |
2255 | 512 | } item_chunk; | 526 | } item_chunk; |
2256 | 527 | |||
2257 | 528 | #ifdef NEED_ALIGN | ||
2258 | 529 | static inline char *ITEM_schunk(item *it) { | ||
2259 | 530 | int offset = it->nkey + 1 + it->nsuffix | ||
2260 | 531 | + ((it->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0); | ||
2261 | 532 | int remain = offset % 8; | ||
2262 | 533 | if (remain != 0) { | ||
2263 | 534 | offset += 8 - remain; | ||
2264 | 535 | } | ||
2265 | 536 | return ((char *) &(it->data)) + offset; | ||
2266 | 537 | } | ||
2267 | 538 | #else | ||
2268 | 539 | #define ITEM_schunk(item) ((char*) &((item)->data) + (item)->nkey + 1 \ | ||
2269 | 540 | + (item)->nsuffix \ | ||
2270 | 541 | + (((item)->it_flags & ITEM_CAS) ? sizeof(uint64_t) : 0)) | ||
2271 | 542 | #endif | ||
2272 | 543 | |||
2273 | 513 | #ifdef EXTSTORE | 544 | #ifdef EXTSTORE |
2274 | 514 | typedef struct { | 545 | typedef struct { |
2275 | 515 | unsigned int page_version; /* from IO header */ | 546 | unsigned int page_version; /* from IO header */ |
2276 | @@ -545,7 +576,7 @@ typedef struct _io_wrap { | |||
2277 | 545 | unsigned int iovec_data; /* specific index of data iovec */ | 576 | unsigned int iovec_data; /* specific index of data iovec */ |
2278 | 546 | bool miss; /* signal a miss to unlink hdr_it */ | 577 | bool miss; /* signal a miss to unlink hdr_it */ |
2279 | 547 | bool badcrc; /* signal a crc failure */ | 578 | bool badcrc; /* signal a crc failure */ |
2281 | 548 | bool active; // FIXME: canary for test. remove | 579 | bool active; /* tells if IO was dispatched or not */ |
2282 | 549 | } io_wrap; | 580 | } io_wrap; |
2283 | 550 | #endif | 581 | #endif |
2284 | 551 | /** | 582 | /** |
2285 | @@ -554,6 +585,7 @@ typedef struct _io_wrap { | |||
2286 | 554 | struct conn { | 585 | struct conn { |
2287 | 555 | int sfd; | 586 | int sfd; |
2288 | 556 | sasl_conn_t *sasl_conn; | 587 | sasl_conn_t *sasl_conn; |
2289 | 588 | bool sasl_started; | ||
2290 | 557 | bool authenticated; | 589 | bool authenticated; |
2291 | 558 | enum conn_states state; | 590 | enum conn_states state; |
2292 | 559 | enum bin_substates substate; | 591 | enum bin_substates substate; |
2293 | @@ -713,7 +745,7 @@ void sidethread_conn_close(conn *c); | |||
2294 | 713 | 745 | ||
2295 | 714 | /* Lock wrappers for cache functions that are called from main loop. */ | 746 | /* Lock wrappers for cache functions that are called from main loop. */ |
2296 | 715 | enum delta_result_type add_delta(conn *c, const char *key, | 747 | enum delta_result_type add_delta(conn *c, const char *key, |
2298 | 716 | const size_t nkey, const int incr, | 748 | const size_t nkey, bool incr, |
2299 | 717 | const int64_t delta, char *buf, | 749 | const int64_t delta, char *buf, |
2300 | 718 | uint64_t *cas); | 750 | uint64_t *cas); |
2301 | 719 | void accept_new_conns(const bool do_accept); | 751 | void accept_new_conns(const bool do_accept); |
2302 | diff --git a/memcached.spec b/memcached.spec | |||
2303 | index 697c38f..295b7d9 100644 | |||
2304 | --- a/memcached.spec | |||
2305 | +++ b/memcached.spec | |||
2306 | @@ -17,7 +17,7 @@ BuildRequires: systemd-units | |||
2307 | 17 | %endif | 17 | %endif |
2308 | 18 | 18 | ||
2309 | 19 | Name: memcached | 19 | Name: memcached |
2311 | 20 | Version: 1.5.6 | 20 | Version: 1.5.10 |
2312 | 21 | Release: 1%{?dist} | 21 | Release: 1%{?dist} |
2313 | 22 | Summary: High Performance, Distributed Memory Object Cache | 22 | Summary: High Performance, Distributed Memory Object Cache |
2314 | 23 | 23 | ||
2315 | @@ -79,9 +79,9 @@ install -Dp -m0755 scripts/memcached.service %{buildroot}%{_unitdir}/%{name}.ser | |||
2316 | 79 | install -Dp -m0755 scripts/memcached@.service %{buildroot}%{_unitdir}/%{name}@.service | 79 | install -Dp -m0755 scripts/memcached@.service %{buildroot}%{_unitdir}/%{name}@.service |
2317 | 80 | 80 | ||
2318 | 81 | if [ %{safer_systemd} -gt 0 ]; then | 81 | if [ %{safer_systemd} -gt 0 ]; then |
2320 | 82 | sed -e -i 's/^##safer##//g' %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service | 82 | sed -e 's/^##safer##//g' -i %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service |
2321 | 83 | else | 83 | else |
2323 | 84 | sed -e -i 's/^##safer##/#/g' %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service | 84 | sed -e 's/^##safer##/#/g' -i %{buildroot}%{_unitdir}/%{name}.service %{buildroot}%{_unitdir}/%{name}@.service |
2324 | 85 | fi | 85 | fi |
2325 | 86 | %else | 86 | %else |
2326 | 87 | install -Dp -m0755 scripts/memcached.sysv %{buildroot}%{_initrddir}/%{name} | 87 | install -Dp -m0755 scripts/memcached.sysv %{buildroot}%{_initrddir}/%{name} |
2327 | @@ -187,7 +187,7 @@ exit 0 | |||
2328 | 187 | - above suggestions from Bernard Johnson | 187 | - above suggestions from Bernard Johnson |
2329 | 188 | 188 | ||
2330 | 189 | * Mon May 7 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-2 | 189 | * Mon May 7 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-2 |
2332 | 190 | - Tidyness improvements suggested by Ruben Kerkhof in bugzilla #238994 | 190 | - Tidiness improvements suggested by Ruben Kerkhof in bugzilla #238994 |
2333 | 191 | 191 | ||
2334 | 192 | * Fri May 4 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-1 | 192 | * Fri May 4 2007 Paul Lindner <lindner@inuus.com> - 1.2.2-1 |
2335 | 193 | - Initial spec file created via rpmdev-newspec | 193 | - Initial spec file created via rpmdev-newspec |
2336 | diff --git a/sasl_defs.c b/sasl_defs.c | |||
2337 | index 93a7b8b..c60d1bf 100644 | |||
2338 | --- a/sasl_defs.c | |||
2339 | +++ b/sasl_defs.c | |||
2340 | @@ -7,7 +7,7 @@ | |||
2341 | 7 | 7 | ||
2342 | 8 | char my_sasl_hostname[1025]; | 8 | char my_sasl_hostname[1025]; |
2343 | 9 | 9 | ||
2345 | 10 | #ifdef HAVE_SASL_CB_GETCONF | 10 | #if defined(HAVE_SASL_CB_GETCONF) || defined(HAVE_SASL_CB_GETCONFPATH) |
2346 | 11 | /* The locations we may search for a SASL config file if the user didn't | 11 | /* The locations we may search for a SASL config file if the user didn't |
2347 | 12 | * specify one in the environment variable SASL_CONF_PATH | 12 | * specify one in the environment variable SASL_CONF_PATH |
2348 | 13 | */ | 13 | */ |
2349 | @@ -82,7 +82,7 @@ static int sasl_server_userdb_checkpass(sasl_conn_t *conn, | |||
2350 | 82 | } | 82 | } |
2351 | 83 | #endif | 83 | #endif |
2352 | 84 | 84 | ||
2354 | 85 | #ifdef HAVE_SASL_CB_GETCONF | 85 | #if defined(HAVE_SASL_CB_GETCONF) || defined(HAVE_SASL_CB_GETCONFPATH) |
2355 | 86 | static int sasl_getconf(void *context, const char **path) | 86 | static int sasl_getconf(void *context, const char **path) |
2356 | 87 | { | 87 | { |
2357 | 88 | *path = getenv("SASL_CONF_PATH"); | 88 | *path = getenv("SASL_CONF_PATH"); |
2358 | @@ -152,6 +152,10 @@ static sasl_callback_t sasl_callbacks[] = { | |||
2359 | 152 | 152 | ||
2360 | 153 | #ifdef HAVE_SASL_CB_GETCONF | 153 | #ifdef HAVE_SASL_CB_GETCONF |
2361 | 154 | { SASL_CB_GETCONF, sasl_getconf, NULL }, | 154 | { SASL_CB_GETCONF, sasl_getconf, NULL }, |
2362 | 155 | #else | ||
2363 | 156 | #ifdef HAVE_SASL_CB_GETCONFPATH | ||
2364 | 157 | { SASL_CB_GETCONFPATH, (sasl_callback_ft)sasl_getconf, NULL }, | ||
2365 | 158 | #endif | ||
2366 | 155 | #endif | 159 | #endif |
2367 | 156 | 160 | ||
2368 | 157 | { SASL_CB_LIST_END, NULL, NULL } | 161 | { SASL_CB_LIST_END, NULL, NULL } |
2369 | diff --git a/scripts/memcached-automove b/scripts/memcached-automove | |||
2370 | index f881129..4a3d3dd 100755 | |||
2371 | --- a/scripts/memcached-automove | |||
2372 | +++ b/scripts/memcached-automove | |||
2373 | @@ -54,7 +54,7 @@ def determine_move(history, diffs, totals): | |||
2374 | 54 | - use age as average over window. smooths over items falling out of WARM. | 54 | - use age as average over window. smooths over items falling out of WARM. |
2375 | 55 | also gives a little weight: if still evicting, a few more pages than | 55 | also gives a little weight: if still evicting, a few more pages than |
2376 | 56 | necessary may be moved, pulling the classes closer together. Hopefully | 56 | necessary may be moved, pulling the classes closer together. Hopefully |
2378 | 57 | avoidnig ping-ponging. | 57 | avoiding ping-ponging. |
2379 | 58 | """ | 58 | """ |
2380 | 59 | # rotate windows | 59 | # rotate windows |
2381 | 60 | history['w'].append({}) | 60 | history['w'].append({}) |
2382 | diff --git a/scripts/memcached-automove-extstore b/scripts/memcached-automove-extstore | |||
2383 | index db66d8c..2bb0e35 100755 | |||
2384 | --- a/scripts/memcached-automove-extstore | |||
2385 | +++ b/scripts/memcached-automove-extstore | |||
2386 | @@ -64,7 +64,7 @@ def determine_move(history, stats, diffs, memfree): | |||
2387 | 64 | - if global pool is below minimum remove pages from oldest large class. | 64 | - if global pool is below minimum remove pages from oldest large class. |
2388 | 65 | - if global pool is above maximum, move pages to youngest large class. | 65 | - if global pool is above maximum, move pages to youngest large class. |
2389 | 66 | - extstore manages a desired number of free chunks in each slab class. | 66 | - extstore manages a desired number of free chunks in each slab class. |
2391 | 67 | - autmover adjusts above limits once per minute based on current sizes. | 67 | - automover adjusts above limits once per minute based on current sizes. |
2392 | 68 | - if youngest is below the age ratio limit of oldest, move a page to it. | 68 | - if youngest is below the age ratio limit of oldest, move a page to it. |
2393 | 69 | """ | 69 | """ |
2394 | 70 | # rotate windows | 70 | # rotate windows |
2395 | @@ -85,7 +85,6 @@ def determine_move(history, stats, diffs, memfree): | |||
2396 | 85 | print("global pool: [{}]".format(stats['slab_global_page_pool'])) | 85 | print("global pool: [{}]".format(stats['slab_global_page_pool'])) |
2397 | 86 | 86 | ||
2398 | 87 | pool_low = window_key_check(history, 'slab_pool_low') | 87 | pool_low = window_key_check(history, 'slab_pool_low') |
2399 | 88 | pool_high = window_key_check(history, 'slab_pool_high') | ||
2400 | 89 | for sid, slab in diffs.items(): | 88 | for sid, slab in diffs.items(): |
2401 | 90 | small_slab = False | 89 | small_slab = False |
2402 | 91 | free_enough = False | 90 | free_enough = False |
2403 | diff --git a/scripts/memcached-tool b/scripts/memcached-tool | |||
2404 | index 79f3770..cab36a5 100755 | |||
2405 | --- a/scripts/memcached-tool | |||
2406 | +++ b/scripts/memcached-tool | |||
2407 | @@ -20,6 +20,7 @@ use IO::Socket::INET; | |||
2408 | 20 | my $addr = shift; | 20 | my $addr = shift; |
2409 | 21 | my $mode = shift || "display"; | 21 | my $mode = shift || "display"; |
2410 | 22 | my ($from, $to); | 22 | my ($from, $to); |
2411 | 23 | my $limit; | ||
2412 | 23 | 24 | ||
2413 | 24 | if ($mode eq "display") { | 25 | if ($mode eq "display") { |
2414 | 25 | undef $mode if @ARGV; | 26 | undef $mode if @ARGV; |
2415 | @@ -30,7 +31,11 @@ if ($mode eq "display") { | |||
2416 | 30 | undef $mode if $to < 6 || $to > 17; | 31 | undef $mode if $to < 6 || $to > 17; |
2417 | 31 | print STDERR "ERROR: parameters out of range\n\n" unless $mode; | 32 | print STDERR "ERROR: parameters out of range\n\n" unless $mode; |
2418 | 32 | } elsif ($mode eq 'dump') { | 33 | } elsif ($mode eq 'dump') { |
2420 | 33 | ; | 34 | if (@ARGV) { |
2421 | 35 | $limit = shift; | ||
2422 | 36 | undef $mode if $limit < 1; | ||
2423 | 37 | print STDERR "ERROR: invalid limit (should be a positive number)\n\n" unless $mode; | ||
2424 | 38 | } | ||
2425 | 34 | } elsif ($mode eq 'stats') { | 39 | } elsif ($mode eq 'stats') { |
2426 | 35 | ; | 40 | ; |
2427 | 36 | } elsif ($mode eq 'settings') { | 41 | } elsif ($mode eq 'settings') { |
2428 | @@ -45,12 +50,12 @@ undef $mode if @ARGV; | |||
2429 | 45 | 50 | ||
2430 | 46 | die | 51 | die |
2431 | 47 | "Usage: memcached-tool <host[:port] | /path/to/socket> [mode]\n | 52 | "Usage: memcached-tool <host[:port] | /path/to/socket> [mode]\n |
2438 | 48 | memcached-tool 10.0.0.5:11211 display # shows slabs | 53 | memcached-tool 10.0.0.5:11211 display # shows slabs |
2439 | 49 | memcached-tool 10.0.0.5:11211 # same. (default is display) | 54 | memcached-tool 10.0.0.5:11211 # same. (default is display) |
2440 | 50 | memcached-tool 10.0.0.5:11211 stats # shows general stats | 55 | memcached-tool 10.0.0.5:11211 stats # shows general stats |
2441 | 51 | memcached-tool 10.0.0.5:11211 settings # shows settings stats | 56 | memcached-tool 10.0.0.5:11211 settings # shows settings stats |
2442 | 52 | memcached-tool 10.0.0.5:11211 sizes # shows sizes stats | 57 | memcached-tool 10.0.0.5:11211 sizes # shows sizes stats |
2443 | 53 | memcached-tool 10.0.0.5:11211 dump # dumps keys and values | 58 | memcached-tool 10.0.0.5:11211 dump [limit] # dumps keys and values |
2444 | 54 | 59 | ||
2445 | 55 | WARNING! sizes is a development command. | 60 | WARNING! sizes is a development command. |
2446 | 56 | As of 1.4 it is still the only command which will lock your memcached instance for some time. | 61 | As of 1.4 it is still the only command which will lock your memcached instance for some time. |
2447 | @@ -60,65 +65,74 @@ or at least speed it up. | |||
2448 | 60 | " unless $addr && $mode; | 65 | " unless $addr && $mode; |
2449 | 61 | 66 | ||
2450 | 62 | 67 | ||
2459 | 63 | my $sock; | 68 | sub server_connect { |
2460 | 64 | if ($addr =~ m:/:) { | 69 | my $sock; |
2461 | 65 | $sock = IO::Socket::UNIX->new( | 70 | if ($addr =~ m:/:) { |
2462 | 66 | Peer => $addr, | 71 | $sock = IO::Socket::UNIX->new( |
2463 | 67 | ); | 72 | Peer => $addr, |
2464 | 68 | } | 73 | ); |
2465 | 69 | else { | 74 | } |
2466 | 70 | $addr .= ':11211' unless $addr =~ /:\d+$/; | 75 | else { |
2467 | 76 | $addr .= ':11211' unless $addr =~ /:\d+$/; | ||
2468 | 71 | 77 | ||
2473 | 72 | $sock = IO::Socket::INET->new( | 78 | $sock = IO::Socket::INET->new( |
2474 | 73 | PeerAddr => $addr, | 79 | PeerAddr => $addr, |
2475 | 74 | Proto => 'tcp', | 80 | Proto => 'tcp', |
2476 | 75 | ); | 81 | ); |
2477 | 82 | } | ||
2478 | 83 | die "Couldn't connect to $addr\n" unless $sock; | ||
2479 | 84 | return $sock; | ||
2480 | 76 | } | 85 | } |
2481 | 77 | die "Couldn't connect to $addr\n" unless $sock; | ||
2482 | 78 | 86 | ||
2488 | 79 | if ($mode eq 'dump') { | 87 | my $sock = server_connect(); |
2484 | 80 | my %items; | ||
2485 | 81 | my $totalitems; | ||
2486 | 82 | |||
2487 | 83 | print $sock "stats items\r\n"; | ||
2489 | 84 | 88 | ||
2490 | 89 | if ($mode eq 'dump') { | ||
2491 | 90 | print STDERR "Dumping memcache contents"; | ||
2492 | 91 | print STDERR " (limiting to $limit keys)" unless !$limit; | ||
2493 | 92 | print STDERR "\n"; | ||
2494 | 93 | print $sock "lru_crawler metadump all\r\n"; | ||
2495 | 94 | my %keyexp; | ||
2496 | 95 | my $keycount = 0; | ||
2497 | 85 | while (<$sock>) { | 96 | while (<$sock>) { |
2518 | 86 | last if /^END/; | 97 | last if /^END/ or ($limit and $keycount == $limit); |
2519 | 87 | if (/^STAT items:(\d*):number (\d*)/) { | 98 | # return format looks like this |
2520 | 88 | $items{$1} = $2; | 99 | # key=foo exp=2147483647 la=1521046038 cas=717111 fetch=no cls=13 size=1232 |
2521 | 89 | $totalitems += $2; | 100 | if (/^key=(\S+) exp=(-?\d+) .*/) { |
2522 | 90 | } | 101 | my $k = $1; |
2523 | 91 | } | 102 | $k =~ s/%(.{2})/chr hex $1/eg; |
2524 | 92 | print STDERR "Dumping memcache contents\n"; | 103 | |
2525 | 93 | print STDERR " Number of buckets: " . scalar(keys(%items)) . "\n"; | 104 | if ($2 == -1) { |
2526 | 94 | print STDERR " Number of items : $totalitems\n"; | 105 | $keyexp{$k} = 0; |
2527 | 95 | 106 | } else { | |
2528 | 96 | foreach my $bucket (sort(keys(%items))) { | 107 | $keyexp{$k} = $2; |
2509 | 97 | print STDERR "Dumping bucket $bucket - " . $items{$bucket} . " total items\n"; | ||
2510 | 98 | print $sock "stats cachedump $bucket $items{$bucket}\r\n"; | ||
2511 | 99 | my %keyexp; | ||
2512 | 100 | while (<$sock>) { | ||
2513 | 101 | last if /^END/; | ||
2514 | 102 | # return format looks like this | ||
2515 | 103 | # ITEM foo [6 b; 1176415152 s] | ||
2516 | 104 | if (/^ITEM (\S+) \[.* (\d+) s\]/) { | ||
2517 | 105 | $keyexp{$1} = $2; | ||
2529 | 106 | } | 108 | } |
2530 | 107 | } | 109 | } |
2531 | 110 | $keycount++; | ||
2532 | 111 | } | ||
2533 | 108 | 112 | ||
2547 | 109 | foreach my $k (keys(%keyexp)) { | 113 | if ($limit) { |
2548 | 110 | print $sock "get $k\r\n"; | 114 | # Need to reopen the connection here to stop the metadump in |
2549 | 111 | my $response = <$sock>; | 115 | # case the key limit was reached. |
2550 | 112 | if ($response =~ /VALUE (\S+) (\d+) (\d+)/) { | 116 | # |
2551 | 113 | my $flags = $2; | 117 | # XXX: Once a limit on # of keys returned is introduced in |
2552 | 114 | my $len = $3; | 118 | # `lru_crawler metadump`, this should be removed and the proper |
2553 | 115 | my $val; | 119 | # parameter passed in the query above. |
2554 | 116 | read $sock, $val, $len; | 120 | close($sock); |
2555 | 117 | print "add $k $flags $keyexp{$k} $len\r\n$val\r\n"; | 121 | $sock = server_connect(); |
2556 | 118 | # get the END | 122 | } |
2557 | 119 | $_ = <$sock>; | 123 | |
2558 | 120 | $_ = <$sock>; | 124 | foreach my $k (keys(%keyexp)) { |
2559 | 121 | } | 125 | print $sock "get $k\r\n"; |
2560 | 126 | my $response = <$sock>; | ||
2561 | 127 | if ($response =~ /VALUE (\S+) (\d+) (\d+)/) { | ||
2562 | 128 | my $flags = $2; | ||
2563 | 129 | my $len = $3; | ||
2564 | 130 | my $val; | ||
2565 | 131 | read $sock, $val, $len; | ||
2566 | 132 | print "add $k $flags $keyexp{$k} $len\r\n$val\r\n"; | ||
2567 | 133 | # get the END | ||
2568 | 134 | $_ = <$sock>; | ||
2569 | 135 | $_ = <$sock>; | ||
2570 | 122 | } | 136 | } |
2571 | 123 | } | 137 | } |
2572 | 124 | exit; | 138 | exit; |
2573 | diff --git a/scripts/memcached-tool.1 b/scripts/memcached-tool.1 | |||
2574 | index 6bb021b..a863bd4 100644 | |||
2575 | --- a/scripts/memcached-tool.1 | |||
2576 | +++ b/scripts/memcached-tool.1 | |||
2577 | @@ -57,9 +57,12 @@ Number of times the underlying slab class was unable to store a new item. | |||
2578 | 57 | Print general-purpose statistics of the daemon. Each line contains the name of | 57 | Print general-purpose statistics of the daemon. Each line contains the name of |
2579 | 58 | the statistic and its value. | 58 | the statistic and its value. |
2580 | 59 | .TP | 59 | .TP |
2582 | 60 | .B dump | 60 | .B dump [limit] |
2583 | 61 | Make a partial dump of the cache written in the add statements of the | 61 | Make a partial dump of the cache written in the add statements of the |
2585 | 62 | memcached protocol. | 62 | memcached protocol. If |
2586 | 63 | .B limit | ||
2587 | 64 | is given and is a strictly positive | ||
2588 | 65 | integer, then the dump is limited to that number of items. | ||
2589 | 63 | 66 | ||
2590 | 64 | .SH SEE ALSO | 67 | .SH SEE ALSO |
2591 | 65 | .BR memcached (1), | 68 | .BR memcached (1), |
2592 | diff --git a/slabs.c b/slabs.c | |||
2593 | index 200d575..2bcc350 100644 | |||
2594 | --- a/slabs.c | |||
2595 | +++ b/slabs.c | |||
2596 | @@ -8,6 +8,7 @@ | |||
2597 | 8 | * memcached protocol. | 8 | * memcached protocol. |
2598 | 9 | */ | 9 | */ |
2599 | 10 | #include "memcached.h" | 10 | #include "memcached.h" |
2600 | 11 | #include <sys/mman.h> | ||
2601 | 11 | #include <sys/stat.h> | 12 | #include <sys/stat.h> |
2602 | 12 | #include <sys/socket.h> | 13 | #include <sys/socket.h> |
2603 | 13 | #include <sys/resource.h> | 14 | #include <sys/resource.h> |
2604 | @@ -98,6 +99,57 @@ unsigned int slabs_clsid(const size_t size) { | |||
2605 | 98 | return res; | 99 | return res; |
2606 | 99 | } | 100 | } |
2607 | 100 | 101 | ||
2608 | 102 | #if defined(__linux__) && defined(MADV_HUGEPAGE) | ||
2609 | 103 | /* Function split out for better error path handling */ | ||
2610 | 104 | static void * alloc_large_chunk_linux(const size_t limit) | ||
2611 | 105 | { | ||
2612 | 106 | size_t pagesize = 0; | ||
2613 | 107 | void *ptr = NULL; | ||
2614 | 108 | FILE *fp; | ||
2615 | 109 | int ret; | ||
2616 | 110 | |||
2617 | 111 | /* Get the size of huge pages */ | ||
2618 | 112 | fp = fopen("/proc/meminfo", "r"); | ||
2619 | 113 | if (fp != NULL) { | ||
2620 | 114 | char buf[64]; | ||
2621 | 115 | |||
2622 | 116 | while ((fgets(buf, sizeof(buf), fp))) | ||
2623 | 117 | if (!strncmp(buf, "Hugepagesize:", 13)) { | ||
2624 | 118 | ret = sscanf(buf + 13, "%zu\n", &pagesize); | ||
2625 | 119 | |||
2626 | 120 | /* meminfo huge page size is in KiBs */ | ||
2627 | 121 | pagesize <<= 10; | ||
2628 | 122 | } | ||
2629 | 123 | fclose(fp); | ||
2630 | 124 | } | ||
2631 | 125 | |||
2632 | 126 | if (!pagesize) { | ||
2633 | 127 | fprintf(stderr, "Failed to get supported huge page size\n"); | ||
2634 | 128 | return NULL; | ||
2635 | 129 | } | ||
2636 | 130 | |||
2637 | 131 | if (settings.verbose > 1) | ||
2638 | 132 | fprintf(stderr, "huge page size: %zu\n", pagesize); | ||
2639 | 133 | |||
2640 | 134 | /* This works because glibc simply uses mmap when the alignment is | ||
2641 | 135 | * above a certain limit. */ | ||
2642 | 136 | ret = posix_memalign(&ptr, pagesize, limit); | ||
2643 | 137 | if (ret != 0) { | ||
2644 | 138 | fprintf(stderr, "Failed to get aligned memory chunk: %d\n", ret); | ||
2645 | 139 | return NULL; | ||
2646 | 140 | } | ||
2647 | 141 | |||
2648 | 142 | ret = madvise(ptr, limit, MADV_HUGEPAGE); | ||
2649 | 143 | if (ret < 0) { | ||
2650 | 144 | fprintf(stderr, "Failed to set transparent hugepage hint: %d\n", ret); | ||
2651 | 145 | free(ptr); | ||
2652 | 146 | ptr = NULL; | ||
2653 | 147 | } | ||
2654 | 148 | |||
2655 | 149 | return ptr; | ||
2656 | 150 | } | ||
2657 | 151 | #endif | ||
2658 | 152 | |||
2659 | 101 | /** | 153 | /** |
2660 | 102 | * Determines the chunk sizes and initializes the slab class descriptors | 154 | * Determines the chunk sizes and initializes the slab class descriptors |
2661 | 103 | * accordingly. | 155 | * accordingly. |
2662 | @@ -106,11 +158,24 @@ void slabs_init(const size_t limit, const double factor, const bool prealloc, co | |||
2663 | 106 | int i = POWER_SMALLEST - 1; | 158 | int i = POWER_SMALLEST - 1; |
2664 | 107 | unsigned int size = sizeof(item) + settings.chunk_size; | 159 | unsigned int size = sizeof(item) + settings.chunk_size; |
2665 | 108 | 160 | ||
2666 | 161 | /* Some platforms use runtime transparent hugepages. If for any reason | ||
2667 | 162 | * the initial allocation fails, the required settings do not persist | ||
2668 | 163 | * for remaining allocations. As such it makes little sense to do slab | ||
2669 | 164 | * preallocation. */ | ||
2670 | 165 | bool __attribute__ ((unused)) do_slab_prealloc = false; | ||
2671 | 166 | |||
2672 | 109 | mem_limit = limit; | 167 | mem_limit = limit; |
2673 | 110 | 168 | ||
2674 | 111 | if (prealloc) { | 169 | if (prealloc) { |
2675 | 170 | #if defined(__linux__) && defined(MADV_HUGEPAGE) | ||
2676 | 171 | mem_base = alloc_large_chunk_linux(mem_limit); | ||
2677 | 172 | if (mem_base) | ||
2678 | 173 | do_slab_prealloc = true; | ||
2679 | 174 | #else | ||
2680 | 112 | /* Allocate everything in a big chunk with malloc */ | 175 | /* Allocate everything in a big chunk with malloc */ |
2681 | 113 | mem_base = malloc(mem_limit); | 176 | mem_base = malloc(mem_limit); |
2682 | 177 | do_slab_prealloc = true; | ||
2683 | 178 | #endif | ||
2684 | 114 | if (mem_base != NULL) { | 179 | if (mem_base != NULL) { |
2685 | 115 | mem_current = mem_base; | 180 | mem_current = mem_base; |
2686 | 116 | mem_avail = mem_limit; | 181 | mem_avail = mem_limit; |
2687 | @@ -161,7 +226,7 @@ void slabs_init(const size_t limit, const double factor, const bool prealloc, co | |||
2688 | 161 | 226 | ||
2689 | 162 | } | 227 | } |
2690 | 163 | 228 | ||
2692 | 164 | if (prealloc) { | 229 | if (prealloc && do_slab_prealloc) { |
2693 | 165 | slabs_preallocate(power_largest); | 230 | slabs_preallocate(power_largest); |
2694 | 166 | } | 231 | } |
2695 | 167 | } | 232 | } |
2696 | @@ -315,7 +380,7 @@ static void *do_slabs_alloc(const size_t size, unsigned int id, uint64_t *total_ | |||
2697 | 315 | } | 380 | } |
2698 | 316 | 381 | ||
2699 | 317 | static void do_slabs_free_chunked(item *it, const size_t size) { | 382 | static void do_slabs_free_chunked(item *it, const size_t size) { |
2701 | 318 | item_chunk *chunk = (item_chunk *) ITEM_data(it); | 383 | item_chunk *chunk = (item_chunk *) ITEM_schunk(it); |
2702 | 319 | slabclass_t *p; | 384 | slabclass_t *p; |
2703 | 320 | 385 | ||
2704 | 321 | it->it_flags = ITEM_SLABBED; | 386 | it->it_flags = ITEM_SLABBED; |
2705 | @@ -339,7 +404,15 @@ static void do_slabs_free_chunked(item *it, const size_t size) { | |||
2706 | 339 | p->slots = it; | 404 | p->slots = it; |
2707 | 340 | p->sl_curr++; | 405 | p->sl_curr++; |
2708 | 341 | // TODO: macro | 406 | // TODO: macro |
2709 | 407 | #ifdef NEED_ALIGN | ||
2710 | 408 | int total = it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk); | ||
2711 | 409 | if (total % 8 != 0) { | ||
2712 | 410 | total += 8 - (total % 8); | ||
2713 | 411 | } | ||
2714 | 412 | p->requested -= total; | ||
2715 | 413 | #else | ||
2716 | 342 | p->requested -= it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk); | 414 | p->requested -= it->nkey + 1 + it->nsuffix + sizeof(item) + sizeof(item_chunk); |
2717 | 415 | #endif | ||
2718 | 343 | if (settings.use_cas) { | 416 | if (settings.use_cas) { |
2719 | 344 | p->requested -= sizeof(uint64_t); | 417 | p->requested -= sizeof(uint64_t); |
2720 | 345 | } | 418 | } |
2721 | @@ -948,7 +1021,7 @@ static int slab_rebalance_move(void) { | |||
2722 | 948 | do_item_replace(it, new_it, hv); | 1021 | do_item_replace(it, new_it, hv); |
2723 | 949 | /* Need to walk the chunks and repoint head */ | 1022 | /* Need to walk the chunks and repoint head */ |
2724 | 950 | if (new_it->it_flags & ITEM_CHUNKED) { | 1023 | if (new_it->it_flags & ITEM_CHUNKED) { |
2726 | 951 | item_chunk *fch = (item_chunk *) ITEM_data(new_it); | 1024 | item_chunk *fch = (item_chunk *) ITEM_schunk(new_it); |
2727 | 952 | fch->next->prev = fch; | 1025 | fch->next->prev = fch; |
2728 | 953 | while (fch) { | 1026 | while (fch) { |
2729 | 954 | fch->head = new_it; | 1027 | fch->head = new_it; |
2730 | diff --git a/storage.c b/storage.c | |||
2731 | index 6e61cd8..45554cf 100644 | |||
2732 | --- a/storage.c | |||
2733 | +++ b/storage.c | |||
2734 | @@ -6,27 +6,18 @@ | |||
2735 | 6 | #include <stdlib.h> | 6 | #include <stdlib.h> |
2736 | 7 | #include <string.h> | 7 | #include <string.h> |
2737 | 8 | #include <limits.h> | 8 | #include <limits.h> |
2738 | 9 | #include <ctype.h> | ||
2739 | 9 | 10 | ||
2740 | 10 | #define PAGE_BUCKET_DEFAULT 0 | 11 | #define PAGE_BUCKET_DEFAULT 0 |
2741 | 11 | #define PAGE_BUCKET_COMPACT 1 | 12 | #define PAGE_BUCKET_COMPACT 1 |
2742 | 12 | #define PAGE_BUCKET_CHUNKED 2 | 13 | #define PAGE_BUCKET_CHUNKED 2 |
2743 | 13 | #define PAGE_BUCKET_LOWTTL 3 | 14 | #define PAGE_BUCKET_LOWTTL 3 |
2744 | 14 | 15 | ||
2747 | 15 | int lru_maintainer_store(void *storage, const int clsid) { | 16 | /*** WRITE FLUSH THREAD ***/ |
2748 | 16 | //int i; | 17 | |
2749 | 18 | static int storage_write(void *storage, const int clsid, const int item_age) { | ||
2750 | 17 | int did_moves = 0; | 19 | int did_moves = 0; |
2751 | 18 | int item_age = settings.ext_item_age; | ||
2752 | 19 | bool mem_limit_reached = false; | ||
2753 | 20 | unsigned int chunks_free; | ||
2754 | 21 | struct lru_pull_tail_return it_info; | 20 | struct lru_pull_tail_return it_info; |
2755 | 22 | // FIXME: need to directly ask the slabber how big a class is | ||
2756 | 23 | if (slabs_clsid(settings.ext_item_size) > clsid) | ||
2757 | 24 | return 0; | ||
2758 | 25 | chunks_free = slabs_available_chunks(clsid, &mem_limit_reached, | ||
2759 | 26 | NULL, NULL); | ||
2760 | 27 | // if we are low on chunks and no spare, push out early. | ||
2761 | 28 | if (chunks_free < settings.ext_free_memchunks[clsid] && mem_limit_reached) | ||
2762 | 29 | item_age = 0; | ||
2763 | 30 | 21 | ||
2764 | 31 | it_info.it = NULL; | 22 | it_info.it = NULL; |
2765 | 32 | lru_pull_tail(clsid, COLD_LRU, 0, LRU_PULL_RETURN_ITEM, 0, &it_info); | 23 | lru_pull_tail(clsid, COLD_LRU, 0, LRU_PULL_RETURN_ITEM, 0, &it_info); |
2766 | @@ -42,14 +33,7 @@ int lru_maintainer_store(void *storage, const int clsid) { | |||
2767 | 42 | uint32_t flags; | 33 | uint32_t flags; |
2768 | 43 | if ((it->it_flags & ITEM_HDR) == 0 && | 34 | if ((it->it_flags & ITEM_HDR) == 0 && |
2769 | 44 | (item_age == 0 || current_time - it->time > item_age)) { | 35 | (item_age == 0 || current_time - it->time > item_age)) { |
2778 | 45 | // FIXME: flag conversion again | 36 | FLAGS_CONV(settings.inline_ascii_response, it, flags); |
2771 | 46 | if (settings.inline_ascii_response) { | ||
2772 | 47 | flags = (uint32_t) strtoul(ITEM_suffix(it), (char **) NULL, 10); | ||
2773 | 48 | } else if (it->nsuffix > 0) { | ||
2774 | 49 | flags = *((uint32_t *)ITEM_suffix(it)); | ||
2775 | 50 | } else { | ||
2776 | 51 | flags = 0; | ||
2777 | 52 | } | ||
2779 | 53 | item *hdr_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, sizeof(item_hdr)); | 37 | item *hdr_it = do_item_alloc(ITEM_key(it), it->nkey, flags, it->exptime, sizeof(item_hdr)); |
2780 | 54 | /* Run the storage write understanding the start of the item is dirty. | 38 | /* Run the storage write understanding the start of the item is dirty. |
2781 | 55 | * We will fill it (time/exptime/etc) from the header item on read. | 39 | * We will fill it (time/exptime/etc) from the header item on read. |
2782 | @@ -57,7 +41,7 @@ int lru_maintainer_store(void *storage, const int clsid) { | |||
2783 | 57 | if (hdr_it != NULL) { | 41 | if (hdr_it != NULL) { |
2784 | 58 | int bucket = (it->it_flags & ITEM_CHUNKED) ? | 42 | int bucket = (it->it_flags & ITEM_CHUNKED) ? |
2785 | 59 | PAGE_BUCKET_CHUNKED : PAGE_BUCKET_DEFAULT; | 43 | PAGE_BUCKET_CHUNKED : PAGE_BUCKET_DEFAULT; |
2787 | 60 | // Compres soon to expire items into similar pages. | 44 | // Compress soon to expire items into similar pages. |
2788 | 61 | if (it->exptime - current_time < settings.ext_low_ttl) { | 45 | if (it->exptime - current_time < settings.ext_low_ttl) { |
2789 | 62 | bucket = PAGE_BUCKET_LOWTTL; | 46 | bucket = PAGE_BUCKET_LOWTTL; |
2790 | 63 | } | 47 | } |
2791 | @@ -67,7 +51,9 @@ int lru_maintainer_store(void *storage, const int clsid) { | |||
2792 | 67 | // NOTE: when the item is read back in, the slab mover | 51 | // NOTE: when the item is read back in, the slab mover |
2793 | 68 | // may see it. Important to have refcount>=2 or ~ITEM_LINKED | 52 | // may see it. Important to have refcount>=2 or ~ITEM_LINKED |
2794 | 69 | assert(it->refcount >= 2); | 53 | assert(it->refcount >= 2); |
2796 | 70 | if (extstore_write_request(storage, bucket, &io) == 0) { | 54 | // NOTE: write bucket vs free page bucket will disambiguate once |
2797 | 55 | // lowttl feature is better understood. | ||
2798 | 56 | if (extstore_write_request(storage, bucket, bucket, &io) == 0) { | ||
2799 | 71 | // cuddle the hash value into the time field so we don't have | 57 | // cuddle the hash value into the time field so we don't have |
2800 | 72 | // to recalculate it. | 58 | // to recalculate it. |
2801 | 73 | item *buf_it = (item *) io.buf; | 59 | item *buf_it = (item *) io.buf; |
2802 | @@ -76,12 +62,12 @@ int lru_maintainer_store(void *storage, const int clsid) { | |||
2803 | 76 | // TODO: should be in items.c | 62 | // TODO: should be in items.c |
2804 | 77 | if (it->it_flags & ITEM_CHUNKED) { | 63 | if (it->it_flags & ITEM_CHUNKED) { |
2805 | 78 | // Need to loop through the item and copy | 64 | // Need to loop through the item and copy |
2807 | 79 | item_chunk *sch = (item_chunk *) ITEM_data(it); | 65 | item_chunk *sch = (item_chunk *) ITEM_schunk(it); |
2808 | 80 | int remain = orig_ntotal; | 66 | int remain = orig_ntotal; |
2809 | 81 | int copied = 0; | 67 | int copied = 0; |
2810 | 82 | // copy original header | 68 | // copy original header |
2811 | 83 | int hdrtotal = ITEM_ntotal(it) - it->nbytes; | 69 | int hdrtotal = ITEM_ntotal(it) - it->nbytes; |
2813 | 84 | memcpy((char *)io.buf+32, (char *)it+32, hdrtotal - 32); | 70 | memcpy((char *)io.buf+STORE_OFFSET, (char *)it+STORE_OFFSET, hdrtotal - STORE_OFFSET); |
2814 | 85 | copied = hdrtotal; | 71 | copied = hdrtotal; |
2815 | 86 | // copy data in like it were one large object. | 72 | // copy data in like it were one large object. |
2816 | 87 | while (sch && remain) { | 73 | while (sch && remain) { |
2817 | @@ -93,11 +79,11 @@ int lru_maintainer_store(void *storage, const int clsid) { | |||
2818 | 93 | sch = sch->next; | 79 | sch = sch->next; |
2819 | 94 | } | 80 | } |
2820 | 95 | } else { | 81 | } else { |
2822 | 96 | memcpy((char *)io.buf+32, (char *)it+32, io.len-32); | 82 | memcpy((char *)io.buf+STORE_OFFSET, (char *)it+STORE_OFFSET, io.len-STORE_OFFSET); |
2823 | 97 | } | 83 | } |
2824 | 98 | // crc what we copied so we can do it sequentially. | 84 | // crc what we copied so we can do it sequentially. |
2825 | 99 | buf_it->it_flags &= ~ITEM_LINKED; | 85 | buf_it->it_flags &= ~ITEM_LINKED; |
2827 | 100 | buf_it->exptime = crc32c(0, (char*)io.buf+32, orig_ntotal-32); | 86 | buf_it->exptime = crc32c(0, (char*)io.buf+STORE_OFFSET, orig_ntotal-STORE_OFFSET); |
2828 | 101 | extstore_write(storage, &io); | 87 | extstore_write(storage, &io); |
2829 | 102 | item_hdr *hdr = (item_hdr *) ITEM_data(hdr_it); | 88 | item_hdr *hdr = (item_hdr *) ITEM_data(hdr_it); |
2830 | 103 | hdr->page_version = io.page_version; | 89 | hdr->page_version = io.page_version; |
2831 | @@ -125,6 +111,128 @@ int lru_maintainer_store(void *storage, const int clsid) { | |||
2832 | 125 | return did_moves; | 111 | return did_moves; |
2833 | 126 | } | 112 | } |
2834 | 127 | 113 | ||
2835 | 114 | static pthread_t storage_write_tid; | ||
2836 | 115 | static pthread_mutex_t storage_write_plock; | ||
2837 | 116 | #define WRITE_SLEEP_MAX 1000000 | ||
2838 | 117 | #define WRITE_SLEEP_MIN 500 | ||
2839 | 118 | |||
2840 | 119 | static void *storage_write_thread(void *arg) { | ||
2841 | 120 | void *storage = arg; | ||
2842 | 121 | // NOTE: ignoring overflow since that would take years of uptime in a | ||
2843 | 122 | // specific load pattern of never going to sleep. | ||
2844 | 123 | unsigned int backoff[MAX_NUMBER_OF_SLAB_CLASSES] = {0}; | ||
2845 | 124 | unsigned int counter = 0; | ||
2846 | 125 | useconds_t to_sleep = WRITE_SLEEP_MIN; | ||
2847 | 126 | logger *l = logger_create(); | ||
2848 | 127 | if (l == NULL) { | ||
2849 | 128 | fprintf(stderr, "Failed to allocate logger for storage compaction thread\n"); | ||
2850 | 129 | abort(); | ||
2851 | 130 | } | ||
2852 | 131 | |||
2853 | 132 | pthread_mutex_lock(&storage_write_plock); | ||
2854 | 133 | |||
2855 | 134 | while (1) { | ||
2856 | 135 | // cache per-loop to avoid calls to the slabs_clsid() search loop | ||
2857 | 136 | int min_class = slabs_clsid(settings.ext_item_size); | ||
2858 | 137 | bool do_sleep = true; | ||
2859 | 138 | counter++; | ||
2860 | 139 | if (to_sleep > WRITE_SLEEP_MAX) | ||
2861 | 140 | to_sleep = WRITE_SLEEP_MAX; | ||
2862 | 141 | |||
2863 | 142 | for (int x = 0; x < MAX_NUMBER_OF_SLAB_CLASSES; x++) { | ||
2864 | 143 | bool did_move = false; | ||
2865 | 144 | bool mem_limit_reached = false; | ||
2866 | 145 | unsigned int chunks_free; | ||
2867 | 146 | int item_age; | ||
2868 | 147 | int target = settings.ext_free_memchunks[x]; | ||
2869 | 148 | if (min_class > x || (backoff[x] && (counter % backoff[x] != 0))) { | ||
2870 | 149 | // Long sleeps means we should retry classes sooner. | ||
2871 | 150 | if (to_sleep > WRITE_SLEEP_MIN * 10) | ||
2872 | 151 | backoff[x] /= 2; | ||
2873 | 152 | continue; | ||
2874 | 153 | } | ||
2875 | 154 | |||
2876 | 155 | // Avoid extra slab lock calls during heavy writing. | ||
2877 | 156 | chunks_free = slabs_available_chunks(x, &mem_limit_reached, | ||
2878 | 157 | NULL, NULL); | ||
2879 | 158 | |||
2880 | 159 | // storage_write() will fail and cut loop after filling write buffer. | ||
2881 | 160 | while (1) { | ||
2882 | 161 | // if we are low on chunks and no spare, push out early. | ||
2883 | 162 | if (chunks_free < target && mem_limit_reached) { | ||
2884 | 163 | item_age = 0; | ||
2885 | 164 | } else { | ||
2886 | 165 | item_age = settings.ext_item_age; | ||
2887 | 166 | } | ||
2888 | 167 | if (storage_write(storage, x, item_age)) { | ||
2889 | 168 | chunks_free++; // Allow stopping if we've done enough this loop | ||
2890 | 169 | did_move = true; | ||
2891 | 170 | do_sleep = false; | ||
2892 | 171 | if (to_sleep > WRITE_SLEEP_MIN) | ||
2893 | 172 | to_sleep /= 2; | ||
2894 | 173 | } else { | ||
2895 | 174 | break; | ||
2896 | 175 | } | ||
2897 | 176 | } | ||
2898 | 177 | |||
2899 | 178 | if (!did_move) { | ||
2900 | 179 | backoff[x]++; | ||
2901 | 180 | } else if (backoff[x]) { | ||
2902 | 181 | backoff[x] /= 2; | ||
2903 | 182 | } | ||
2904 | 183 | } | ||
2905 | 184 | |||
2906 | 185 | // flip lock so we can be paused or stopped | ||
2907 | 186 | pthread_mutex_unlock(&storage_write_plock); | ||
2908 | 187 | if (do_sleep) { | ||
2909 | 188 | usleep(to_sleep); | ||
2910 | 189 | to_sleep *= 2; | ||
2911 | 190 | } | ||
2912 | 191 | pthread_mutex_lock(&storage_write_plock); | ||
2913 | 192 | } | ||
2914 | 193 | return NULL; | ||
2915 | 194 | } | ||
2916 | 195 | |||
2917 | 196 | // TODO | ||
2918 | 197 | // logger needs logger_destroy() to exist/work before this is safe. | ||
2919 | 198 | /*int stop_storage_write_thread(void) { | ||
2920 | 199 | int ret; | ||
2921 | 200 | pthread_mutex_lock(&lru_maintainer_lock); | ||
2922 | 201 | do_run_lru_maintainer_thread = 0; | ||
2923 | 202 | pthread_mutex_unlock(&lru_maintainer_lock); | ||
2924 | 203 | // WAKEUP SIGNAL | ||
2925 | 204 | if ((ret = pthread_join(lru_maintainer_tid, NULL)) != 0) { | ||
2926 | 205 | fprintf(stderr, "Failed to stop LRU maintainer thread: %s\n", strerror(ret)); | ||
2927 | 206 | return -1; | ||
2928 | 207 | } | ||
2929 | 208 | settings.lru_maintainer_thread = false; | ||
2930 | 209 | return 0; | ||
2931 | 210 | }*/ | ||
2932 | 211 | |||
2933 | 212 | void storage_write_pause(void) { | ||
2934 | 213 | pthread_mutex_lock(&storage_write_plock); | ||
2935 | 214 | } | ||
2936 | 215 | |||
2937 | 216 | void storage_write_resume(void) { | ||
2938 | 217 | pthread_mutex_unlock(&storage_write_plock); | ||
2939 | 218 | } | ||
2940 | 219 | |||
2941 | 220 | int start_storage_write_thread(void *arg) { | ||
2942 | 221 | int ret; | ||
2943 | 222 | |||
2944 | 223 | pthread_mutex_init(&storage_write_plock, NULL); | ||
2945 | 224 | if ((ret = pthread_create(&storage_write_tid, NULL, | ||
2946 | 225 | storage_write_thread, arg)) != 0) { | ||
2947 | 226 | fprintf(stderr, "Can't create storage_write thread: %s\n", | ||
2948 | 227 | strerror(ret)); | ||
2949 | 228 | return -1; | ||
2950 | 229 | } | ||
2951 | 230 | |||
2952 | 231 | return 0; | ||
2953 | 232 | } | ||
2954 | 233 | |||
2955 | 234 | /*** COMPACTOR ***/ | ||
2956 | 235 | |||
2957 | 128 | /* Fetch stats from the external storage system and decide to compact. | 236 | /* Fetch stats from the external storage system and decide to compact. |
2958 | 129 | * If we're more than half full, start skewing how aggressively to run | 237 | * If we're more than half full, start skewing how aggressively to run |
2959 | 130 | * compaction, up to a desired target when all pages are full. | 238 | * compaction, up to a desired target when all pages are full. |
2960 | @@ -260,7 +368,7 @@ static void storage_compact_readback(void *storage, logger *l, | |||
2961 | 260 | io.len = ntotal; | 368 | io.len = ntotal; |
2962 | 261 | io.mode = OBJ_IO_WRITE; | 369 | io.mode = OBJ_IO_WRITE; |
2963 | 262 | for (tries = 10; tries > 0; tries--) { | 370 | for (tries = 10; tries > 0; tries--) { |
2965 | 263 | if (extstore_write_request(storage, PAGE_BUCKET_COMPACT, &io) == 0) { | 371 | if (extstore_write_request(storage, PAGE_BUCKET_COMPACT, PAGE_BUCKET_COMPACT, &io) == 0) { |
2966 | 264 | memcpy(io.buf, it, io.len); | 372 | memcpy(io.buf, it, io.len); |
2967 | 265 | extstore_write(storage, &io); | 373 | extstore_write(storage, &io); |
2968 | 266 | do_update = true; | 374 | do_update = true; |
2969 | @@ -456,4 +564,88 @@ int start_storage_compact_thread(void *arg) { | |||
2970 | 456 | return 0; | 564 | return 0; |
2971 | 457 | } | 565 | } |
2972 | 458 | 566 | ||
2973 | 567 | /*** UTILITY ***/ | ||
2974 | 568 | // /path/to/file:100G:bucket1 | ||
2975 | 569 | // FIXME: Modifies argument. copy instead? | ||
2976 | 570 | struct extstore_conf_file *storage_conf_parse(char *arg, unsigned int page_size) { | ||
2977 | 571 | struct extstore_conf_file *cf = NULL; | ||
2978 | 572 | char *b = NULL; | ||
2979 | 573 | char *p = strtok_r(arg, ":", &b); | ||
2980 | 574 | char unit = 0; | ||
2981 | 575 | uint64_t multiplier = 0; | ||
2982 | 576 | int base_size = 0; | ||
2983 | 577 | if (p == NULL) | ||
2984 | 578 | goto error; | ||
2985 | 579 | // First arg is the filepath. | ||
2986 | 580 | cf = calloc(1, sizeof(struct extstore_conf_file)); | ||
2987 | 581 | cf->file = strdup(p); | ||
2988 | 582 | |||
2989 | 583 | p = strtok_r(NULL, ":", &b); | ||
2990 | 584 | if (p == NULL) { | ||
2991 | 585 | fprintf(stderr, "must supply size to ext_path, ie: ext_path=/f/e:64m (M|G|T|P supported)\n"); | ||
2992 | 586 | goto error; | ||
2993 | 587 | } | ||
2994 | 588 | unit = tolower(p[strlen(p)-1]); | ||
2995 | 589 | p[strlen(p)-1] = '\0'; | ||
2996 | 590 | // sigh. | ||
2997 | 591 | switch (unit) { | ||
2998 | 592 | case 'm': | ||
2999 | 593 | multiplier = 1024 * 1024; | ||
3000 | 594 | break; | ||
3001 | 595 | case 'g': | ||
3002 | 596 | multiplier = 1024 * 1024 * 1024; | ||
3003 | 597 | break; | ||
3004 | 598 | case 't': | ||
3005 | 599 | multiplier = 1024 * 1024; | ||
3006 | 600 | multiplier *= 1024 * 1024; | ||
3007 | 601 | break; | ||
3008 | 602 | case 'p': | ||
3009 | 603 | multiplier = 1024 * 1024; | ||
3010 | 604 | multiplier *= 1024 * 1024 * 1024; | ||
3011 | 605 | break; | ||
3012 | 606 | } | ||
3013 | 607 | base_size = atoi(p); | ||
3014 | 608 | multiplier *= base_size; | ||
3015 | 609 | // page_count is nearest-but-not-larger-than pages * psize | ||
3016 | 610 | cf->page_count = multiplier / page_size; | ||
3017 | 611 | assert(page_size * cf->page_count <= multiplier); | ||
3018 | 612 | |||
3019 | 613 | // final token would be a default free bucket | ||
3020 | 614 | p = strtok_r(NULL, ",", &b); | ||
3021 | 615 | // TODO: We reuse the original DEFINES for now, | ||
3022 | 616 | // but if lowttl gets split up this needs to be its own set. | ||
3023 | 617 | if (p != NULL) { | ||
3024 | 618 | if (strcmp(p, "compact") == 0) { | ||
3025 | 619 | cf->free_bucket = PAGE_BUCKET_COMPACT; | ||
3026 | 620 | } else if (strcmp(p, "lowttl") == 0) { | ||
3027 | 621 | cf->free_bucket = PAGE_BUCKET_LOWTTL; | ||
3028 | 622 | } else if (strcmp(p, "chunked") == 0) { | ||
3029 | 623 | cf->free_bucket = PAGE_BUCKET_CHUNKED; | ||
3030 | 624 | } else if (strcmp(p, "default") == 0) { | ||
3031 | 625 | cf->free_bucket = PAGE_BUCKET_DEFAULT; | ||
3032 | 626 | } else { | ||
3033 | 627 | fprintf(stderr, "Unknown extstore bucket: %s\n", p); | ||
3034 | 628 | goto error; | ||
3035 | 629 | } | ||
3036 | 630 | } else { | ||
3037 | 631 | // TODO: is this necessary? | ||
3038 | 632 | cf->free_bucket = PAGE_BUCKET_DEFAULT; | ||
3039 | 633 | } | ||
3040 | 634 | |||
3041 | 635 | // TODO: disabling until compact algorithm is improved. | ||
3042 | 636 | if (cf->free_bucket != PAGE_BUCKET_DEFAULT) { | ||
3043 | 637 | fprintf(stderr, "ext_path only presently supports the default bucket\n"); | ||
3044 | 638 | goto error; | ||
3045 | 639 | } | ||
3046 | 640 | |||
3047 | 641 | return cf; | ||
3048 | 642 | error: | ||
3049 | 643 | if (cf) { | ||
3050 | 644 | if (cf->file) | ||
3051 | 645 | free(cf->file); | ||
3052 | 646 | free(cf); | ||
3053 | 647 | } | ||
3054 | 648 | return NULL; | ||
3055 | 649 | } | ||
3056 | 650 | |||
3057 | 459 | #endif | 651 | #endif |
3058 | diff --git a/storage.h b/storage.h | |||
3059 | index 7af6cc0..60f499b 100644 | |||
3060 | --- a/storage.h | |||
3061 | +++ b/storage.h | |||
3062 | @@ -1,9 +1,15 @@ | |||
3063 | 1 | #ifndef STORAGE_H | 1 | #ifndef STORAGE_H |
3064 | 2 | #define STORAGE_H | 2 | #define STORAGE_H |
3065 | 3 | 3 | ||
3067 | 4 | int lru_maintainer_store(void *storage, const int clsid); | 4 | int start_storage_write_thread(void *arg); |
3068 | 5 | void storage_write_pause(void); | ||
3069 | 6 | void storage_write_resume(void); | ||
3070 | 5 | int start_storage_compact_thread(void *arg); | 7 | int start_storage_compact_thread(void *arg); |
3071 | 6 | void storage_compact_pause(void); | 8 | void storage_compact_pause(void); |
3072 | 7 | void storage_compact_resume(void); | 9 | void storage_compact_resume(void); |
3073 | 10 | struct extstore_conf_file *storage_conf_parse(char *arg, unsigned int page_size); | ||
3074 | 11 | |||
3075 | 12 | // Ignore pointers and header bits from the CRC | ||
3076 | 13 | #define STORE_OFFSET offsetof(item, nbytes) | ||
3077 | 8 | 14 | ||
3078 | 9 | #endif | 15 | #endif |
3079 | diff --git a/t/binary-extstore.t b/t/binary-extstore.t | |||
3080 | index 390e26a..676d13b 100755 | |||
3081 | --- a/t/binary-extstore.t | |||
3082 | +++ b/t/binary-extstore.t | |||
3083 | @@ -17,7 +17,7 @@ if (!supports_extstore()) { | |||
3084 | 17 | 17 | ||
3085 | 18 | $ext_path = "/tmp/extstore.$$"; | 18 | $ext_path = "/tmp/extstore.$$"; |
3086 | 19 | 19 | ||
3088 | 20 | my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,no_lru_crawler,slab_automove=0"); | 20 | my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,no_lru_crawler,slab_automove=0"); |
3089 | 21 | ok($server, "started the server"); | 21 | ok($server, "started the server"); |
3090 | 22 | 22 | ||
3091 | 23 | # Based almost 100% off testClient.py which is: | 23 | # Based almost 100% off testClient.py which is: |
3092 | @@ -176,7 +176,15 @@ $set->('x', 10, 19, "somevalue"); | |||
3093 | 176 | $set->("mfoo$_", 0, 19, $value); | 176 | $set->("mfoo$_", 0, 19, $value); |
3094 | 177 | } | 177 | } |
3095 | 178 | sleep 4; | 178 | sleep 4; |
3097 | 179 | $empty->('mfoo1'); | 179 | # FIXME: Need to sample through a few values, or fix eviction to be |
3098 | 180 | # more accurate. On 32bit systems some pages unused to this point get | ||
3099 | 181 | # filled after the first few items, then the eviction algo pulls those | ||
3100 | 182 | # pages since they have the lowest version number, leaving older objects | ||
3101 | 183 | # in memory and evicting newer ones. | ||
3102 | 184 | for (1 .. ($keycount*3)) { | ||
3103 | 185 | next unless $_ % 100 == 0; | ||
3104 | 186 | eval { $mc->get("mfoo$_"); }; | ||
3105 | 187 | } | ||
3106 | 180 | 188 | ||
3107 | 181 | my %s = $mc->stats(''); | 189 | my %s = $mc->stats(''); |
3108 | 182 | cmp_ok($s{extstore_objects_evicted}, '>', 0); | 190 | cmp_ok($s{extstore_objects_evicted}, '>', 0); |
3109 | diff --git a/t/binary-sasl.t b/t/binary-sasl.t | |||
3110 | index 85ef069..e923584 100755 | |||
3111 | --- a/t/binary-sasl.t | |||
3112 | +++ b/t/binary-sasl.t | |||
3113 | @@ -13,7 +13,7 @@ use Test::More; | |||
3114 | 13 | 13 | ||
3115 | 14 | if (supports_sasl()) { | 14 | if (supports_sasl()) { |
3116 | 15 | if ($ENV{'RUN_SASL_TESTS'}) { | 15 | if ($ENV{'RUN_SASL_TESTS'}) { |
3118 | 16 | plan tests => 33; | 16 | plan tests => 34; |
3119 | 17 | } else { | 17 | } else { |
3120 | 18 | plan skip_all => 'Skipping SASL tests'; | 18 | plan skip_all => 'Skipping SASL tests'; |
3121 | 19 | exit 0; | 19 | exit 0; |
3122 | @@ -92,7 +92,7 @@ use constant RES_MAGIC => 0x81; | |||
3123 | 92 | my $pwd=getcwd; | 92 | my $pwd=getcwd; |
3124 | 93 | $ENV{'SASL_CONF_PATH'} = "$pwd/t/sasl"; | 93 | $ENV{'SASL_CONF_PATH'} = "$pwd/t/sasl"; |
3125 | 94 | 94 | ||
3127 | 95 | my $server = new_memcached('-B binary -S '); | 95 | my $server = new_memcached('-B binary -S -l 127.0.0.1 '); |
3128 | 96 | 96 | ||
3129 | 97 | my $mc = MC::Client->new; | 97 | my $mc = MC::Client->new; |
3130 | 98 | 98 | ||
3131 | @@ -192,7 +192,9 @@ for my $dir (split(/:/, $ENV{PATH}), | |||
3132 | 192 | } | 192 | } |
3133 | 193 | } | 193 | } |
3134 | 194 | 194 | ||
3136 | 195 | system("echo testpass | $saslpasswd_path -a memcached -c -p testuser"); | 195 | my $sasl_realm = 'memcached.realm'; |
3137 | 196 | |||
3138 | 197 | system("echo testpass | $saslpasswd_path -a memcached -u $sasl_realm -c -p testuser"); | ||
3139 | 196 | 198 | ||
3140 | 197 | $mc = MC::Client->new; | 199 | $mc = MC::Client->new; |
3141 | 198 | 200 | ||
3142 | @@ -261,6 +263,11 @@ $empty->('x', 'somevalue'); | |||
3143 | 261 | cmp_ok($status,'==',ERR_AUTH_ERROR, "error code matches"); | 263 | cmp_ok($status,'==',ERR_AUTH_ERROR, "error code matches"); |
3144 | 262 | } | 264 | } |
3145 | 263 | 265 | ||
3146 | 266 | { | ||
3147 | 267 | my $mc = MC::Client->new; | ||
3148 | 268 | is ($mc->sasl_step('testuser', 'testpass'), 0x20, "sasl_step_fails_no_segfault"); | ||
3149 | 269 | } | ||
3150 | 270 | |||
3151 | 264 | # check the SASL stats, make sure they track things correctly | 271 | # check the SASL stats, make sure they track things correctly |
3152 | 265 | # note: the enabled or not is presence checked in stats.t | 272 | # note: the enabled or not is presence checked in stats.t |
3153 | 266 | 273 | ||
3154 | @@ -273,8 +280,8 @@ $empty->('x', 'somevalue'); | |||
3155 | 273 | 280 | ||
3156 | 274 | { | 281 | { |
3157 | 275 | my %stats = $mc->stats(''); | 282 | my %stats = $mc->stats(''); |
3160 | 276 | is ($stats{'auth_cmds'}, 5, "auth commands counted"); | 283 | is ($stats{'auth_cmds'}, 6, "auth commands counted"); |
3161 | 277 | is ($stats{'auth_errors'}, 3, "auth errors correct"); | 284 | is ($stats{'auth_errors'}, 4, "auth errors correct"); |
3162 | 278 | } | 285 | } |
3163 | 279 | 286 | ||
3164 | 280 | 287 | ||
3165 | @@ -309,10 +316,17 @@ sub new { | |||
3166 | 309 | sub authenticate { | 316 | sub authenticate { |
3167 | 310 | my ($self, $user, $pass, $mech)= @_; | 317 | my ($self, $user, $pass, $mech)= @_; |
3168 | 311 | $mech ||= 'PLAIN'; | 318 | $mech ||= 'PLAIN'; |
3170 | 312 | my $buf = sprintf("%c%s%c%s", 0, $user, 0, $pass); | 319 | my $buf = sprintf("%c%s@%s%c%s", 0, $user, $sasl_realm, 0, $pass); |
3171 | 313 | my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_AUTH, $mech, $buf, ''); | 320 | my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_AUTH, $mech, $buf, ''); |
3172 | 314 | return $status; | 321 | return $status; |
3173 | 315 | } | 322 | } |
3174 | 323 | sub sasl_step { | ||
3175 | 324 | my ($self, $user, $pass, $mech)= @_; | ||
3176 | 325 | $mech ||= 'PLAIN'; | ||
3177 | 326 | my $buf = sprintf("%c%s@%s%c%s", 0, $user, $sasl_realm, 0, $pass); | ||
3178 | 327 | my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_STEP, $mech, $buf, ''); | ||
3179 | 328 | return $status; | ||
3180 | 329 | } | ||
3181 | 316 | sub list_mechs { | 330 | sub list_mechs { |
3182 | 317 | my ($self)= @_; | 331 | my ($self)= @_; |
3183 | 318 | my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_LIST_MECHS, '', '', ''); | 332 | my ($status, $rv, undef) = $self->_do_command(::CMD_SASL_LIST_MECHS, '', '', ''); |
3184 | @@ -661,4 +675,3 @@ sub auth_error { | |||
3185 | 661 | unlink $sasldb; | 675 | unlink $sasldb; |
3186 | 662 | 676 | ||
3187 | 663 | # vim: filetype=perl | 677 | # vim: filetype=perl |
3188 | 664 | |||
3189 | diff --git a/t/chunked-extstore.t b/t/chunked-extstore.t | |||
3190 | index ebd0f0b..3515f21 100644 | |||
3191 | --- a/t/chunked-extstore.t | |||
3192 | +++ b/t/chunked-extstore.t | |||
3193 | @@ -18,9 +18,26 @@ if (!supports_extstore()) { | |||
3194 | 18 | 18 | ||
3195 | 19 | $ext_path = "/tmp/extstore.$$"; | 19 | $ext_path = "/tmp/extstore.$$"; |
3196 | 20 | 20 | ||
3198 | 21 | my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,slab_chunk_max=16384,slab_automove=0"); | 21 | my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_chunk_max=16384,slab_automove=0,ext_compact_under=1"); |
3199 | 22 | my $sock = $server->sock; | 22 | my $sock = $server->sock; |
3200 | 23 | 23 | ||
3201 | 24 | # Wait until all items have flushed | ||
3202 | 25 | sub wait_for_ext { | ||
3203 | 26 | my $sum = 1; | ||
3204 | 27 | while ($sum != 0) { | ||
3205 | 28 | my $s = mem_stats($sock, "items"); | ||
3206 | 29 | $sum = 0; | ||
3207 | 30 | for my $key (keys %$s) { | ||
3208 | 31 | if ($key =~ m/items:(\d+):number/) { | ||
3209 | 32 | # Ignore classes which can contain extstore items | ||
3210 | 33 | next if $1 < 3; | ||
3211 | 34 | $sum += $s->{$key}; | ||
3212 | 35 | } | ||
3213 | 36 | } | ||
3214 | 37 | sleep 1 if $sum != 0; | ||
3215 | 38 | } | ||
3216 | 39 | } | ||
3217 | 40 | |||
3218 | 24 | # We're testing to ensure item chaining doesn't corrupt or poorly overlap | 41 | # We're testing to ensure item chaining doesn't corrupt or poorly overlap |
3219 | 25 | # data, so create a non-repeating pattern. | 42 | # data, so create a non-repeating pattern. |
3220 | 26 | my @parts = (); | 43 | my @parts = (); |
3221 | @@ -58,7 +75,7 @@ for (1..5) { | |||
3222 | 58 | print $sock "set toast$_ 0 0 $biglen\r\n$big\r\n"; | 75 | print $sock "set toast$_ 0 0 $biglen\r\n$big\r\n"; |
3223 | 59 | is(scalar <$sock>, "STORED\r\n", "stored big"); | 76 | is(scalar <$sock>, "STORED\r\n", "stored big"); |
3224 | 60 | } | 77 | } |
3226 | 61 | sleep 4; | 78 | wait_for_ext(); |
3227 | 62 | 79 | ||
3228 | 63 | for (1..40) { | 80 | for (1..40) { |
3229 | 64 | mem_get_is($sock, "toast$_", $big); | 81 | mem_get_is($sock, "toast$_", $big); |
3230 | @@ -76,17 +93,19 @@ for (1..5) { | |||
3231 | 76 | 93 | ||
3232 | 77 | # fill to eviction | 94 | # fill to eviction |
3233 | 78 | { | 95 | { |
3235 | 79 | my $keycount = 1000; | 96 | my $keycount = 1250; |
3236 | 80 | for (1 .. $keycount) { | 97 | for (1 .. $keycount) { |
3237 | 81 | print $sock "set mfoo$_ 0 0 $plen noreply\r\n$pattern\r\n"; | 98 | print $sock "set mfoo$_ 0 0 $plen noreply\r\n$pattern\r\n"; |
3238 | 99 | wait_for_ext() if $_ % 500 == 0; | ||
3239 | 82 | } | 100 | } |
3241 | 83 | sleep 6; | 101 | # because item_age is set to 2s. |
3242 | 102 | wait_for_ext(); | ||
3243 | 84 | 103 | ||
3244 | 85 | my $stats = mem_stats($sock); | 104 | my $stats = mem_stats($sock); |
3245 | 86 | cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted'); | 105 | cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted'); |
3246 | 87 | cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted'); | 106 | cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted'); |
3247 | 88 | cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted'); | 107 | cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted'); |
3249 | 89 | is($stats->{extstore_pages_free}, 1, '1 page is free'); | 108 | cmp_ok($stats->{extstore_pages_free}, '<', 2, 'most pages are used'); |
3250 | 90 | is($stats->{miss_from_extstore}, 0, 'no misses'); | 109 | is($stats->{miss_from_extstore}, 0, 'no misses'); |
3251 | 91 | 110 | ||
3252 | 92 | # original "pattern" key should be gone. | 111 | # original "pattern" key should be gone. |
3253 | @@ -106,14 +125,18 @@ for (1..5) { | |||
3254 | 106 | print $sock "delete toast$_ noreply\r\n" if $_ % 2 == 0; | 125 | print $sock "delete toast$_ noreply\r\n" if $_ % 2 == 0; |
3255 | 107 | } | 126 | } |
3256 | 108 | 127 | ||
3258 | 109 | for (1..1000) { | 128 | for (1..1250) { |
3259 | 110 | # Force a read so objects don't get skipped. | 129 | # Force a read so objects don't get skipped. |
3261 | 111 | mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1; | 130 | print $sock "add mfoo$_ 0 0 1 noreply\r\n1\r\n" if $_ % 2 == 1; |
3262 | 112 | } | 131 | } |
3265 | 113 | for (1..1000) { | 132 | for (1..1250) { |
3266 | 114 | # Force a read so objects don't get skipped. | 133 | # Delete lots of objects to trigger compaction. |
3267 | 115 | print $sock "delete mfoo$_ noreply\r\n" if $_ % 2 == 0; | 134 | print $sock "delete mfoo$_ noreply\r\n" if $_ % 2 == 0; |
3268 | 116 | } | 135 | } |
3269 | 136 | print $sock "extstore compact_under 4\r\n"; | ||
3270 | 137 | my $res = <$sock>; | ||
3271 | 138 | print $sock "extstore drop_under 3\r\n"; | ||
3272 | 139 | $res = <$sock>; | ||
3273 | 117 | 140 | ||
3274 | 118 | sleep 4; | 141 | sleep 4; |
3275 | 119 | 142 | ||
3276 | @@ -122,7 +145,7 @@ for (1..5) { | |||
3277 | 122 | cmp_ok($stats->{extstore_compact_rescues}, '>', 0, 'some compaction rescues happened'); | 145 | cmp_ok($stats->{extstore_compact_rescues}, '>', 0, 'some compaction rescues happened'); |
3278 | 123 | 146 | ||
3279 | 124 | # Some of the early items got evicted | 147 | # Some of the early items got evicted |
3281 | 125 | for (100..1000) { | 148 | for (750..1250) { |
3282 | 126 | # everything should validate properly. | 149 | # everything should validate properly. |
3283 | 127 | mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1; | 150 | mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1; |
3284 | 128 | } | 151 | } |
3285 | @@ -133,20 +156,12 @@ for (1..5) { | |||
3286 | 133 | print $sock "extstore recache_rate 1\r\n"; | 156 | print $sock "extstore recache_rate 1\r\n"; |
3287 | 134 | is(scalar <$sock>, "OK\r\n", "upped recache rate"); | 157 | is(scalar <$sock>, "OK\r\n", "upped recache rate"); |
3288 | 135 | 158 | ||
3290 | 136 | for (800..1000) { | 159 | for (1150..1250) { |
3291 | 137 | mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1; | 160 | mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1; |
3292 | 138 | } | 161 | } |
3293 | 139 | 162 | ||
3294 | 140 | my $stats = mem_stats($sock); | 163 | my $stats = mem_stats($sock); |
3304 | 141 | cmp_ok($stats->{recache_from_extstore}, '>', 100, 'recaching happening'); | 164 | cmp_ok($stats->{recache_from_extstore}, '>', 25, 'recaching happening'); |
3296 | 142 | |||
3297 | 143 | for (800..1000) { | ||
3298 | 144 | mem_get_is($sock, "mfoo$_", $pattern) if $_ % 2 == 1; | ||
3299 | 145 | } | ||
3300 | 146 | |||
3301 | 147 | my $stats2 = mem_stats($sock); | ||
3302 | 148 | is($stats->{recache_from_extstore}, $stats2->{recache_from_extstore}, | ||
3303 | 149 | 'values already recached'); | ||
3305 | 150 | } | 165 | } |
3306 | 151 | 166 | ||
3307 | 152 | done_testing(); | 167 | done_testing(); |
3308 | diff --git a/t/error-extstore.t b/t/error-extstore.t | |||
3309 | 153 | new file mode 100644 | 168 | new file mode 100644 |
3310 | index 0000000..6df1528 | |||
3311 | --- /dev/null | |||
3312 | +++ b/t/error-extstore.t | |||
3313 | @@ -0,0 +1,130 @@ | |||
3314 | 1 | #!/usr/bin/perl | ||
3315 | 2 | # Test the "Error on get" path for extstore. | ||
3316 | 3 | # the entire error handling code for process_get_command() never worked, and | ||
3317 | 4 | # would infinite loop. get_extstore() can hit it sometimes. | ||
3318 | 5 | |||
3319 | 6 | use strict; | ||
3320 | 7 | use warnings; | ||
3321 | 8 | |||
3322 | 9 | use Test::More; | ||
3323 | 10 | use FindBin qw($Bin); | ||
3324 | 11 | use lib "$Bin/lib"; | ||
3325 | 12 | use MemcachedTest; | ||
3326 | 13 | |||
3327 | 14 | my $ext_path; | ||
3328 | 15 | |||
3329 | 16 | if (!supports_extstore()) { | ||
3330 | 17 | plan skip_all => 'extstore not enabled'; | ||
3331 | 18 | exit 0; | ||
3332 | 19 | } | ||
3333 | 20 | |||
3334 | 21 | $ext_path = "/tmp/extstore.$$"; | ||
3335 | 22 | |||
3336 | 23 | my $server = new_memcached("-m 64 -I 4m -U 0 -o ext_page_size=8,ext_wbuf_size=8,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_automove=0,ext_compact_under=1"); | ||
3337 | 24 | my $sock = $server->sock; | ||
3338 | 25 | |||
3339 | 26 | # Wait until all items have flushed | ||
3340 | 27 | sub wait_for_ext { | ||
3341 | 28 | my $sum = 1; | ||
3342 | 29 | while ($sum != 0) { | ||
3343 | 30 | my $s = mem_stats($sock, "items"); | ||
3344 | 31 | $sum = 0; | ||
3345 | 32 | for my $key (keys %$s) { | ||
3346 | 33 | if ($key =~ m/items:(\d+):number/) { | ||
3347 | 34 | # Ignore classes which can contain extstore items | ||
3348 | 35 | next if $1 < 3; | ||
3349 | 36 | $sum += $s->{$key}; | ||
3350 | 37 | } | ||
3351 | 38 | } | ||
3352 | 39 | sleep 1 if $sum != 0; | ||
3353 | 40 | } | ||
3354 | 41 | } | ||
3355 | 42 | |||
3356 | 43 | # We're testing to ensure item chaining doesn't corrupt or poorly overlap | ||
3357 | 44 | # data, so create a non-repeating pattern. | ||
3358 | 45 | my @parts = (); | ||
3359 | 46 | for (1 .. 8000) { | ||
3360 | 47 | push(@parts, $_); | ||
3361 | 48 | } | ||
3362 | 49 | my $pattern = join(':', @parts); | ||
3363 | 50 | my $plen = length($pattern); | ||
3364 | 51 | |||
3365 | 52 | # Set some large items and let them flush to extstore. | ||
3366 | 53 | for (1..5) { | ||
3367 | 54 | my $size = 3000 * 1024; | ||
3368 | 55 | my $data = "x" x $size; | ||
3369 | 56 | print $sock "set foo$_ 0 0 $size\r\n$data\r\n"; | ||
3370 | 57 | my $res = <$sock>; | ||
3371 | 58 | is($res, "STORED\r\n", "stored some big items"); | ||
3372 | 59 | } | ||
3373 | 60 | |||
3374 | 61 | wait_for_ext(); | ||
3375 | 62 | |||
3376 | 63 | { | ||
3377 | 64 | my $long_key = "f" x 512; | ||
3378 | 65 | print $sock "get foo1 foo2 foo3 $long_key\r\n"; | ||
3379 | 66 | ok(scalar <$sock> =~ /CLIENT_ERROR bad command line format/, 'long key fails'); | ||
3380 | 67 | my $stats = mem_stats($sock); | ||
3381 | 68 | cmp_ok($stats->{get_aborted_extstore}, '>', 1, 'some extstore queries aborted'); | ||
3382 | 69 | } | ||
3383 | 70 | |||
3384 | 71 | # Disable automatic page balancing, then move enough pages that the large | ||
3385 | 72 | # items can no longer be loaded from extstore | ||
3386 | 73 | { | ||
3387 | 74 | print $sock "slabs automove 0\r\n"; | ||
3388 | 75 | my $res = <$sock>; | ||
3389 | 76 | my $source = 0; | ||
3390 | 77 | while (1) { | ||
3391 | 78 | print $sock "slabs reassign $source 1\r\n"; | ||
3392 | 79 | $res = <$sock>; | ||
3393 | 80 | if ($res =~ m/NOSPARE/) { | ||
3394 | 81 | $source = -1; | ||
3395 | 82 | my $stats = mem_stats($sock, 'slabs'); | ||
3396 | 83 | for my $key (grep { /total_pages/ } keys %$stats) { | ||
3397 | 84 | if ($key =~ m/(\d+):total_pages/) { | ||
3398 | 85 | next if $1 < 3; | ||
3399 | 86 | $source = $1 if $stats->{$key} > 1; | ||
3400 | 87 | } | ||
3401 | 88 | } | ||
3402 | 89 | last if $source == -1; | ||
3403 | 90 | } | ||
3404 | 91 | select undef, undef, undef, 0.10; | ||
3405 | 92 | } | ||
3406 | 93 | } | ||
3407 | 94 | |||
3408 | 95 | # fetching the large keys should now fail. | ||
3409 | 96 | { | ||
3410 | 97 | print $sock "get foo1\r\n"; | ||
3411 | 98 | my $res = <$sock>; | ||
3412 | 99 | $res =~ s/[\r\n]//g; | ||
3413 | 100 | is($res, 'SERVER_ERROR out of memory writing get response', 'can no longer read back item'); | ||
3414 | 101 | my $stats = mem_stats($sock); | ||
3415 | 102 | is($stats->{get_oom_extstore}, 1, 'check extstore oom counter'); | ||
3416 | 103 | } | ||
3417 | 104 | |||
3418 | 105 | # Leaving this for future generations. | ||
3419 | 106 | # The process_get_command() function had several memory leaks. | ||
3420 | 107 | my $LEAK_TEST = 0; | ||
3421 | 108 | if ($LEAK_TEST) { | ||
3422 | 109 | my $tries = 0; | ||
3423 | 110 | while ($tries) { | ||
3424 | 111 | print $sock "slabs reassign 1 39\r\n"; | ||
3425 | 112 | my $res = <$sock>; | ||
3426 | 113 | if ($res =~ m/BUSY/) { | ||
3427 | 114 | select undef, undef, undef, 0.10; | ||
3428 | 115 | } else { | ||
3429 | 116 | $tries--; | ||
3430 | 117 | } | ||
3431 | 118 | } | ||
3432 | 119 | my $long_key = "f" x 512; | ||
3433 | 120 | while (1) { | ||
3434 | 121 | print $sock "get foo1 foo2 foo3 $long_key\r\n"; | ||
3435 | 122 | my $res = <$sock>; | ||
3436 | 123 | } | ||
3437 | 124 | } | ||
3438 | 125 | |||
3439 | 126 | done_testing(); | ||
3440 | 127 | |||
3441 | 128 | END { | ||
3442 | 129 | unlink $ext_path if $ext_path; | ||
3443 | 130 | } | ||
3444 | diff --git a/t/extstore-buckets.t b/t/extstore-buckets.t | |||
3445 | index a2c1c90..e3027ad 100644 | |||
3446 | --- a/t/extstore-buckets.t | |||
3447 | +++ b/t/extstore-buckets.t | |||
3448 | @@ -17,7 +17,7 @@ if (!supports_extstore()) { | |||
3449 | 17 | 17 | ||
3450 | 18 | $ext_path = "/tmp/extstore.$$"; | 18 | $ext_path = "/tmp/extstore.$$"; |
3451 | 19 | 19 | ||
3453 | 20 | my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0,ext_path=$ext_path,ext_low_ttl=60,slab_automove=1"); | 20 | my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0,ext_path=$ext_path:64m,ext_low_ttl=60,slab_automove=1"); |
3454 | 21 | my $sock = $server->sock; | 21 | my $sock = $server->sock; |
3455 | 22 | 22 | ||
3456 | 23 | my $value; | 23 | my $value; |
3457 | diff --git a/t/extstore-jbod.t b/t/extstore-jbod.t | |||
3458 | 24 | new file mode 100644 | 24 | new file mode 100644 |
3459 | index 0000000..1618803 | |||
3460 | --- /dev/null | |||
3461 | +++ b/t/extstore-jbod.t | |||
3462 | @@ -0,0 +1,69 @@ | |||
3463 | 1 | #!/usr/bin/perl | ||
3464 | 2 | |||
3465 | 3 | use strict; | ||
3466 | 4 | use warnings; | ||
3467 | 5 | use Test::More; | ||
3468 | 6 | use FindBin qw($Bin); | ||
3469 | 7 | use lib "$Bin/lib"; | ||
3470 | 8 | use MemcachedTest; | ||
3471 | 9 | use Data::Dumper qw/Dumper/; | ||
3472 | 10 | |||
3473 | 11 | my $ext_path; | ||
3474 | 12 | my $ext_path2; | ||
3475 | 13 | |||
3476 | 14 | if (!supports_extstore()) { | ||
3477 | 15 | plan skip_all => 'extstore not enabled'; | ||
3478 | 16 | exit 0; | ||
3479 | 17 | } | ||
3480 | 18 | |||
3481 | 19 | $ext_path = "/tmp/extstore1.$$"; | ||
3482 | 20 | $ext_path2 = "/tmp/extstore2.$$"; | ||
3483 | 21 | |||
3484 | 22 | my $server = new_memcached("-m 256 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,ext_path=$ext_path2:96m,slab_automove=1"); | ||
3485 | 23 | my $sock = $server->sock; | ||
3486 | 24 | |||
3487 | 25 | my $value; | ||
3488 | 26 | { | ||
3489 | 27 | my @chars = ("C".."Z"); | ||
3490 | 28 | for (1 .. 20000) { | ||
3491 | 29 | $value .= $chars[rand @chars]; | ||
3492 | 30 | } | ||
3493 | 31 | } | ||
3494 | 32 | |||
3495 | 33 | # fill some larger objects | ||
3496 | 34 | { | ||
3497 | 35 | # interleave sets with 0 ttl vs long ttl's. | ||
3498 | 36 | my $keycount = 3700; | ||
3499 | 37 | for (1 .. $keycount) { | ||
3500 | 38 | print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n"; | ||
3501 | 39 | print $sock "set lfoo$_ 0 0 20000 noreply\r\n$value\r\n"; | ||
3502 | 40 | } | ||
3503 | 41 | # wait for a flush | ||
3504 | 42 | wait_ext_flush($sock); | ||
3505 | 43 | # delete half | ||
3506 | 44 | mem_get_is($sock, "nfoo1", $value); | ||
3507 | 45 | for (1 .. $keycount) { | ||
3508 | 46 | print $sock "delete lfoo$_ noreply\r\n"; | ||
3509 | 47 | } | ||
3510 | 48 | print $sock "lru_crawler crawl all\r\n"; | ||
3511 | 49 | <$sock>; | ||
3512 | 50 | sleep 10; | ||
3513 | 51 | # fetch | ||
3514 | 52 | # check extstore counters | ||
3515 | 53 | my $stats = mem_stats($sock); | ||
3516 | 54 | is($stats->{evictions}, 0, 'no RAM evictions'); | ||
3517 | 55 | cmp_ok($stats->{extstore_page_allocs}, '>', 0, 'at least one page allocated'); | ||
3518 | 56 | cmp_ok($stats->{extstore_objects_written}, '>', $keycount / 2, 'some objects written'); | ||
3519 | 57 | cmp_ok($stats->{extstore_bytes_written}, '>', length($value) * 2, 'some bytes written'); | ||
3520 | 58 | cmp_ok($stats->{get_extstore}, '>', 0, 'one object was fetched'); | ||
3521 | 59 | cmp_ok($stats->{extstore_objects_read}, '>', 0, 'one object read'); | ||
3522 | 60 | cmp_ok($stats->{extstore_bytes_read}, '>', length($value), 'some bytes read'); | ||
3523 | 61 | cmp_ok($stats->{extstore_page_reclaims}, '>', 1, 'at least two pages reclaimed'); | ||
3524 | 62 | } | ||
3525 | 63 | |||
3526 | 64 | done_testing(); | ||
3527 | 65 | |||
3528 | 66 | END { | ||
3529 | 67 | unlink $ext_path if $ext_path; | ||
3530 | 68 | unlink $ext_path2 if $ext_path2; | ||
3531 | 69 | } | ||
3532 | diff --git a/t/extstore.t b/t/extstore.t | |||
3533 | index df4ca05..1790a54 100644 | |||
3534 | --- a/t/extstore.t | |||
3535 | +++ b/t/extstore.t | |||
3536 | @@ -17,9 +17,27 @@ if (!supports_extstore()) { | |||
3537 | 17 | 17 | ||
3538 | 18 | $ext_path = "/tmp/extstore.$$"; | 18 | $ext_path = "/tmp/extstore.$$"; |
3539 | 19 | 19 | ||
3541 | 20 | my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_page_count=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path,slab_automove=0"); | 20 | my $server = new_memcached("-m 64 -U 0 -o ext_page_size=8,ext_wbuf_size=2,ext_threads=1,ext_io_depth=2,ext_item_size=512,ext_item_age=2,ext_recache_rate=10000,ext_max_frag=0.9,ext_path=$ext_path:64m,slab_automove=0,ext_compact_under=1"); |
3542 | 21 | my $sock = $server->sock; | 21 | my $sock = $server->sock; |
3543 | 22 | 22 | ||
3544 | 23 | # Wait until all items have flushed | ||
3545 | 24 | sub wait_for_ext { | ||
3546 | 25 | my $target = shift || 0; | ||
3547 | 26 | my $sum = $target + 1; | ||
3548 | 27 | while ($sum > $target) { | ||
3549 | 28 | my $s = mem_stats($sock, "items"); | ||
3550 | 29 | $sum = 0; | ||
3551 | 30 | for my $key (keys %$s) { | ||
3552 | 31 | if ($key =~ m/items:(\d+):number/) { | ||
3553 | 32 | # Ignore classes which can contain extstore items | ||
3554 | 33 | next if $1 < 3; | ||
3555 | 34 | $sum += $s->{$key}; | ||
3556 | 35 | } | ||
3557 | 36 | } | ||
3558 | 37 | sleep 1 if $sum > $target; | ||
3559 | 38 | } | ||
3560 | 39 | } | ||
3561 | 40 | |||
3562 | 23 | my $value; | 41 | my $value; |
3563 | 24 | { | 42 | { |
3564 | 25 | my @chars = ("C".."Z"); | 43 | my @chars = ("C".."Z"); |
3565 | @@ -47,7 +65,7 @@ mem_get_is($sock, "foo", "hi"); | |||
3566 | 47 | print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n"; | 65 | print $sock "set nfoo$_ 0 0 20000 noreply\r\n$value\r\n"; |
3567 | 48 | } | 66 | } |
3568 | 49 | # wait for a flush | 67 | # wait for a flush |
3570 | 50 | sleep 4; | 68 | wait_for_ext(); |
3571 | 51 | # fetch | 69 | # fetch |
3572 | 52 | # TODO: Fetch back all values | 70 | # TODO: Fetch back all values |
3573 | 53 | mem_get_is($sock, "nfoo1", $value); | 71 | mem_get_is($sock, "nfoo1", $value); |
3574 | @@ -83,22 +101,28 @@ mem_get_is($sock, "foo", "hi"); | |||
3575 | 83 | 101 | ||
3576 | 84 | # fill to eviction | 102 | # fill to eviction |
3577 | 85 | { | 103 | { |
3579 | 86 | my $keycount = 3000; | 104 | my $keycount = 4000; |
3580 | 87 | for (1 .. $keycount) { | 105 | for (1 .. $keycount) { |
3581 | 88 | print $sock "set mfoo$_ 0 0 20000 noreply\r\n$value\r\n"; | 106 | print $sock "set mfoo$_ 0 0 20000 noreply\r\n$value\r\n"; |
3582 | 107 | # wait to avoid evictions | ||
3583 | 108 | wait_for_ext(500) if ($_ % 2000 == 0); | ||
3584 | 89 | } | 109 | } |
3586 | 90 | sleep 4; | 110 | # because item_age is set to 2s |
3587 | 111 | wait_for_ext(); | ||
3588 | 91 | my $stats = mem_stats($sock); | 112 | my $stats = mem_stats($sock); |
3589 | 113 | is($stats->{evictions}, 0, 'no evictions'); | ||
3590 | 92 | is($stats->{miss_from_extstore}, 0, 'no misses'); | 114 | is($stats->{miss_from_extstore}, 0, 'no misses'); |
3592 | 93 | mem_get_is($sock, "canary", undef); | 115 | # FIXME: test is flaky; something can rescue the canary because of a race |
3593 | 116 | # condition. might need to roundtrip twice or disable compaction? | ||
3594 | 117 | #mem_get_is($sock, "canary", undef); | ||
3595 | 94 | 118 | ||
3596 | 95 | # check counters | 119 | # check counters |
3597 | 96 | $stats = mem_stats($sock); | 120 | $stats = mem_stats($sock); |
3598 | 97 | cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted'); | 121 | cmp_ok($stats->{extstore_page_evictions}, '>', 0, 'at least one page evicted'); |
3599 | 98 | cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted'); | 122 | cmp_ok($stats->{extstore_objects_evicted}, '>', 0, 'at least one object evicted'); |
3600 | 99 | cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted'); | 123 | cmp_ok($stats->{extstore_bytes_evicted}, '>', 0, 'some bytes evicted'); |
3603 | 100 | is($stats->{extstore_pages_free}, 0, '0 pages are free'); | 124 | cmp_ok($stats->{extstore_pages_free}, '<', 2, 'few pages are free'); |
3604 | 101 | is($stats->{miss_from_extstore}, 1, 'exactly one miss'); | 125 | #is($stats->{miss_from_extstore}, 1, 'exactly one miss'); |
3605 | 102 | 126 | ||
3606 | 103 | # refresh some keys so rescues happen while drop_unread == 1. | 127 | # refresh some keys so rescues happen while drop_unread == 1. |
3607 | 104 | for (1 .. $keycount / 2) { | 128 | for (1 .. $keycount / 2) { |
3608 | @@ -111,6 +135,10 @@ mem_get_is($sock, "foo", "hi"); | |||
3609 | 111 | my $res = <$sock>; | 135 | my $res = <$sock>; |
3610 | 112 | print $sock "extstore max_frag 0\r\n"; | 136 | print $sock "extstore max_frag 0\r\n"; |
3611 | 113 | $res = <$sock>; | 137 | $res = <$sock>; |
3612 | 138 | print $sock "extstore compact_under 4\r\n"; | ||
3613 | 139 | $res = <$sock>; | ||
3614 | 140 | print $sock "extstore drop_under 3\r\n"; | ||
3615 | 141 | $res = <$sock>; | ||
3616 | 114 | for (1 .. $keycount) { | 142 | for (1 .. $keycount) { |
3617 | 115 | next unless $_ % 2 == 0; | 143 | next unless $_ % 2 == 0; |
3618 | 116 | print $sock "delete mfoo$_ noreply\r\n"; | 144 | print $sock "delete mfoo$_ noreply\r\n"; |
3619 | @@ -131,7 +159,7 @@ mem_get_is($sock, "foo", "hi"); | |||
3620 | 131 | for (1 .. $keycount) { | 159 | for (1 .. $keycount) { |
3621 | 132 | print $sock "set bfoo$_ 0 0 20000 noreply\r\n$value\r\n"; | 160 | print $sock "set bfoo$_ 0 0 20000 noreply\r\n$value\r\n"; |
3622 | 133 | } | 161 | } |
3624 | 134 | sleep 4; | 162 | wait_for_ext(); |
3625 | 135 | 163 | ||
3626 | 136 | # incr should be blocked. | 164 | # incr should be blocked. |
3627 | 137 | print $sock "incr bfoo1 1\r\n"; | 165 | print $sock "incr bfoo1 1\r\n"; |
3628 | @@ -139,7 +167,7 @@ mem_get_is($sock, "foo", "hi"); | |||
3629 | 139 | 167 | ||
3630 | 140 | # append/prepend *could* work, but it would require pulling the item back in. | 168 | # append/prepend *could* work, but it would require pulling the item back in. |
3631 | 141 | print $sock "append bfoo1 0 0 2\r\nhi\r\n"; | 169 | print $sock "append bfoo1 0 0 2\r\nhi\r\n"; |
3633 | 142 | is(scalar <$sock>, "NOT_STORED\r\n", 'append falis'); | 170 | is(scalar <$sock>, "NOT_STORED\r\n", 'append fails'); |
3634 | 143 | print $sock "prepend bfoo1 0 0 2\r\nhi\r\n"; | 171 | print $sock "prepend bfoo1 0 0 2\r\nhi\r\n"; |
3635 | 144 | is(scalar <$sock>, "NOT_STORED\r\n", 'prepend fails'); | 172 | is(scalar <$sock>, "NOT_STORED\r\n", 'prepend fails'); |
3636 | 145 | } | 173 | } |
3637 | diff --git a/t/flush-all.t b/t/flush-all.t | |||
3638 | index e30d819..d9c536a 100755 | |||
3639 | --- a/t/flush-all.t | |||
3640 | +++ b/t/flush-all.t | |||
3641 | @@ -40,7 +40,7 @@ is(scalar <$sock>, "OK\r\n", "did flush_all in future"); | |||
3642 | 40 | print $sock "set foo 0 0 4\r\n1234\r\n"; | 40 | print $sock "set foo 0 0 4\r\n1234\r\n"; |
3643 | 41 | is(scalar <$sock>, "STORED\r\n", "stored foo = '1234'"); | 41 | is(scalar <$sock>, "STORED\r\n", "stored foo = '1234'"); |
3644 | 42 | mem_get_is($sock, "foo", '1234'); | 42 | mem_get_is($sock, "foo", '1234'); |
3646 | 43 | sleep(3); | 43 | sleep(5); |
3647 | 44 | mem_get_is($sock, "foo", undef); | 44 | mem_get_is($sock, "foo", undef); |
3648 | 45 | 45 | ||
3649 | 46 | print $sock "set foo 0 0 5\r\n12345\r\n"; | 46 | print $sock "set foo 0 0 5\r\n12345\r\n"; |
3650 | diff --git a/t/issue_67.t b/t/issue_67.t | |||
3651 | index b2d374f..1dbaba2 100644 | |||
3652 | --- a/t/issue_67.t | |||
3653 | +++ b/t/issue_67.t | |||
3654 | @@ -1,11 +1,12 @@ | |||
3655 | 1 | #!/usr/bin/perl | 1 | #!/usr/bin/perl |
3656 | 2 | 2 | ||
3657 | 3 | use strict; | 3 | use strict; |
3659 | 4 | use Test::More tests => 22; | 4 | use Test::More tests => 24; |
3660 | 5 | use FindBin qw($Bin); | 5 | use FindBin qw($Bin); |
3661 | 6 | use lib "$Bin/lib"; | 6 | use lib "$Bin/lib"; |
3662 | 7 | use MemcachedTest; | 7 | use MemcachedTest; |
3663 | 8 | use Carp qw(croak); | 8 | use Carp qw(croak); |
3664 | 9 | use Socket qw(sockaddr_in INADDR_ANY PF_INET SOCK_STREAM); | ||
3665 | 9 | 10 | ||
3666 | 10 | use Cwd; | 11 | use Cwd; |
3667 | 11 | my $builddir = getcwd; | 12 | my $builddir = getcwd; |
3668 | @@ -29,12 +30,25 @@ sub validate_port { | |||
3669 | 29 | if ($expected == -1) { | 30 | if ($expected == -1) { |
3670 | 30 | ok(!defined($got), "$name expected no port, got $got"); | 31 | ok(!defined($got), "$name expected no port, got $got"); |
3671 | 31 | } elsif ($expected == 0) { | 32 | } elsif ($expected == 0) { |
3673 | 32 | ok($got != 11211, "$name expected random port (got $got)"); | 33 | ok(defined($got) && $got != 11211, "$name expected random port (got $got)"); |
3674 | 33 | } else { | 34 | } else { |
3675 | 34 | is($got, $expected, "$name"); | 35 | is($got, $expected, "$name"); |
3676 | 35 | } | 36 | } |
3677 | 36 | } | 37 | } |
3678 | 37 | 38 | ||
3679 | 39 | sub skip_if_default_addr_in_use(&) { | ||
3680 | 40 | my ($block) = @_; | ||
3681 | 41 | |||
3682 | 42 | socket(my $socket, PF_INET, SOCK_STREAM, 0) or die $!; | ||
3683 | 43 | my $addr_in_use = !bind($socket, sockaddr_in(11211, INADDR_ANY)); | ||
3684 | 44 | close($socket); | ||
3685 | 45 | |||
3686 | 46 | SKIP: { | ||
3687 | 47 | skip 'Default address is in use. Do you have a running instance?', 2 if $addr_in_use; | ||
3688 | 48 | return $block->(); | ||
3689 | 49 | } | ||
3690 | 50 | } | ||
3691 | 51 | |||
3692 | 38 | sub run_server { | 52 | sub run_server { |
3693 | 39 | my ($args) = @_; | 53 | my ($args) = @_; |
3694 | 40 | 54 | ||
3695 | @@ -75,14 +89,13 @@ sub when { | |||
3696 | 75 | validate_port($name, $ports{'UDP INET'}, $expected_udp); | 89 | validate_port($name, $ports{'UDP INET'}, $expected_udp); |
3697 | 76 | } | 90 | } |
3698 | 77 | 91 | ||
3701 | 78 | # Disabling the defaults since it conflicts with a running instance. | 92 | skip_if_default_addr_in_use { when('no arguments', '', 11211, -1) }; |
3700 | 79 | # when('no arguments', '', 11211, 11211); | ||
3702 | 80 | when('specifying tcp port', '-p 11212', 11212, -1); | 93 | when('specifying tcp port', '-p 11212', 11212, -1); |
3703 | 81 | when('specifying udp port', '-U 11222', 11222, 11222); | 94 | when('specifying udp port', '-U 11222', 11222, 11222); |
3705 | 82 | when('specifying tcp ephemeral port', '-p -1', 0, 0); | 95 | when('specifying tcp ephemeral port', '-p -1', 0, -1); |
3706 | 83 | when('specifying udp ephemeral port', '-U -1', 0, 0); | 96 | when('specifying udp ephemeral port', '-U -1', 0, 0); |
3707 | 84 | when('tcp port disabled', '-p 0', -1, -1); | 97 | when('tcp port disabled', '-p 0', -1, -1); |
3709 | 85 | when('udp port disabled', '-U 0', 11211, -1); | 98 | skip_if_default_addr_in_use { when('udp port disabled', '-U 0', 11211, -1) }; |
3710 | 86 | when('specifying tcp and udp ports', '-p 11232 -U 11233', 11232, 11233); | 99 | when('specifying tcp and udp ports', '-p 11232 -U 11233', 11232, 11233); |
3711 | 87 | when('specifying tcp and disabling udp', '-p 11242 -U 0', 11242, -1); | 100 | when('specifying tcp and disabling udp', '-p 11242 -U 0', 11242, -1); |
3712 | 88 | when('specifying udp and disabling tcp', '-p -1 -U 11252', 0, 11252); | 101 | when('specifying udp and disabling tcp', '-p -1 -U 11252', 0, 11252); |
3713 | diff --git a/t/lib/MemcachedTest.pm b/t/lib/MemcachedTest.pm | |||
3714 | index 4e1da66..416daaf 100644 | |||
3715 | --- a/t/lib/MemcachedTest.pm | |||
3716 | +++ b/t/lib/MemcachedTest.pm | |||
3717 | @@ -14,13 +14,33 @@ my $builddir = getcwd; | |||
3718 | 14 | my @unixsockets = (); | 14 | my @unixsockets = (); |
3719 | 15 | 15 | ||
3720 | 16 | @EXPORT = qw(new_memcached sleep mem_get_is mem_gets mem_gets_is mem_stats | 16 | @EXPORT = qw(new_memcached sleep mem_get_is mem_gets mem_gets_is mem_stats |
3722 | 17 | supports_sasl free_port supports_drop_priv supports_extstore); | 17 | supports_sasl free_port supports_drop_priv supports_extstore |
3723 | 18 | wait_ext_flush); | ||
3724 | 18 | 19 | ||
3725 | 19 | sub sleep { | 20 | sub sleep { |
3726 | 20 | my $n = shift; | 21 | my $n = shift; |
3727 | 21 | select undef, undef, undef, $n; | 22 | select undef, undef, undef, $n; |
3728 | 22 | } | 23 | } |
3729 | 23 | 24 | ||
3730 | 25 | # Wait until all items have flushed | ||
3731 | 26 | sub wait_ext_flush { | ||
3732 | 27 | my $sock = shift; | ||
3733 | 28 | my $target = shift || 0; | ||
3734 | 29 | my $sum = $target + 1; | ||
3735 | 30 | while ($sum > $target) { | ||
3736 | 31 | my $s = mem_stats($sock, "items"); | ||
3737 | 32 | $sum = 0; | ||
3738 | 33 | for my $key (keys %$s) { | ||
3739 | 34 | if ($key =~ m/items:(\d+):number/) { | ||
3740 | 35 | # Ignore classes which can contain extstore items | ||
3741 | 36 | next if $1 < 3; | ||
3742 | 37 | $sum += $s->{$key}; | ||
3743 | 38 | } | ||
3744 | 39 | } | ||
3745 | 40 | sleep 1 if $sum > $target; | ||
3746 | 41 | } | ||
3747 | 42 | } | ||
3748 | 43 | |||
3749 | 24 | sub mem_stats { | 44 | sub mem_stats { |
3750 | 25 | my ($sock, $type) = @_; | 45 | my ($sock, $type) = @_; |
3751 | 26 | $type = $type ? " $type" : ""; | 46 | $type = $type ? " $type" : ""; |
3752 | diff --git a/t/lru-crawler.t b/t/lru-crawler.t | |||
3753 | index 18375c2..4e94fb3 100644 | |||
3754 | --- a/t/lru-crawler.t | |||
3755 | +++ b/t/lru-crawler.t | |||
3756 | @@ -2,7 +2,7 @@ | |||
3757 | 2 | 2 | ||
3758 | 3 | use strict; | 3 | use strict; |
3759 | 4 | use warnings; | 4 | use warnings; |
3761 | 5 | use Test::More tests => 221; | 5 | use Test::More tests => 222; |
3762 | 6 | use FindBin qw($Bin); | 6 | use FindBin qw($Bin); |
3763 | 7 | use lib "$Bin/lib"; | 7 | use lib "$Bin/lib"; |
3764 | 8 | use MemcachedTest; | 8 | use MemcachedTest; |
3765 | @@ -60,6 +60,18 @@ while (1) { | |||
3766 | 60 | is($items->{"items:1:crawler_reclaimed"}, 30, "slab1 has 30 reclaims"); | 60 | is($items->{"items:1:crawler_reclaimed"}, 30, "slab1 has 30 reclaims"); |
3767 | 61 | } | 61 | } |
3768 | 62 | 62 | ||
3769 | 63 | # Check that crawler metadump works correctly. | ||
3770 | 64 | { | ||
3771 | 65 | print $sock "lru_crawler metadump all\r\n"; | ||
3772 | 66 | my $count = 0; | ||
3773 | 67 | while (<$sock>) { | ||
3774 | 68 | last if /^(\.|END)/; | ||
3775 | 69 | /^(key=) (\S+).*([^\r\n]+)/; | ||
3776 | 70 | $count++; | ||
3777 | 71 | } | ||
3778 | 72 | is ($count, 60); | ||
3779 | 73 | } | ||
3780 | 74 | |||
3781 | 63 | for (1 .. 30) { | 75 | for (1 .. 30) { |
3782 | 64 | mem_get_is($sock, "ifoo$_", "ok"); | 76 | mem_get_is($sock, "ifoo$_", "ok"); |
3783 | 65 | mem_get_is($sock, "lfoo$_", "ok"); | 77 | mem_get_is($sock, "lfoo$_", "ok"); |
3784 | diff --git a/t/lru-maintainer.t b/t/lru-maintainer.t | |||
3785 | index ac2ba7b..04fa76a 100644 | |||
3786 | --- a/t/lru-maintainer.t | |||
3787 | +++ b/t/lru-maintainer.t | |||
3788 | @@ -56,6 +56,18 @@ for (my $key = 0; $key < 100; $key++) { | |||
3789 | 56 | # Items need two fetches to become active | 56 | # Items need two fetches to become active |
3790 | 57 | mem_get_is($sock, "canary", $value); | 57 | mem_get_is($sock, "canary", $value); |
3791 | 58 | mem_get_is($sock, "canary", $value); | 58 | mem_get_is($sock, "canary", $value); |
3792 | 59 | $stats = mem_stats($sock); | ||
3793 | 60 | # The maintainer thread needs to juggle a bit to actually rescue an | ||
3794 | 61 | # item. If it's slow we could evict after resuming setting. | ||
3795 | 62 | sleep 1; | ||
3796 | 63 | for (0..4) { | ||
3797 | 64 | my $s2 = mem_stats($sock); | ||
3798 | 65 | if ($s2->{lru_maintainer_juggles} - $stats->{lru_maintainer_juggles} < 5) { | ||
3799 | 66 | sleep 1; | ||
3800 | 67 | next; | ||
3801 | 68 | } | ||
3802 | 69 | last; | ||
3803 | 70 | } | ||
3804 | 59 | } | 71 | } |
3805 | 60 | print $sock "set key$key 0 0 66560\r\n$value\r\n"; | 72 | print $sock "set key$key 0 0 66560\r\n$value\r\n"; |
3806 | 61 | is(scalar <$sock>, "STORED\r\n", "stored key$key"); | 73 | is(scalar <$sock>, "STORED\r\n", "stored key$key"); |
3807 | diff --git a/t/misbehave.t b/t/misbehave.t | |||
3808 | 62 | old mode 100644 | 74 | old mode 100644 |
3809 | 63 | new mode 100755 | 75 | new mode 100755 |
3810 | index ccf88f5..13cb7f3 | |||
3811 | --- a/t/misbehave.t | |||
3812 | +++ b/t/misbehave.t | |||
3813 | @@ -3,6 +3,7 @@ | |||
3814 | 3 | use strict; | 3 | use strict; |
3815 | 4 | use Test::More; | 4 | use Test::More; |
3816 | 5 | use FindBin qw($Bin); | 5 | use FindBin qw($Bin); |
3817 | 6 | use Socket qw(MSG_PEEK MSG_DONTWAIT); | ||
3818 | 6 | use lib "$Bin/lib"; | 7 | use lib "$Bin/lib"; |
3819 | 7 | use MemcachedTest; | 8 | use MemcachedTest; |
3820 | 8 | 9 | ||
3821 | @@ -13,8 +14,15 @@ if (supports_drop_priv()) { | |||
3822 | 13 | exit 0; | 14 | exit 0; |
3823 | 14 | } | 15 | } |
3824 | 15 | 16 | ||
3826 | 16 | my $server = new_memcached(); | 17 | my $server = new_memcached('-o drop_privileges'); |
3827 | 17 | my $sock = $server->sock; | 18 | my $sock = $server->sock; |
3828 | 18 | 19 | ||
3829 | 19 | print $sock "misbehave\r\n"; | 20 | print $sock "misbehave\r\n"; |
3831 | 20 | is(scalar <$sock>, "OK\r\n", "did not allow misbehaving"); | 21 | sleep(1); |
3832 | 22 | |||
3833 | 23 | # check if the socket is dead now | ||
3834 | 24 | my $buff; | ||
3835 | 25 | my $ret = recv($sock, $buff, 1, MSG_PEEK | MSG_DONTWAIT); | ||
3836 | 26 | is($ret, undef, "did not allow misbehaving"); | ||
3837 | 27 | |||
3838 | 28 | $server->DESTROY(); | ||
3839 | diff --git a/thread.c b/thread.c | |||
3840 | index d17a387..618ffac 100644 | |||
3841 | --- a/thread.c | |||
3842 | +++ b/thread.c | |||
3843 | @@ -139,22 +139,24 @@ void pause_threads(enum pause_thread_types type) { | |||
3844 | 139 | buf[0] = 0; | 139 | buf[0] = 0; |
3845 | 140 | switch (type) { | 140 | switch (type) { |
3846 | 141 | case PAUSE_ALL_THREADS: | 141 | case PAUSE_ALL_THREADS: |
3847 | 142 | lru_maintainer_pause(); | ||
3848 | 143 | slabs_rebalancer_pause(); | 142 | slabs_rebalancer_pause(); |
3849 | 143 | lru_maintainer_pause(); | ||
3850 | 144 | lru_crawler_pause(); | 144 | lru_crawler_pause(); |
3851 | 145 | #ifdef EXTSTORE | 145 | #ifdef EXTSTORE |
3852 | 146 | storage_compact_pause(); | 146 | storage_compact_pause(); |
3853 | 147 | storage_write_pause(); | ||
3854 | 147 | #endif | 148 | #endif |
3855 | 148 | case PAUSE_WORKER_THREADS: | 149 | case PAUSE_WORKER_THREADS: |
3856 | 149 | buf[0] = 'p'; | 150 | buf[0] = 'p'; |
3857 | 150 | pthread_mutex_lock(&worker_hang_lock); | 151 | pthread_mutex_lock(&worker_hang_lock); |
3858 | 151 | break; | 152 | break; |
3859 | 152 | case RESUME_ALL_THREADS: | 153 | case RESUME_ALL_THREADS: |
3860 | 153 | lru_maintainer_resume(); | ||
3861 | 154 | slabs_rebalancer_resume(); | 154 | slabs_rebalancer_resume(); |
3862 | 155 | lru_maintainer_resume(); | ||
3863 | 155 | lru_crawler_resume(); | 156 | lru_crawler_resume(); |
3864 | 156 | #ifdef EXTSTORE | 157 | #ifdef EXTSTORE |
3865 | 157 | storage_compact_resume(); | 158 | storage_compact_resume(); |
3866 | 159 | storage_write_resume(); | ||
3867 | 158 | #endif | 160 | #endif |
3868 | 159 | case RESUME_WORKER_THREADS: | 161 | case RESUME_WORKER_THREADS: |
3869 | 160 | pthread_mutex_unlock(&worker_hang_lock); | 162 | pthread_mutex_unlock(&worker_hang_lock); |
3870 | @@ -625,7 +627,7 @@ void item_unlink(item *item) { | |||
3871 | 625 | * Does arithmetic on a numeric item value. | 627 | * Does arithmetic on a numeric item value. |
3872 | 626 | */ | 628 | */ |
3873 | 627 | enum delta_result_type add_delta(conn *c, const char *key, | 629 | enum delta_result_type add_delta(conn *c, const char *key, |
3875 | 628 | const size_t nkey, int incr, | 630 | const size_t nkey, bool incr, |
3876 | 629 | const int64_t delta, char *buf, | 631 | const int64_t delta, char *buf, |
3877 | 630 | uint64_t *cas) { | 632 | uint64_t *cas) { |
3878 | 631 | enum delta_result_type ret; | 633 | enum delta_result_type ret; |
3879 | diff --git a/timedrun.c b/timedrun.c | |||
3880 | index 6888453..a869004 100644 | |||
3881 | --- a/timedrun.c | |||
3882 | +++ b/timedrun.c | |||
3883 | @@ -7,11 +7,11 @@ | |||
3884 | 7 | 7 | ||
3885 | 8 | #include <assert.h> | 8 | #include <assert.h> |
3886 | 9 | 9 | ||
3888 | 10 | static int caught = 0; | 10 | volatile sig_atomic_t caught_sig = 0; |
3889 | 11 | 11 | ||
3891 | 12 | static void caught_signal(int which) | 12 | static void signal_handler(int which) |
3892 | 13 | { | 13 | { |
3894 | 14 | caught = which; | 14 | caught_sig = which; |
3895 | 15 | } | 15 | } |
3896 | 16 | 16 | ||
3897 | 17 | static int wait_for_process(pid_t pid) | 17 | static int wait_for_process(pid_t pid) |
3898 | @@ -21,7 +21,7 @@ static int wait_for_process(pid_t pid) | |||
3899 | 21 | int i = 0; | 21 | int i = 0; |
3900 | 22 | struct sigaction sig_handler; | 22 | struct sigaction sig_handler; |
3901 | 23 | 23 | ||
3903 | 24 | sig_handler.sa_handler = caught_signal; | 24 | sig_handler.sa_handler = signal_handler; |
3904 | 25 | sig_handler.sa_flags = 0; | 25 | sig_handler.sa_flags = 0; |
3905 | 26 | 26 | ||
3906 | 27 | sigaction(SIGALRM, &sig_handler, NULL); | 27 | sigaction(SIGALRM, &sig_handler, NULL); |
3907 | @@ -44,8 +44,8 @@ static int wait_for_process(pid_t pid) | |||
3908 | 44 | switch (i) { | 44 | switch (i) { |
3909 | 45 | case 0: | 45 | case 0: |
3910 | 46 | /* On the first iteration, pass the signal through */ | 46 | /* On the first iteration, pass the signal through */ |
3913 | 47 | sig = caught > 0 ? caught : SIGTERM; | 47 | sig = caught_sig > 0 ? caught_sig : SIGTERM; |
3914 | 48 | if (caught == SIGALRM) { | 48 | if (caught_sig == SIGALRM) { |
3915 | 49 | fprintf(stderr, "Timeout.. killing the process\n"); | 49 | fprintf(stderr, "Timeout.. killing the process\n"); |
3916 | 50 | } | 50 | } |
3917 | 51 | break; | 51 | break; |
3918 | diff --git a/version.m4 b/version.m4 | |||
3919 | index 4a02156..ff2636d 100644 | |||
3920 | --- a/version.m4 | |||
3921 | +++ b/version.m4 | |||
3922 | @@ -1 +1 @@ | |||
3924 | 1 | m4_define([VERSION_NUMBER], [1.5.6]) | 1 | m4_define([VERSION_NUMBER], [1.5.10]) |
A PPA build should be available in ppa:racb/ experimental shortly.