Author: Alex Rousskov
Author Date: 2021-01-17 06:49:45 UTC

Fix memory allocation for from-server network reads (#736)

Use tcp_recv_bufsize (not read_ahead_gap) for the response socket
read(2) buffer. This change may increase default Squid memory footprint
by ~50KB per to-server connection in most environments and decrease
Squid memory footprint by approximately (read_ahead_gap - 64KB) in many
performance-sensitive deployments that run with an n*MB read_ahead_gap.
The admin now has better control over memory usage and I/O performance.
Using tcp_recv_bufsize for kernel and Squid receive buffer is natural
and simple.

HttpStateData::inBuf is used for socket read(2) when receiving data from
an origin server or cache peer. The buffer does _not_ accumulate content
between reads except when parsing response headers and chunked encoding
metadata. In fact, the buffer is usually empty between reads. Squid
accumulates response content in Store or adaptation BodyPipe instead.

Squid was incorrectly using a mostly unrelated read_ahead_gap directive
to size inBuf. The default read_ahead_gap value is 16KB. Admins increase
read_ahead_gap because accumulating more response data often improves
overall Squid performance or user satisfaction. Large increases caused
excessive (and unpooled!) inBuf memory allocations. In some cases, these
memory allocations blocked the entire worker for a few milliseconds!

We could just replace Config.readAheadGap with Config.tcpRcvBufsz (or
another directive) when calculating inBuf capacity, ignoring other
problems in the surrounding code, but that would make it impractical to
tell the admins what this change really means because there would be too
many exceptions where Squid would still not behave as (re)configured.

* Small read_ahead_gap values stalled response header and body parsing.

* Squid was deferring response reads when inBuf became full due to
  reconfiguration but deferring does not always reduce buffer size.

* Squid was not always deferring response reads when running out of
  buffer space, stalling some transactions.

* 64-bit configuration values could overflow int-based local variables.

* inBuf was grown earlier than the code wanted to grow it (before the
  socket became ready for reading) and not grown when the new/grown
  space was actually needed/used (just before reading from the socket).

* Transactions killed by a Must() failure when a reconfiguration made
  inBuf "full" just before read(2) in HttpStateData::readReply().

* Wrong/misleading/stale comments/debugs documenting key decisions.

SBuf implementation and parsing API exacerbate the bug by forcing memory
re-allocations for inBuf even when the underlying buffer storage has
enough space (and often is completely empty!). We will address these
problems separately, but this fix reduces their negative impact.

Also do not wait for the rest of the HTTP response header or chunk
metadata if we have no space to store more incoming bytes. The parser
does not know how much more buffer space we have (if any). It is our
responsibility to check that some progress is still possible.

I replaced "Non-HTTP-compliant" with "cannot parse" because we do not
know whether the parsing failure is related to HTTP compliance. For
example, buffer space limitations are not a compliance-related problem.

The changes near wasThereAnException look wrong, but it is the existing
wasThereAnException name that is "reversed". Compiled code should be OK.

Also fixed commit 1ad6851 to delay buffer allocations until we are ready
to read from the server. That old commit made buffer allocations in
maybeMakeSpaceAvailable() optional so that they can be delayed until we
actually read from the peer socket. Delaying buffer allocation until
read(2) is the right thing to do. The callers (correctly) passed a
boolean flag to trigger the allocation, but the method interpreted the
flag as if it meant "do not allocate". That is, commit tried to delay
allocations but actually allocated early.

Why was the 1ad6851 bug unnoticed since 2015? The outside observers
could not see the problem because Squid continued to allocate the
buffer, and there was no code that would fiddle with that buffer while
Squid was waiting for the socket to become ready for reading (unlike bug
4206 that served as an inspiration for commit 1ad6851).

Also fixed "read_ahead_gap none" (i.e. unlimited gap) handling that was
broken since commit 5791270 (at least). I am not sure we should support
this -- the admin can just give a large positive value -- but now we do.

Also improved reply_header_max_size validation to help adjust
(incorrect) admin assumptions that Squid can handle response headers
exceeding SBug::maxSize.

Also improved tcp_recv_bufsize/read_ahead_gap documentation, validation.

Author: David Carlier
Author Date: 2021-01-15 15:41:47 UTC

Profiling: CPU timing implemented for MAC non-x86 (#757)

Author: Alex Rousskov
Author Date: 2020-11-10 21:42:18 UTC

Transactions exceeding client_lifetime are logged as _ABORTED (#748)

... rather than timed out (_TIMEOUT).

To record the right cause of death, we have to call terminateAll()
rather than setting logType.err.timedout directly. Otherwise, when
ConnStateData::swanSong() calls terminateAll(0), it overwrites our
direct setting.

Author: eduard-bagdasaryan
Author Date: 2020-08-12 22:47:14 UTC

Ignore SMP queue responses made stale by worker restarts (#711)

When a worker restarts (for any reason), the disker-to-worker queue may
contain disker responses to I/O requests sent by the previous
incarnation of the restarted worker process (the "previous generation"
responses). Since the current response:request mapping mechanism relies
on a 32-bit integer counter, and a worker process always starts counting
from 0, there is a small chance that the restarted worker may see a
previous generation response that accidentally matches the current
generation request ID.

For writing transactions, accepting a previous generation response may
mean unlocking a cache entry too soon, making not yet written slots
available to other workers that might read wrong content. For reading
transactions, accepting a previous generation response may mean
immediately serving wrong response content (that have been already
overwritten on disk with the information that the restarted worker is
now waiting for).

To avoid these problems, each disk I/O request now stores the worker
process ID. Workers ignore responses to requests originated by a
different/mismatching worker ID.

Author: Amos Jeffries
Author Date: 2020-08-16 02:36:21 UTC

Merge branch 'master' into yadij-patch-1

Author: Armin Wolfermann
Author Date: 2020-02-04 20:15:00 UTC

fix security patch

Author: Francesco Chemolli
Author Date: 2018-10-21 13:06:33 UTC

Mingw build fixes

There are type errors and data access errors in windows-specific bits of
code.

Author: Garri Djavadyan
Author Date: 2016-12-17 13:56:49 UTC

Bug 4169: HIT marked as MISS when If-None-Match does not match

Author: Garri Djavadyan
Author Date: 2016-12-17 12:59:41 UTC

Bug 4169: HIT marked as MISS when If-None-Match does not match

Author: Garri Djavadyan
Author Date: 2016-12-17 10:04:42 UTC

Bug 4169: HIT marked as MISS when If-None-Match does not match

Author: Amos Jeffries
Author Date: 2016-12-17 06:37:44 UTC

Do not override user defined -std option

Author: Source Maintenance
Author Date: 2015-04-23 13:02:10 UTC

Docs: Update CONTRIBUTORS

Author: Amos Jeffries
Author Date: 2012-08-08 08:16:36 UTC

merged from trunk

Author: serassio <>
Author Date: 2008-03-02 23:54:50 UTC

Windows port: Add reconfigure action to cachemgr.

Author: robertc <>
Author Date: 2002-10-24 20:53:10 UTC

cbdata debug history