Merge lp:~martin-v/zodb/ipv6 into lp:zodb

[Martin v. Löwis]

This adds IPv6 support to ZEO. It relies on the ZConfig changes in https://code.launchpad.net/~martin-v/zconfig/ipv6/+merge/35944.

On the server side, the major change is the interpretation of listen addresses. '' now indicates wild-card-IPv4-and-IPv6. For wildcard IPv4, 0.0.0.0 needs to be specified (which used to work before as well), for wildcard IPv6, :: must be used. On systems that don't support dual-stack sockets, '' continues to mean 0.0.0.0. To bind ZEO explicitly to an IPv6 address, [addr]:port needs to be used.

On the client side, there are two major changes:

- IPv6 addresses can now be specified, using [addr]:port (or ('addr', port) in the API).

- DNS usage has changed. The client RPC now looks up all addresses, in the order recommended by the operating system (i.e. through getaddrinfo). The resulting list is considered a sublist of the regular fail-over address list, i.e. all addresses are being contacted simultaneously, and the first one connecting is used. This was discussed with Christian Theune at the DZUG sprint.

As tests, the connection tests now include IPv6 versions of all tests.

The only other change to the tests was a doctest which now prints 127.0.0.1 instead of localhost.

[lp:~martin-v/zodb/ipv6]

3957. By Martin v. Löwis on 2010-09-19

Remove debugging code.

3958. By Martin v. Löwis on 2010-09-19

Only check for IPPROTO_IPV6 if socket creation succeeded.
Look for AttributeError, not NameError, when checking IPPROTO_IPV6 presence.

[Jim Fulton (jim-zope)]

> - DNS usage has changed. The client RPC now looks up all addresses, in
> the order recommended by the operating system (i.e. through
> getaddrinfo). The resulting list is considered a sublist of the
> regular fail-over address list, i.e. all addresses are being
> contacted simultaneously, and the first one connecting is used. This
> was discussed with Christian Theune at the DZUG sprint.

What is the rational for this?

Why would a user want this?

[Jim Fulton (jim-zope)]

I should note that I have limited confidence in the thread-tastic
abstraction-astronomic implementation of the client-storage connection
logic. :) I plan to reimplement it with a simpler event-based approach
and, in the mean time, an leery of asking it to work harder.

[Jim Fulton (jim-zope)]

I'm fanatical about source lines being less than or equal to 80
characters in length. :)

[Martin v. Löwis (martin-v)]

Re: DNS. Users currently configure DNS names in zrpc clients to contact ZEO servers, so I guess there is a rationale for that (i.e. users want to use names instead of addresses - in particular with IPv6).

Now, if you allow DNS names, the question is what lookups should you do. The common approach is to look for both AAAA and A, and process them in the order in which the operating system returns them. This is what all client software supporting IPv6 does, so it's also what users expect it to do.

So now that we have one or more IPv6 addresses per name, and one or more IPv4 addresses, which ones should we use? zrpc currently supports connecting to multiple addresses simultaneously, and then taking the one that comes back first. With multiple addresses coming out of DNS, it can speed-up connection establishment when they are tried simultaneously. There is also precedence in other client software doing this, e.g. Apple Safari. As you might have to test each address sequentially, you can just as well do so in parallel.

In addition, it was easier to implement that way. If you think it's undesirable, I can try to come up with a way to do them sequentially.

[Martin v. Löwis (martin-v)]

Re: line length. Will redo.

[Jim Fulton (jim-zope)]

Meta comment: I'm new to IPv6, so if I ask dumb questions or say dumb
things, please be patient. :)

> Re: DNS. Users currently configure DNS names in zrpc clients to contact ZEO
> servers, so I guess there is a rationale for that (i.e. users want to use
> names instead of addresses - in particular with IPv6).

Yup. +1 on supporting names. :)

> Now, if you allow DNS names, the question is what lookups should you do. The
> common approach is to look for both AAAA and A, and process them in the order
> in which the operating system returns them. This is what all client software
> supporting IPv6 does, so it's also what users expect it to do.

Maybe I'm missinterpreting what you mean by "process".

I'd be surprised is "all client software" tried each address returned.
I would expect most client software to simply use an address in bind
or connect and get watever address is prefered for the type of socket
they created.

> So now that we have one or more IPv6 addresses per name, and one or more IPv4
> addresses, which ones should we use? zrpc currently supports connecting to
> multiple addresses simultaneously, and then taking the one that comes back
> first.

It's slightly more complicated, but yeah. :)

> With multiple addresses coming out of DNS, it can speed-up connection
> establishment when they are tried simultaneously.

I'm not sure what you mean here. But yeah, if you're going to do
multiple things, doing them in parallel is generally a good strategy.

> There is also precedence in
> other client software doing this, e.g. Apple Safari. As you might have to test
> each address sequentially, you can just as well do so in parallel.
>
> In addition, it was easier to implement that way. If you think it's
> undesirable, I can try to come up with a way to do them
> sequentially.

I'm not clear on what was easier to implement.

I'll try to guess. :)

I see a number of possible issues that this might try to solve.

1. Given a host name, you have to decide whether to bind or connect
   using a v4 or v6 socket.

2. If multiple machines have the same host name, maybe you want to
   treat them as a high-availability pool and connect to whichever one
   is providing the desired service.

   Is there any other reason why there would be multiple v4 addresses
   or multiple v6 addresses for the same host name. (Leaving aside a
   host having obe v4 address and one v6 address)?

Are these the problems you're trying to solve?

For 1, it would be just as easy to grab the first result from
getaddrinfo.

BTW, the patch for the server always uses a v4 socket if a host name
is given, but perhaps I don't understand how v6 interacts with bind.

For 2, I can see real benefits in giving multiple redundant servers
the same host name.

What I'm worried about having clients connect to the same server more
than once. There are 2 reasons for this:

1. It becomes ambiguous whether a v4 or v6 socket will be used.

2. As I mentioned earlier, the ZEO client connection logic has issues.
   We use ZRS to provide a primary secondary replication scheme.
   We used to have primaries and secondaries listen on the same port
   and apps that wanted writable connections would eit...

[Jim Fulton (jim-zope)]

How did you see my comments? Did you get email, or did you poll?

I didn't see your replies until I checked back.

[Martin v. Löwis (martin-v)]

> Maybe I'm missinterpreting what you mean by "process".
>
> I'd be surprised is "all client software" tried each address returned.
> I would expect most client software to simply use an address in bind
> or connect and get watever address is prefered for the type of socket
> they created.

With IPv6, it is now common and recommended that each host gets at least
two addresses: one for IPv4 and one for IPv6 (dual stack). In addition,
some systems have multiple IPv6 addresses (e.g. assigned by various
tunneling systems). It's not universally clear that any of these
addresses is better than any other: it depends, in particular, on the
client connectivity (IPv6 available or not, IPv4 available or not).

The new resolver API recommended for use it getaddrinfo(3). It gives you
a list of addrinfo(family, socktype, proto, canonname, addr), mixing
IPv4 and IPv6 addresses. Today, systems have a policy (e.g.
/etc/gai.conf) to determine in which order to return them (e.g. IPv6
global first, then IPv4, then IPv6 tunnel). Applications must not
normally chose for themselves whether they want IPv4 or IPv6
connections, but should let the system decide.

In turn, you can't actually create the socket until getaddrinfo
returns, as it will tell you what socket family to use. Hence
the typically connect code looks like this:

  for family,address in getaddrinfo():
     try:
         s = socket.socket(family)
         s.connect(address, port)
     except socket.error:
         continue
  if not s:
     raise "None of these addresses worked"

The IPv6 server code is entirely different: typically, you either
bind to all addresses (v4 and v6) of a system (preferably using
dual-stack sockets using a wildcard address), or you bind to a (list
of) specifically configured addresses; no DNS involved. When binding
to a list of addresses, you need to create multiple server sockets.
IIUC, ZEO currently doesn't support that (and IMO doesn't need to).

>> With multiple addresses coming out of DNS, it can speed-up connection
>> establishment when they are tried simultaneously.
>
> I'm not sure what you mean here. But yeah, if you're going to do
> multiple things, doing them in parallel is generally a good strategy.

See above: the for loop can run in parallel.

>> In addition, it was easier to implement that way. If you think it's
>> undesirable, I can try to come up with a way to do them
>> sequentially.
>
> I'm not clear on what was easier to implement.
>
> I'll try to guess. :)
>
> I see a number of possible issues that this might try to solve.
>
> 1. Given a host name, you have to decide whether to bind or connect
> using a v4 or v6 socket.

No, you would need to iterate over all getaddrinfo results either way.

> 2. If multiple machines have the same host name, maybe you want to
> treat them as a high-availability pool and connect to whichever one
> is providing the desired service.

That is now possible, but only a side effect. But yes, it was easier
to treat a list of addresses returned from DNS as a high-availability
pool - easier than modifying ConnectWrapper to execute the loop.

> Is there any other reason why there would be multiple v4 addresses
> ...

[Martin v. Löwis (martin-v)]

Am 27.09.2010 19:01, schrieb Jim Fulton:
> How did you see my comments? Did you get email, or did you poll?

I got an email, and then replied over the web.

> I didn't see your replies until I checked back.

Hmm. Did you email-reply? This one has your address in the CC.

[Jim Fulton (jim-zope)]

On Mon, Sep 27, 2010 at 2:13 PM, "Martin v. Löwis" <[e-mail]> wrote:
> Am 27.09.2010 19:01, schrieb Jim Fulton:
>> How did you see my comments? Did you get email, or did you poll?
>
> I got an email, and then replied over the web.
>
>> I didn't see your replies until I checked back.
>
> Hmm. Did you email-reply? This one has your address in the CC.

Nope.

<shrug>

Fortunately, I got this one and the previous one via email. :)

Jim

--
Jim Fulton

[Jim Fulton (jim-zope)]

On Mon, Sep 27, 2010 at 2:12 PM, "Martin v. Löwis" <[e-mail]> wrote:
>> Maybe I'm missinterpreting what you mean by "process".
>>
>> I'd be surprised is "all client software" tried each address returned.
>> I would expect most client software to simply use an address in bind
>> or connect and get watever address is prefered for the type of socket
>> they created.
>
> With IPv6, it is now common and recommended that each host gets at least
> two addresses: one for IPv4 and one for IPv6 (dual stack).

Makes sense.

> In addition,
> some systems have multiple IPv6 addresses (e.g. assigned by various
> tunneling systems).

Hm. Sounds complicated. :)

> It's not universally clear that any of these
> addresses is better than any other: it depends, in particular, on the
> client connectivity (IPv6 available or not, IPv4 available or not).

Hm.

> The new resolver API recommended for use it getaddrinfo(3). It gives you
> a list of addrinfo(family, socktype, proto, canonname, addr), mixing
> IPv4 and IPv6 addresses. Today, systems have a policy (e.g.
> /etc/gai.conf) to determine in which order to return them (e.g. IPv6
> global first, then IPv4, then IPv6 tunnel). Applications must not
> normally chose for themselves whether they want IPv4 or IPv6
> connections, but should let the system decide.
>
> In turn, you can't actually create the socket until getaddrinfo
> returns, as it will tell you what socket family to use. Hence
> the typically connect code looks like this:
>
>  for family,address in getaddrinfo():
>     try:
>         s = socket.socket(family)
>         s.connect(address, port)
>     except socket.error:
>         continue
>  if not s:
>     raise "None of these addresses worked"

I'm hoping there's a missing break or return in there somewhere. :)

> The IPv6 server code is entirely different: typically, you either
> bind to all addresses (v4 and v6) of a system (preferably using
> dual-stack sockets using a wildcard address), or you bind to a (list
> of) specifically configured addresses; no DNS involved.

I wonder why the asymmetry. Or, at least, why it wouldn't at least
make sense to at least listen for both v4 and v6 for the same host name.

> When binding
> to a list of addresses, you need to create multiple server sockets.

OK.

> IIUC, ZEO currently doesn't support that

Right.

> (and IMO doesn't need to).

Not sure wrt IPv6.

I have wanted to be able to listen on both a TCP socket and a
unix-domain socket. There's nothing hard about this for ZEO per se,
but figuring how to spell this in ZConfig schema has been just enough
of a pain to make it not happen. :)

>>> With multiple addresses coming out of DNS, it can speed-up connection
>>> establishment when they are tried simultaneously.
>>
>> I'm not sure what you mean here.  But yeah, if you're going to do
>> multiple things, doing them in parallel is generally a good strategy.
>
> See above: the for loop can run in parallel.
>
>>> In addition, it was easier to implement that way. If you think it's
>>> undesirable, I can try to come up with a way to do them
>>> sequentially.
>>
>> I'm not clear on what was easier to implement.
>>
>> I'll try to guess. :)
>>
>> I see a...

[Martin v. Löwis (martin-v)]

>> The IPv6 server code is entirely different: typically, you either
>> bind to all addresses (v4 and v6) of a system (preferably using
>> dual-stack sockets using a wildcard address), or you bind to a (list
>> of) specifically configured addresses; no DNS involved.
>
> I wonder why the asymmetry. Or, at least, why it wouldn't at least
> make sense to at least listen for both v4 and v6 for the same host name.

It always was asymmetric: you can use a wildcard address on the server,
but not on the client, as a special feature of the socket API.

Some servers support binding to multiple explicit interfaces (e.g.
Apache Listen), but I'm not aware of a precedent that supports
DNS in that part of the configuration, and supports DNS returning
multiple addresses (i.e. expanding them to multiple Listen statements).

> I have wanted to be able to listen on both a TCP socket and a
> unix-domain socket. There's nothing hard about this for ZEO per se,
> but figuring how to spell this in ZConfig schema has been just enough
> of a pain to make it not happen. :)

Ok. So do you want me to change the patch to bind to multiple addresses?

I think StorageServer would need to change, to support multiple
addresses, and also multiple Dispatcher objects.

> I don't mind trying multiple paths (addresses) to do the low-level
> connections, but I'm worried about making multiple connections to the
> same server.

I could try to find out how to prevent that: it should be possible
to keep track what DNS names each connection originated from, and then
close any redundant connections that get connected (assuming zrpc
can deal with connections on which no data gets transmitted).

> I'm good with merging the branch. Do you want to do this (using some
> LP tool)? Or do you want me to?

I would rather see you merge the patch. However, if you want me to
make some modifications first, we can also wait (I won't be able to
work on this really for the next two weeks).

[Jim Fulton (jim-zope)]

On Mon, Sep 27, 2010 at 3:56 PM, "Martin v. Löwis" <[e-mail]> wrote:
>>> The IPv6 server code is entirely different: typically, you either
>>> bind to all addresses (v4 and v6) of a system (preferably using
>>> dual-stack sockets using a wildcard address), or you bind to a (list
>>> of) specifically configured addresses; no DNS involved.
>>
>> I wonder why the asymmetry.  Or, at least, why it wouldn't at least
>> make sense to at least listen for both v4 and v6 for the same host name.
>
> It always was asymmetric: you can use a wildcard address on the server,
> but not on the client, as a special feature of the socket API.
>
> Some servers support binding to multiple explicit interfaces (e.g.
> Apache Listen), but I'm not aware of a precedent that supports
> DNS in that part of the configuration, and supports DNS returning
> multiple addresses (i.e. expanding them to multiple Listen statements).
>
>> I have wanted to be able to listen on both a TCP socket and a
>> unix-domain socket.  There's nothing hard about this for ZEO per se,
>> but figuring how to spell this in ZConfig schema has been just enough
>> of a pain to make it not happen. :)
>
> Ok. So do you want me to change the patch to bind to multiple addresses?

No. I was just mentioning that I'd considered adding support for listening
on multiple addresses.

...

>> I don't mind trying multiple paths (addresses) to do the low-level
>> connections, but I'm worried about making multiple connections to the
>> same server.
>
> I could try to find out how to prevent that: it should be possible
> to keep track what DNS names each connection originated from, and then
> close any redundant connections that get connected (assuming zrpc
> can deal with connections on which no data gets transmitted).

Nah. The connection logic is already too complicated.
I plan a major refactoring of it down the road.

>> I'm good with merging the branch.  Do you want to do this (using some
>> LP tool)? Or do you want me to?
>
> I would rather see you merge the patch.

Cool. That seems easier.

> However, if you want me to
> make some modifications first, we can also wait (I won't be able to
> work on this really for the next two weeks).

No, I'm fine with it as it is.

Thanks.

Jim

--
Jim Fulton

[Jim Fulton (jim-zope)]

Merged to trunk. This will be in ZODB 3.10b7.

review: Approve

Unmerged revisions

3958. By Martin v. Löwis on 2010-09-19

Only check for IPPROTO_IPV6 if socket creation succeeded.
Look for AttributeError, not NameError, when checking IPPROTO_IPV6 presence.

3957. By Martin v. Löwis on 2010-09-19

Remove debugging code.

3956. By Martin v. Löwis on 2010-09-19

Adjust test case to display numeric IP instead of hostname.

3955. By Martin v. Löwis on 2010-09-19

Add IPv6 support.

=== modified file 'doc/zeo.txt'
--- doc/zeo.txt	2008-01-19 18:53:09 +0000
+++ doc/zeo.txt	2010-09-19 09:56:47 +0000
@@ -164,6 +164,9 @@
 be used to with a system that arranges to provide hot backups of
 servers in the case of failure.
 
+If a single address resolves to multiple IPv4 or IPv6 addresses,
+the client will connect to an arbitrary of these addresses.
+
 Authentication
 ~~~~~~~~~~~~~~
 
 
=== modified file 'src/ZEO/tests/ConnectionTests.py'
--- src/ZEO/tests/ConnectionTests.py	2010-07-11 14:17:01 +0000
+++ src/ZEO/tests/ConnectionTests.py	2010-09-19 09:56:47 +0000
@@ -16,6 +16,7 @@
 import sys
 import time
 import random
+import socket
 import asyncore
 import threading
 import logging
@@ -1197,3 +1198,18 @@
                 c.close()
             except:
                 pass
+
+# Run IPv6 tests if V6 sockets are supported
+try:
+    socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
+except (socket.error, AttributeError):
+    pass
+else:
+    class V6Setup:
+        def _getAddr(self):
+            return '::1', forker.get_port(self)
+
+    _g = globals()
+    for name, value in _g.items():
+        if isinstance(value, type) and issubclass(value, CommonSetupTearDown):
+            _g[name+"V6"] = type(name+"V6", (V6Setup, value), {})
 
=== modified file 'src/ZEO/tests/testZEO.py'
--- src/ZEO/tests/testZEO.py	2010-09-16 20:19:20 +0000
+++ src/ZEO/tests/testZEO.py	2010-09-19 09:56:47 +0000
@@ -1148,13 +1148,13 @@
     ... # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE
     ZEO.ClientStorage CRITICAL client
     Client has seen newer transactions than server!
-    ZEO.zrpc ERROR (...) CW: error in notifyConnected (('localhost', ...))
+    ZEO.zrpc ERROR (...) CW: error in notifyConnected (('127.0.0.1', ...))
     Traceback (most recent call last):
     ...
     ClientStorageError: client Client has seen newer transactions than server!
     ZEO.ClientStorage CRITICAL client
     Client has seen newer transactions than server!
-    ZEO.zrpc ERROR (...) CW: error in notifyConnected (('localhost', ...))
+    ZEO.zrpc ERROR (...) CW: error in notifyConnected (('127.0.0.1', ...))
     Traceback (most recent call last):
     ...
     ClientStorageError: client Client has seen newer transactions than server!
 
=== modified file 'src/ZEO/zrpc/client.py'
--- src/ZEO/zrpc/client.py	2010-09-09 18:38:51 +0000
+++ src/ZEO/zrpc/client.py	2010-09-19 09:56:47 +0000
@@ -188,7 +188,7 @@
         if (len(addr) == 2
             and isinstance(addr[0], types.StringType)
             and isinstance(addr[1], types.IntType)):
-            return socket.AF_INET
+            return socket.AF_INET # also denotes IPv6
 
         # not anything I know about
         return None
@@ -436,10 +436,22 @@
             del wrappers
         return 0
 
+    def _expand_addrlist(self):
+        for domain, (host, port) in self.addrlist:
+            # AF_INET really means either IPv4 or IPv6, 
+            # possibly indirected by DNS. By design, DNS lookup is deferred until
+            # connections get established, so that DNS reconfiguration can affect failover
+            if domain == socket.AF_INET:
+                for family, socktype, proto, cannoname, sockaddr in socket.getaddrinfo(host, port):
+                    # for IPv6, drop flowinfo, and restrict addresses to [host]:port
+                    yield family, sockaddr[:2]
+            else:
+                yield domain, addr
+
     def _create_wrappers(self):
         # Create socket wrappers
         wrappers = {}  # keys are active wrappers
-        for domain, addr in self.addrlist:
+        for domain, addr in self._expand_addrlist():
             wrap = ConnectWrapper(domain, addr, self.mgr, self.client)
             wrap.connect_procedure()
             if wrap.state == "notified":
 
=== modified file 'src/ZEO/zrpc/server.py'
--- src/ZEO/zrpc/server.py	2010-06-21 14:33:46 +0000
+++ src/ZEO/zrpc/server.py	2010-09-19 09:56:47 +0000
@@ -15,6 +15,23 @@
 import socket
 import types
 
+# _has_dualstack: True if the dual-stack sockets are supported
+try:
+    # Check whether IPv6 sockets can be created
+    s = socket.socket(socket.AF_INET6, socket.SOCK_STREAM)
+except (socket.error, AttributeError):
+    _has_dualstack = False
+else:
+    # Check whether enabling dualstack (disabling v6only) works
+    try:
+        s.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, False)
+    except (socket.error, AttributeError):
+        _has_dualstack = False
+    else:
+        _has_dualstack = True
+    s.close()
+    del s
+
 from ZEO.zrpc.connection import Connection
 from ZEO.zrpc.log import log
 import ZEO.zrpc.log
@@ -35,7 +52,18 @@
 
     def _open_socket(self):
         if type(self.addr) == types.TupleType:
-            self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
+            if self.addr[0] == '' and _has_dualstack:
+                # Wildcard listen on all interfaces, both IPv4 and IPv6 if possible
+                self.create_socket(socket.AF_INET6, socket.SOCK_STREAM)
+                self.socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, False)
+            elif ':' in self.addr[0]:
+                self.create_socket(socket.AF_INET6, socket.SOCK_STREAM)
+                if _has_dualstack:
+                    # On Linux, IPV6_V6ONLY is off by default. 
+                    # If the user explicitly asked for IPv6, don't bind to IPv4
+                    self.socket.setsockopt(socket.IPPROTO_IPV6, socket.IPV6_V6ONLY, True)
+            else:
+                self.create_socket(socket.AF_INET, socket.SOCK_STREAM)
         else:
             self.create_socket(socket.AF_UNIX, socket.SOCK_STREAM)
         self.set_reuse_addr()
@@ -56,6 +84,9 @@
             log("accepted failed: %s" % msg)
             return
 
+        # Drop flow-info from IPv6 addresses
+        addr = addr[:2]
+
         # We could short-circuit the attempt below in some edge cases
         # and avoid a log message by checking for addr being None.
         # Unfortunately, our test for the code below,
 
=== modified file 'src/ZEO/zrpc/smac.py'
--- src/ZEO/zrpc/smac.py	2010-06-21 14:33:46 +0000
+++ src/ZEO/zrpc/smac.py	2010-09-19 09:56:47 +0000
@@ -120,6 +120,10 @@
 
         self.__super_init(sock, map)
 
+        # asyncore overwrites addr with the getpeername result
+        # restore our value
+        self.addr = addr
+
     def setSessionKey(self, sesskey):
         log("set session key %r" % sesskey)