Merge lp:~stefanor/ibid/dc-too-long-503017 into lp:~ibid-core/ibid/old-trunk-1.6

[Stefano Rivera]

[Stefano Rivera (stefanor)]

Urgent merge, but probably a little controversial

[Stefano Rivera (stefanor)]

Urgent merge, but probably a little controversial

[Jonathan Hitchcock (vhata)]

Hardcoded line-lengths bad.

review: Needs Fixing

[Stefano Rivera (stefanor)]

Now available without hardcoded line-lengths

[Michael Gorven (mgorven)]

Looks fine.
 review approve

review: Approve

[lp:~stefanor/ibid/dc-too-long-503017]

832. By Stefano Rivera on 2010-01-06

horizontal ellipsis = 3 utf-8 bytes

[Jonathan Hitchcock (vhata)]

Not sure I like max_message_length vs message_max_length - which is which? And why does silc suddenly grow one that refers to public length? And why does jabber not have a similar one to silc, since they both only limit in public?

Also unhappy about sources modifying some of their settings based on other settings - e.g. removing 'action' from supports if there is no action_prefix set. If a source author says that his source supports actions, then believe him, even if he forget to set an action_prefix (make one up yourself?).

[lp:~stefanor/ibid/dc-too-long-503017]

833. By Stefano Rivera on 2010-01-06

Add setup() method to sources and docstrings to IbidSourceFactory

834. By Stefano Rivera on 2010-01-06

Use the new source setup method

835. By Stefano Rivera on 2010-01-06

Whitespace

[Stefano Rivera (stefanor)]

> If a source author says that his source supports actions, then believe him,
> even if he forget to set an action_prefix (make one up yourself?).

Um, if only DC were that simple... Got it?

[lp:~stefanor/ibid/dc-too-long-503017]

836. By Stefano Rivera on 2010-01-06

Whoops, ibid.sources is a list of names not sources

[Jonathan Hitchcock (vhata)]

I still don't like this:

def setup(self):
   if self.action_prefix is None and 'action' in self.supports:
      self.supports.remove('action')
   if self.action_prefix is not None and 'action' not in self.supports:
      self.supports.append('action')

Your explanation:

12:49 <tumbleweed> Vhata: the DC protocol don't actually support actions
12:49 <tumbleweed> some severs support public actions with "/me says foo"
12:49 <tumbleweed> some use the syntax "+me says foo"

However, I think you're doing it the wrong way round. I feel that "supports" is more authoritative than setting an option. So, rather:

   if self.action_prefix is None and 'action' in self.supports:
      self.action_prefix = '* '
   if self.action_prefix is not None and 'action' not in self.supports:
      pass # who cares? they can set the option if they want. Ignore it.

See also my comments about the inconsistency between the jabber and silc private/public max_message_length, and the confusing names...

review: Needs Fixing

[lp:~stefanor/ibid/dc-too-long-503017]

837. By Stefano Rivera on 2010-01-07

Don't have action in the supports list by default

[Jonathan Hitchcock (vhata)]

Right, Stefano's explained the weird shit that DC does.

I retract that stuff...

[lp:~stefanor/ibid/dc-too-long-503017]

838. By Stefano Rivera on 2010-01-07

Rename function that was very similar (in name) to a related variable

[Jonathan Hitchcock (vhata)]

*phew*

approve.

review: Approve

=== modified file 'ibid/core.py'
--- ibid/core.py	2010-01-04 22:04:04 +0000
+++ ibid/core.py	2010-01-07 12:26:15 +0000
@@ -273,6 +273,8 @@
     def reload_config(self):
         for processor in ibid.processors:
             processor.setup()
+        for source in ibid.sources:
+            ibid.sources[source].setup()
         self.log.info(u"Notified all processors of config reload")
 
 def regexp(pattern, item):
 
=== modified file 'ibid/plugins/core.py'
--- ibid/plugins/core.py	2010-01-02 17:03:20 +0000
+++ ibid/plugins/core.py	2010-01-07 12:26:15 +0000
@@ -172,28 +172,74 @@
 class Format(Processor):
     priority = 2000
 
+    def _truncate(self, line, length):
+        if length is not None:
+            eline = line.encode('utf-8')
+            if len(eline) > length:
+                # horizontal ellipsis = 3 utf-8 bytes
+                return eline[:length-3].decode('utf-8', 'ignore') \
+                       + u'\N{horizontal ellipsis}'
+        return line
+
     def process(self, event):
         filtered = []
         for response in event.responses:
             source = response['source'].lower()
             supports = ibid.sources[source].supports
+            maxlen = ibid.sources[source].truncation_point(response, event)
 
             if response.get('action', False) and 'action' not in supports:
                 response['reply'] = u'*%s*' % response['reply']
 
             conflate = response.get('conflate', True)
+            # Expand response into multiple single-line responses:
             if (not conflate and 'multiline' not in supports):
                 for line in response['reply'].split('\n'):
-                    r = {'reply': line}
-                    for k in response.iterkeys():
-                        if k not in ('reply'):
-                            r[k] = response[k]
-                    filtered.append(r)
+                    r = {'reply': self._truncate(line, maxlen)}
+                    for k in response.iterkeys():
+                        if k not in ('reply'):
+                            r[k] = response[k]
+                    filtered.append(r)
+
+            # Expand response into multiple multi-line responses:
+            elif (not conflate and 'multiline' in supports
+                               and maxlen is not None):
+                message = response['reply']
+                while len(message.encode('utf-8')) > maxlen:
+                    splitpoint = len(message.encode('utf-8')[:maxlen] \
+                                            .decode('utf-8', 'ignore'))
+                    parts = [message[:splitpoint].rstrip(),
+                             message[splitpoint:].lstrip()]
+                    for sep in u'\n.;:, ':
+                        if sep in u'\n ':
+                            search = message[:splitpoint+1]
+                        else:
+                            search = message[:splitpoint]
+                        if sep in search:
+                            splitpoint = search.rindex(sep)
+                            parts = [message[:splitpoint+1].rstrip(),
+                                     message[splitpoint+1:]]
+                            break
+                    r = {'reply': parts[0]}
+                    for k in response.iterkeys():
+                        if k not in ('reply'):
+                            r[k] = response[k]
+                    filtered.append(r)
+                    message = parts[1]
+
+                response['reply'] = message
+                filtered.append(response)
+
             else:
+                line = response['reply']
+                # Remove any characters that make no sense on IRC-like sources:
                 if 'multiline' not in supports:
-                    response['reply'] = response['reply'].expandtabs(1) \
-                            .replace('\n', conflate == True
-                                           and u' ' or conflate or u'')
+                    line = line.expandtabs(1) \
+                               .replace('\n', conflate == True
+                                              and u' ' or conflate or u'')
+
+                response['reply'] = self._truncate(line, maxlen)
+
                 filtered.append(response)
 
         event.responses = filtered
 
=== modified file 'ibid/source/__init__.py'
--- ibid/source/__init__.py	2009-07-22 21:11:01 +0000
+++ ibid/source/__init__.py	2010-01-07 12:26:15 +0000
@@ -19,23 +19,44 @@
 
     def __init__(self, name):
         self.name = name
+        self.setup()
+
+    def setup(self):
+        "Apply configuration. Called on every config reload"
+        pass
 
     def setServiceParent(self, service):
+        "Start the source and connect"
         raise NotImplementedError
 
     def connect(self):
+        "Connect (if disconncted)"
         return self.setServiceParent(None)
 
     def disconnect(self):
+        "Disconnect source"
         raise NotImplementedError
 
     def url(self):
+        "Return a URL describing the source"
         return None
 
     def logging_name(self, identity):
         "Given an identity or connection, return a name suitable for logging"
         return identity
 
+    def truncation_point(self, response, event=None):
+        """Given a target, and possibly a related event, return the number of
+        bytes to clip at, or None to indicate that a complete message will
+        be delivered.
+        """
+        if (event is not None
+                and response.get('target', None) == event.get('channel', None)
+                and event.get('public', True)):
+            return 490
+
+        return None
+
 from ibid.config import Option
 
 options = {
 
=== modified file 'ibid/source/campfire.py'
--- ibid/source/campfire.py	2009-12-30 22:18:47 +0000
+++ ibid/source/campfire.py	2010-01-07 12:26:15 +0000
@@ -125,4 +125,7 @@
     def leave(self, room_name):
         return self.client.leave(room_name)
 
+    def truncation_point(self, response, event=None):
+        return None
+
 # vi: set et sta sw=4 ts=4:
 
=== modified file 'ibid/source/dc.py'
--- ibid/source/dc.py	2009-12-30 22:18:47 +0000
+++ ibid/source/dc.py	2010-01-07 12:26:15 +0000
@@ -129,8 +129,10 @@
         elif response.get('action', False):
             if self.factory.action_prefix and target is None:
                 self.say(target, u'%s %s' % (self.factory.action_prefix, message))
+            elif self.factory.action_prefix:
+                self.say(target, u'*%s*' % message)
             else:
-                self.say(target, u'* %s %s' % (self.my_nickname, message))
+                self.say(target, message)
 
             self.factory.log.debug(u"Sent action to %s: %s", target, message)
         else:
@@ -156,7 +158,7 @@
 class SourceFactory(protocol.ReconnectingClientFactory, IbidSourceFactory):
     protocol = DCBot
 
-    supports = ('action', 'multiline', 'topic')
+    supports = ['multiline', 'topic']
     auth = ('op',)
 
     port = IntOption('port', 'Server port number', 411)
@@ -170,6 +172,8 @@
     slots = IntOption('slots', 'DC Open Slots', 0)
     action_prefix = Option('action_prefix', 'Command for actions (i.e. +me)', None)
     banned_prefixes = Option('banned_prefixes', 'Prefixes not allowed in bot responses, i.e. !', '')
+    max_message_length = IntOption('max_message_length',
+            'Maximum length of messages', 490)
     ping_interval = FloatOption('ping_interval', 'Seconds idle before sending a PING', 60)
     pong_timeout = FloatOption('pong_timeout', 'Seconds to wait for PONG', 300)
     # ReconnectingClient uses this:
@@ -181,6 +185,12 @@
         self.log = logging.getLogger('source.%s' % self.name)
         self._auth = {}
 
+    def setup(self):
+        if self.action_prefix is None and 'action' in self.supports:
+            self.supports.remove('action')
+        if self.action_prefix is not None and 'action' not in self.supports:
+            self.supports.append('action')
+
     def setServiceParent(self, service):
         if service:
             internet.TCPClient(self.server, self.port, self).setServiceParent(service)
@@ -197,6 +207,9 @@
             self.proto.transport.loseConnection()
         return True
 
+    def truncation_point(self, response, event=None):
+        return self.max_message_length
+
     def _dc_auth_callback(self, nick, result):
         self._auth[nick] = result
 
 
=== modified file 'ibid/source/http.py'
--- ibid/source/http.py	2010-01-06 07:56:27 +0000
+++ ibid/source/http.py	2010-01-07 12:26:15 +0000
@@ -104,10 +104,10 @@
         self.site = server.Site(root)
 
     def setServiceParent(self, service):
-            if service:
-                return internet.TCPServer(self.port, self.site).setServiceParent(service)
-            else:
-                reactor.listenTCP(self.port, self.site)
+        if service:
+            return internet.TCPServer(self.port, self.site).setServiceParent(service)
+        else:
+            reactor.listenTCP(self.port, self.site)
 
     def url(self):
         return self.myurl
 
=== modified file 'ibid/source/irc.py'
--- ibid/source/irc.py	2010-01-02 12:17:07 +0000
+++ ibid/source/irc.py	2010-01-07 12:26:15 +0000
@@ -323,6 +323,9 @@
             return u''
         return identity.split(u'!')[0]
 
+    def truncation_point(self, response, event=None):
+        return 490
+
     def url(self):
         return u'irc://%s@%s:%s' % (self.nick, self.server, self.port)
 
 
=== modified file 'ibid/source/jabber.py'
--- ibid/source/jabber.py	2009-12-30 22:18:47 +0000
+++ ibid/source/jabber.py	2010-01-07 12:26:15 +0000
@@ -168,6 +168,8 @@
     nick = Option('nick', 'Nick for chatrooms', ibid.config['botname'])
     rooms = ListOption('rooms', 'Chatrooms to autojoin', [])
     accept_domains = Option('accept_domains', 'Only accept messages from these domains')
+    max_public_message_length = IntOption('max_public_message_length',
+            'Maximum length of public messages', 512)
 
     def __init__(self, name):
         IbidSourceFactory.__init__(self, name)
@@ -212,4 +214,9 @@
     def logging_name(self, identity):
         return identity.split('/')[0]
 
+    def truncation_point(self, response, event=None):
+        if response.get('target', None) in self.proto.rooms:
+            return self.max_public_message_length
+        return None
+
 # vi: set et sta sw=4 ts=4:
 
=== modified file 'ibid/source/silc.py'
--- ibid/source/silc.py	2010-01-02 12:17:07 +0000
+++ ibid/source/silc.py	2010-01-07 12:26:15 +0000
@@ -69,7 +69,7 @@
             self.send(response)
 
     def send(self, response):
-        message = response['reply'].encode('utf-8')
+        message = response['reply']
         flags=0
         if response.get('action', False):
             flags=4
@@ -186,7 +186,7 @@
         self.command_call('QUIT')
 
     def disconnected(self, message):
-        self.factory.log.info(u"Disconnected (%s)", reason)
+        self.factory.log.info(u"Disconnected (%s)", message)
 
         event = Event(self.factory.name, u'source')
         event.status = u'disconnected'
@@ -208,6 +208,8 @@
     realname = Option('realname', 'Real Name', ibid.config['botname'])
     public_key = Option('public_key', 'Filename of public key', 'silc.pub')
     private_key = Option('private_key', 'Filename of private key', 'silc.prv')
+    max_public_message_length = IntOption('max_public_message_length',
+            'Maximum length of public messages', 512)
 
     def __init__(self, name):
         IbidSourceFactory.__init__(self, name)
@@ -240,4 +242,9 @@
     def logging_name(self, identity):
         return self.client.logging_name(identity)
 
+    def truncation_point(self, response, event=None):
+        if response.get('target', None) in self.client.channels:
+            return self.max_public_message_length
+        return None
+
 # vi: set et sta sw=4 ts=4:
 
=== modified file 'scripts/ibid-plugin'
--- scripts/ibid-plugin	2010-01-04 22:04:04 +0000
+++ scripts/ibid-plugin	2010-01-07 12:26:15 +0000
@@ -62,7 +62,12 @@
     type = 'test'
     permissions = []
     supports = ('action', 'multiline', 'notice')
-    logging_name = lambda self, name: name
+
+    def logging_name(self, name):
+        return name
+
+    def truncation_point(self, response, event=None):
+        return None
 
 ibid.sources[u'test_source'] = TestSource()