Merge lp:~tsyrus/pastebinit/pastebinthat into lp:pastebinit

[Terence Syrus]

Complete overhaul of pastebinit:
* significantly cleaned up code for less coupling and more readability
* changed config to a more sensible structure:
** only posted parameters are in the format section now
** parameters that were previously hidden but listed in the format section now live in the pastebin section
* added unit tests
* merged in all branches from ~blueyed

[lp:~tsyrus/pastebinit/pastebinthat]

216. By Terence Syrus <email address hidden> on 2014-12-01

added unit tests

217. By Terence Syrus <email address hidden> on 2014-12-02

removed unused method

218. By Terence Syrus <email address hidden> on 2014-12-02

removed debug

[Stéphane Graber (stgraber)]

Hi,

I'm sorry I didn't see this before, as you most likely figured out by now, I'm not very actively involved anymore as pastebinit pretty much just works for me and has done so for years.

I'm certainly happy with work to make it less crufty and better tested, unfortunately it looks like the bunch of changes I applied today makes this fail to apply.

Any chance you can rebase this on current pastebinit. I'll try to be faster to respond this time! :)

Unmerged revisions

218. By Terence Syrus <email address hidden> on 2014-12-02

removed debug

217. By Terence Syrus <email address hidden> on 2014-12-02

removed unused method

216. By Terence Syrus <email address hidden> on 2014-12-01

added unit tests

215. By Terence Syrus <email address hidden> on 2014-12-01

major overhaul: tests + cleanup

214. By Terence Syrus <email address hidden> on 2014-11-26

major cleanup
also, updated pastebin.mate-desktop.org.conf

213. By Terence Syrus <email address hidden> on 2014-11-24

merged changes from Daniel Hahler

=== modified file 'pastebin.d/cxg.de.conf'
--- pastebin.d/cxg.de.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/cxg.de.conf	2014-12-02 02:23:09 +0000
@@ -1,14 +1,11 @@
 [pastebin]
 basename = cxg.de
-regexp = http://cxg.de
+id_regexp = http://cxg.de
+result_type = url
+result_regexp = href="(http://cxg.de/.*)"
+page = /paste.php
 
 [format]
 user = desc
 content = pastetext
 format = lang
-page = page
-regexp = regexp
-
-[defaults]
-page = /paste.php
-regexp = href="http://cxg.de/(.*)"
 
=== added file 'pastebin.d/dpaste.com.conf'
--- pastebin.d/dpaste.com.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/dpaste.com.conf	2014-12-02 02:23:09 +0000
@@ -0,0 +1,17 @@
+[pastebin]
+basename = dpaste.com
+id_regexp = http://dpaste.com
+page = /api/v1/
+
+[format]
+user = poster
+content = content
+title = title
+format = language
+hold = hold
+
+[defaults]
+# Empty means "Plain".
+format =
+# Keep it for 30 days after last access. Otherwise only for 7 days.
+hold = on
 
=== modified file 'pastebin.d/fpaste.org.conf'
--- pastebin.d/fpaste.org.conf	2014-01-06 20:58:13 +0000
+++ pastebin.d/fpaste.org.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,6 @@
 [pastebin]
 basename = fpaste.org
-regexp = http://fpaste.org
+id_regexp = http://fpaste.org
 
 [format]
 user = paste_user
 
=== modified file 'pastebin.d/lpaste.net.conf'
--- pastebin.d/lpaste.net.conf	2014-01-06 22:59:43 +0000
+++ pastebin.d/lpaste.net.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,7 @@
 [pastebin]
 basename = lpaste.net
-regexp = http://lpaste.net
+id_regexp = http://lpaste.net
+page = new
 
 [format]
 public = public
@@ -10,10 +11,8 @@
 channel = channel
 content = paste
 email = email
-page = page
 
 [defaults]
-page = new
 public = Public
 channel =
 email =
 
=== modified file 'pastebin.d/p.defau.lt.conf'
--- pastebin.d/p.defau.lt.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/p.defau.lt.conf	2014-12-02 02:23:09 +0000
@@ -1,10 +1,7 @@
 [pastebin]
 basename = p.defau.lt
-regexp = http://p.defau.lt
+id_regexp = http://p.defau.lt
+page = /submit.php
 
 [format]
 content = code
-page = page
-
-[defaults]
-page = /submit.php
 
=== modified file 'pastebin.d/paste.debian.net.conf'
--- pastebin.d/paste.debian.net.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/paste.debian.net.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,6 @@
 [pastebin]
 basename = paste.debian.net
-regexp = http://paste.debian.net
+id_regexp = https?://paste.debian.net
 
 [format]
 user = poster
 
=== modified file 'pastebin.d/paste.drizzle.org.conf'
--- pastebin.d/paste.drizzle.org.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/paste.drizzle.org.conf	2014-12-02 02:23:09 +0000
@@ -1,18 +1,15 @@
 [pastebin]
 basename = paste.drizzle.org
-regexp = http://paste.drizzle.org
+id_regexp = http://paste.drizzle.org
+target_url = http://paste.drizzle.org/show/
+result_type = id
+result_regexp = \"data\": \"([^"]*)
+post_format = json
+page = /json/?method=pastes.newPaste
 
 [format]
 lang = language
-page = page
 content = code
-post_format = post_format
-regexp = regexp
-target_url = target_url
 
 [defaults]
-page = /json/?method=pastes.newPaste
-post_format = json
 lang =
-regexp = \"data\": \"([^"]*)
-target_url = http://paste.drizzle.org/show/
 
=== modified file 'pastebin.d/paste.kde.org.conf'
--- pastebin.d/paste.kde.org.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/paste.kde.org.conf	2014-12-02 02:23:09 +0000
@@ -1,22 +1,18 @@
 [pastebin]
 basename = paste.kde.org
-regexp = http://paste.kde.org
+id_regexp = http://paste.kde.org
+result_type = id
+result_regexp = <id>(.*)</id>
+page = /api/xml/create
 
 [format]
-user = paste_user
-format = paste_lang
-content = paste_data
-password = paste_password
-expire = paste_expire
-private = paste_private
-api = api_submit
-mode = mode
-regexp = regexp
+format = language
+content = data
+expire = expire
+private = private
+title = title
+password = password
 
 [defaults]
-private = 0
 format = text
 expire = 86400
-api = 1
-mode = xml
-regexp = <id>(.*)</id>
 
=== modified file 'pastebin.d/paste.openstack.org.conf'
--- pastebin.d/paste.openstack.org.conf	2012-06-13 17:15:59 +0000
+++ pastebin.d/paste.openstack.org.conf	2014-12-02 02:23:09 +0000
@@ -1,18 +1,15 @@
 [pastebin]
 basename = paste.openstack.org
-regexp = http://paste.openstack.org
+id_regexp = http://paste.openstack.org
+result_type = id
+result_regexp = \"data\": \"([^"]*)
+target_url = http://paste.openstack.org/show/
+post_format = json
+page = /json/?method=pastes.newPaste
 
 [format]
 lang = language
-page = page
 content = code
-post_format = post_format
-regexp = regexp
-target_url = target_url
 
 [defaults]
-page = /json/?method=pastes.newPaste
-post_format = json
 lang =
-regexp = \"data\": \"([^"]*)
-target_url = http://paste.openstack.org/show/
 
=== modified file 'pastebin.d/paste.pound-python.org.conf'
--- pastebin.d/paste.pound-python.org.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/paste.pound-python.org.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,6 @@
 [pastebin]
 basename = paste.pound-python.org
-regexp = http://paste.pound-python.org
+id_regexp = http://paste.pound-python.org
 
 [format]
 content = code
 
=== modified file 'pastebin.d/paste.ubuntu.com.conf'
--- pastebin.d/paste.ubuntu.com.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/paste.ubuntu.com.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,6 @@
 [pastebin]
 basename = paste.ubuntu.com
-regexp = http://paste.ubuntu.com
+id_regexp = http://paste.ubuntu.com
 
 [format]
 user = poster
 
=== modified file 'pastebin.d/paste.ubuntu.org.cn.conf'
--- pastebin.d/paste.ubuntu.org.cn.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/paste.ubuntu.org.cn.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,6 @@
 [pastebin]
 basename = paste.ubuntu.org.cn
-regexp = http://paste.ubuntu.org.cn
+id_regexp = http://paste.ubuntu.org.cn
 
 [format]
 user = poster
 
=== modified file 'pastebin.d/paste2.org.conf'
--- pastebin.d/paste2.org.conf	2014-01-06 23:16:01 +0000
+++ pastebin.d/paste2.org.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,6 @@
 [pastebin]
 basename = paste2.org
-regexp = http://paste2.org
+id_regexp = http://paste2.org
 
 [format]
 title = description
 
=== modified file 'pastebin.d/pastebin.com.conf'
--- pastebin.d/pastebin.com.conf	2012-06-13 17:15:59 +0000
+++ pastebin.d/pastebin.com.conf	2014-12-02 02:23:09 +0000
@@ -1,6 +1,9 @@
 [pastebin]
 basename = pastebin.com
-regexp = http://((([a-zA-Z0-9\-_\.]*)(pastebin\.com)))
+id_regexp = http://((([a-zA-Z0-9\-_\.]*)(pastebin\.com)))
+result_type = url
+result_regexp = (.*)
+page = /api/api_post.php
 
 [format]
 content = api_paste_code
@@ -10,9 +13,7 @@
 expiry = api_paste_expire_date
 format = api_paste_format
 email = api_paste_email
-page = page
 submit = submit
-regexp = regexp
 api_dev_key = api_dev_key
 api_option = api_option
 
@@ -25,5 +26,3 @@
 email =
 api_dev_key = 253ce2f0a45140ee0a44ca99aa492260
 api_option = paste
-page = /api/api_post.php
-regexp = (.*)
 
=== modified file 'pastebin.d/pastebin.mate-desktop.org.conf'
--- pastebin.d/pastebin.mate-desktop.org.conf	2014-01-06 20:40:29 +0000
+++ pastebin.d/pastebin.mate-desktop.org.conf	2014-12-02 02:23:09 +0000
@@ -1,14 +1,12 @@
 [pastebin]
 basename = pastebin.mate-desktop.org
-regexp = http://pastebin.mate-desktop.org
+page = /api/json/create
+id_regexp = http://pastebin.mate-desktop.org
+result_type = id
+result_regexp = "id": "([^"]*)"
+post_format = json
 
 [format]
-content = text
+content = data
 user = name
-format = lang
-page = page
-regexp = regexp
-
-[defaults]
-page = /api/create
-regexp = (.*)
+format = language
 
=== modified file 'pastebin.d/pb.daviey.com.conf'
--- pastebin.d/pb.daviey.com.conf	2012-06-13 17:15:59 +0000
+++ pastebin.d/pb.daviey.com.conf	2014-12-02 02:23:09 +0000
@@ -1,16 +1,15 @@
 [pastebin]
 basename = pb.daviey.com
-regexp = http://pb.daviey.com
+id_regexp = http://pb.daviey.com
+result_regexp = "/(.*/)".*
+result_type = id
+page = api/?_call=new
 
 [format]
 user = author
 content = content
-page = page
 expire_options = expires
 title = title
-regexp = regexp
 
 [defaults]
-page = api/?_call=new
-regexp = "/(.*/)".*
 expire_options = 7889232
 
=== modified file 'pastebin.d/slexy.org.conf'
--- pastebin.d/slexy.org.conf	2012-06-13 17:15:59 +0000
+++ pastebin.d/slexy.org.conf	2014-12-02 02:23:09 +0000
@@ -1,12 +1,12 @@
 [pastebin]
 basename = slexy.org
-regexp = http://slexy.org
+id_regexp = http://slexy.org
+page = /index.php/submit
 
 [format]
 user = author
 content = raw_paste
 title = desc
-page = page
 language = language
 permissions = permissions
 linenumbers = linenumbers
@@ -16,7 +16,6 @@
 
 [defaults]
 submit = Submit Paste
-page = /index.php/submit
 language = text
 permissions = 0
 comment =
 
=== modified file 'pastebin.d/sprunge.us.conf'
--- pastebin.d/sprunge.us.conf	2012-06-13 16:41:21 +0000
+++ pastebin.d/sprunge.us.conf	2014-12-02 02:23:09 +0000
@@ -1,11 +1,9 @@
 [pastebin]
 basename = sprunge.us
-regexp = http://sprunge.us
+id_regexp = http://sprunge.us
+result_regexp=http://sprunge.us/(.*)$
+result_type = id
 
 [format]
 content = sprunge
 format = lang
-regexp = regexp
-
-[defaults]
-regexp=http://sprunge.us/(.*)$
 
=== modified file 'pastebinit'
--- pastebinit	2014-01-18 04:51:35 +0000
+++ pastebinit	2014-12-02 02:23:09 +0000
@@ -4,7 +4,7 @@
 # Author: Stéphane Graber <[e-mail]>
 # Written by Stéphane Graber <[e-mail]>
 #            Daniel Bartlett <[e-mail]>
-# Last modification : Mon Jan  6 18:46:46 EST 2014
+# Last modification : Mon Nov 24 15:07:45 EST 2014
 
 # This program is free software; you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
@@ -23,224 +23,48 @@
 from __future__ import print_function
 
 import sys
+import os
+import re
+import getopt
+import gettext
+import socket
+import xml.dom.minidom
+
 if sys.version[0] == "2":
-    from ConfigParser import SafeConfigParser
+    from ConfigParser import SafeConfigParser, NoOptionError, NoSectionError
     from urllib import urlencode
     from urllib import FancyURLopener
 else:
-    from configparser import SafeConfigParser
+    #SafeConfigParser is now a deprecated alias to ConfigParser
+    from configparser import \
+        ConfigParser as SafeConfigParser, NoOptionError, NoSectionError
     from urllib.parse import urlencode
     from urllib.request import FancyURLopener
 
-# Set the default pastebin
-defaultPB = "http://pastebin.com"
 
-# Now try to override it with a distributor pastebin
 try:
-    import lsb_release
-    release = lsb_release.get_distro_information()['ID'].lower()
-    if release == 'debian':
-        defaultPB = "http://paste.debian.net"
-    elif release == 'fedora':
-        defaultPB = "http://fpaste.org"
-    elif release == 'ubuntu':
-        defaultPB = "http://paste.ubuntu.com"
+    import json
 except ImportError:
-    pass
-
-try:
-    import getopt
-    import gettext
-    import os
-    import re
-    import socket
-    import xml.dom.minidom
-
-    try:
-        import json
-    except ImportError:
-        json = None
-
-    _ = gettext.gettext
-    gettext.textdomain("pastebinit")
-
-    # Timeout after 5s
-    socket.setdefaulttimeout(5)
-
-    # Version number to show in the usage
-    version = "1.4.1"
-    configfile = os.path.expanduser("~/.pastebinit.xml")
-
-    # Custom urlopener to handle 401's
-    class pasteURLopener(FancyURLopener):
-        version = "Pastebinit v%s" % version
-
-        def http_error_401(self, url, fp, errcode, errmsg, headers, data=None):
-            return None
-
-    def preloadPastebins():
-        # Check several places for config files:
-        #  - global config in /etc/pastebin.d
-        #  - for source checkout, config in the checkout
-        #  - user's overrides in ~/.pastebin.d
-        # Files found later override files found earlier.
-        pastebind = {}
-        for confdir in ['/usr/share/pastebin.d', '/etc/pastebin.d',
-                        '/usr/local/etc/pastebin.d',
-                        os.path.expanduser('~/.pastebin.d'),
-                        os.path.join(
-                            os.path.dirname(
-                                os.path.realpath(__file__)), 'pastebin.d')]:
-            try:
-                confdirlist = os.listdir(confdir)
-            except OSError:
-                continue
-
-            for fileitem in confdirlist:
-                if fileitem.startswith('.') or not fileitem.endswith('.conf'):
-                    continue
-
-                filename = os.path.join(confdir, fileitem)
-                instance = SafeConfigParser()
-                try:
-                    instance.read(filename)
-                except UnicodeError:
-                    continue
-
-                if not instance.has_section('pastebin'):
-                    print(_('%s: no section [pastebin]') % filename,
-                          file=sys.stderr)
-                    continue
-
-                if not instance.has_option('pastebin', 'basename'):
-                    print(_("%s: no 'basename' in [pastebin]") % filename,
-                          file=sys.stderr)
-                    continue
-
-                pastebind[instance.get('pastebin', 'basename')] = instance
-        return pastebind
-
-    # pastey.net obfuscates parent ids for replies.  Rather than taking the
-    # post ID given as the parent ID, we must handle this by going to that
-    # post page and looking up what the invisible parent ID field will be
-    # set to for children.
-    def doParentFixup(website, paramname, parentid):
-        if parentid == "":
-            return ""
-        url_opener = pasteURLopener()
-        page = url_opener.open(website + '/' + parentid, None)
-        matches = re.split('<input.*?name="' + paramname + '".*?value="(.*?)"',
-                           page.read())
-        if len(matches) <= 1 or re.match(parentid, matches[1]) is None:
-            # The obfuscated version didn't begin with the partial version,
-            # or unable to find the obfuscated version for some reason!
-            # Create a paste with no parent (should we throw, instead?)
-            return ""
-        return matches[1]
-
-    #Return the parameters depending of the pastebin used
-    def getParameters(website, pastebind, content, user, jabberid, version,
-                      format, parentpid, permatag, title, username,
-                      password):
-        "Return the parameters array for the selected pastebin"
-        params = {}
-        for paste_name, paste_config in pastebind.items():
-            basename = paste_config.get('pastebin', 'basename')
-            if basename == website or paste_name == website:
-                website = "http://%s" % basename
-
-            if re.search(paste_config.get('pastebin', 'regexp'), website):
-                if paste_config.has_option('pastebin', 'sizelimit'):
-                    params['sizelimit'] = paste_config.get('pastebin',
-                                                           'sizelimit')
-
-                for param in paste_config.options('format'):
-                    paramname = paste_config.get('format', param)
-                    if param == 'user':
-                        params[paramname] = user
-                    elif param == 'content':
-                        params[paramname] = content
-                    elif param == 'title':
-                        params[paramname] = title
-                    elif param == 'version':
-                        params[paramname] = version
-                    elif param == 'format':
-                        params[paramname] = format
-                    elif param == 'parentpid':
-                        params[paramname] = doParentFixup(website, paramname,
-                                                          parentpid)
-                    elif param == 'permatag':
-                        params[paramname] = permatag
-                    elif param == 'username':
-                        params[paramname] = username
-                    elif param == 'password':
-                        params[paramname] = password
-                    elif param == 'jabberid':
-                        params[paramname] = jabberid
-                    else:
-                        params[paramname] = paste_config.get('defaults', param)
-        if params:
-            return website, params
-        else:
-            print(_("Unknown website, please post a bugreport to request "
-                    "this pastebin to be added (%s)") % website,
-                  file=sys.stderr)
-            sys.exit(1)
-
-    #XML Handling methods
-    def getText(nodelist):
-        rc = ""
-        for node in nodelist:
-            if node.nodeType == node.TEXT_NODE:
-                rc = rc + node.data
-        return rc
-
-    def getNodes(nodes, title):
-        return nodes.getElementsByTagName(title)
-
-    def getFirstNode(nodes, title):
-        return getNodes(nodes, title)[0]
-
-    def getFirstNodeText(nodes, title):
-        return getText(getFirstNode(nodes, title).childNodes)
-
-    # Display usage instructions
-    def Usage(fd=sys.stdout):
-        print("pastebinit v" + version, file=fd)
-        print(_("Reads on stdin for input or takes a list of filenames "
-                "as parameters"), file=fd)
-        print(_("Optional arguments (not supported by all pastebins):"),
-              file=fd)
-        print(_("\t-a <author:default is '%s'>") % user, file=fd)
-        print(_("\t-b <pastebin url:default is '%s'>") % website, file=fd)
-        print(_("\t-f <format of paste:default is '%s'>") % format, file=fd)
-        print(_("\t-h This help screen"), file=fd)
-        print(_("\t-i <input file>"), file=fd)
-        print(_("\t-l List all supported pastebins"), file=fd)
-        print(_("\t-j <jabberid for notifications:default is '%s'>") %
-              jabberid, file=fd)
-        print(_("\t-m <permatag for all versions of a post:default is blank>"),
-              file=fd)
-        print(_("\t-r <parent posts ID:defaults to none>"), file=fd)
-        print(_("\t-t <title of paste:default is blank>"), file=fd)
-        print(_("\t-u <username> -p <password>"), file=fd)
-        print(_("\t-v Print the version number"), file=fd)
-
-    # Set defaults
-    website = defaultPB
-    user = os.environ.get('USER')
-    jabberid = ""
-    title = ""
-    permatag = ""
-    format = "text"
-    username = ""
-    password = ""
-    filenames = []
-    content = ""
-    parentpid = ""
-
-    #Example configuration file string
-    configexample = """\
+    json = None
+
+#Enables input redirection during tests
+stdin = sys.stdin
+stderr = sys.stderr
+stdout = sys.stdout
+
+_ = gettext.gettext
+gettext.textdomain("pastebinit")
+
+# Timeout after 5s
+socket.setdefaulttimeout(5)
+
+# Version number to show in the usage
+version = "1.4.1"
+config_path = os.path.expanduser("~/.pastebinit.xml")
+verbose = False
+
+#Example configuration file string
+config_example = """\
 <pastebinit>
 <pastebin>http://paste.debian.net</pastebin>
 <author>A pastebinit user</author>
@@ -249,201 +73,396 @@
 </pastebinit>
 """
 
-    #Open configuration file if it exists
-    try:
-        f = open(configfile)
-        configtext = f.read()
+
+class PasteURLopener(FancyURLopener):
+    """Custom urlopener to handle 401's"""
+    version = "Pastebinit v%s" % version
+
+    def __init__(self, fmt=''):
+        self.fmt = fmt
+        FancyURLopener.__init__(self)
+
+    def http_error_401(self, url, fp, errcode, errmsg, headers, data=None):
+        return None
+
+    def format_params(self, params):
+        if self.fmt == 'json':
+            if not json:
+                print(_("Could not find any json library."), file=stderr)
+                sys.exit(1)
+            return json.dumps(params)
+        else:
+            return urlencode(params)
+
+    def make_request(self, url, params):
+        post_params = self.format_params(params)
+
+        if self.fmt == 'json':
+            self.addheader('Content-type', 'text/json')
+
+        if verbose:
+            print("POSTing to: %s" % url, file=stderr)
+            print("\nParams: %s" % str(post_params), file=stderr)
+
+        try:
+            return self.open(url, post_params)
+        except KeyboardInterrupt:
+            print(_("KeyboardInterrupt caught."), file=stderr)
+            sys.exit(1)
+        except Exception as e:
+            print(_("Failed to contact the server: %s") % e, file=stderr)
+            sys.exit(1)
+
+
+def get_default_pastebin():
+    default = "http://pastebin.com"
+    try:
+        import lsb_release
+        return {
+            'debian': "http://paste.debian.net",
+            'fedora': "http://fpaste.org",
+            'ubuntu': "http://paste.ubuntu.com",
+        }[lsb_release.get_distro_information()['ID'].lower()]
+    except KeyError:
+        return default
+    except ImportError:
+        return default
+
+
+def defaults():
+    return {
+        'website': get_default_pastebin(),
+        'user': os.environ.get('USER'),
+        'format': "text",
+        'jabberid': "",
+        'title': "",
+        'permatag': "",
+        'username': "",
+        'password': "",
+        'parentpid': "",
+        'verbose': False,
+    }
+
+
+def usage(fd):
+    print("pastebinit v" + version, file=fd)
+    print(_("Reads on stdin for input or takes a list of filenames "
+            "as parameters"), file=fd)
+    print(_("Optional arguments (not supported by all pastebins):"), file=fd)
+    print(_("\t-a <author:default is '%s'>") % defaults()['user'], file=fd)
+    print(_("\t-b <pastebin url:default is '%s'>") % defaults()['website'],
+          file=fd)
+    print(_("\t-f <format of paste:default is '%s' "
+            "(or from pastebin config)>") % defaults()['format'], file=fd)
+    print(_("\t-h This help screen"), file=fd)
+    print(_("\t-i <input file>"), file=fd)
+    print(_("\t-l List all supported pastebins"), file=fd)
+    print(_("\t-j <jabberid for notifications:default is '%s'>") %
+          defaults()['jabberid'], file=fd)
+    print(_("\t-m <permatag for all versions of a post:default is blank>"),
+          file=fd)
+    print(_("\t-r <parent posts ID:defaults to none>"), file=fd)
+    print(_("\t-t <title of paste:default is blank>"), file=fd)
+    print(_("\t-u <username> -p <password>"), file=fd)
+    print(_("\t-v Print the version number"), file=fd)
+    print(_("\t--verbose Verbose output to stderr"), file=fd)
+
+
+def preloadPastebins():
+    """
+    Check several places for config files:
+     - global config in /etc/pastebin.d
+     - for source checkout, config in the checkout
+     - user's overrides in ~/.pastebin.d
+    Files found later override files found earlier.
+    """
+    pastebind = {}
+    for confdir in ['/usr/share/pastebin.d',
+                    '/etc/pastebin.d',
+                    '/usr/local/etc/pastebin.d',
+                    os.path.expanduser('~/.pastebin.d'),
+                    os.path.join(
+                        os.path.dirname(os.path.realpath(__file__)),
+                        'pastebin.d'),
+                    ]:
+        try:
+            confdirlist = [
+                x for x in os.listdir(confdir)
+                if x.endswith('.conf') and not x.startswith('.')
+            ]
+        except OSError:
+            continue
+
+        for fileitem in confdirlist:
+            filename = os.path.join(confdir, fileitem)
+            parser = SafeConfigParser()
+            try:
+                parser.read(filename)
+            except UnicodeError:
+                print(_('%s: Unicode Error') % filename, file=stderr)
+                continue
+
+            if not parser.has_section('pastebin'):
+                print(_('%s: no section [pastebin]') % filename,
+                      file=stderr)
+                continue
+
+            if not parser.has_option('pastebin', 'basename'):
+                print(_("%s: no 'basename' in [pastebin]") % filename,
+                      file=stderr)
+                continue
+
+            pastebind[parser.get('pastebin', 'basename')] = parser
+    return pastebind
+
+
+def doParentFixup(paramname, settings):
+    """
+    pastey.net obfuscates parent ids for replies.  Rather than taking the
+    post ID given as the parent ID, we must handle this by going to that
+    post page and looking up what the invisible parent ID field will be
+    set to for children.
+    """
+    website = settings['website']
+    parentid = settings['parentid']
+    if not parentid:
+        return ""
+    if not website.endswith("/"):
+        website += "/"
+    matches = re.split(
+        '<input.*?name="' + paramname + '".*?value="(.*?)"',
+        PasteURLopener(settings.get('post_format', '')).open(
+            website + parentid.lstrip('/'), None).read()
+    )
+    if len(matches) <= 1 or re.match(parentid, matches[1]) is None:
+        # The obfuscated version didn't begin with the partial version,
+        # or unable to find the obfuscated version for some reason!
+        # Create a paste with no parent (should we throw, instead?)
+        return ""
+    return matches[1]
+
+
+def getPasteConfig(website):
+    for paste_name, paste_config in preloadPastebins().items():
+        basename = paste_config.get('pastebin', 'basename')
+        if basename == website or paste_name == website:
+            website = "http://%s" % basename
+        if re.search(paste_config.get('pastebin', 'id_regexp'), website):
+            if not website.endswith("/"):
+                website += "/"
+            return paste_config, website
+
+    print(_("Unknown website, please post a bugreport to request this "
+            "pastebin to be added (%s)") % website,
+          file=stderr)
+    sys.exit(1)
+
+
+def readPasteConfig(paste_config, settings):
+    """Return the parameters array for the selected pastebin"""
+
+    for param in paste_config.options('pastebin'):
+        settings[param] = paste_config.get('pastebin', param)
+
+    post_params = {}
+    for param in paste_config.options('format'):
+        paramname = paste_config.get('format', param)
+
+        if param == 'content':
+            pass
+        elif param == 'parentpid':
+            post_params[paramname] = doParentFixup(paramname, settings)
+        else:
+            try:
+                post_params[paramname] = paste_config.get('defaults', param)
+            except NoOptionError:
+                pass
+            except NoSectionError:
+                pass
+
+            try:
+                post_params[paramname] = settings[param]
+            except KeyError:
+                pass
+
+    return post_params
+
+
+def getText(parent_node):
+    """Read the text from an XML node"""
+    rc = ""
+    for node in parent_node.childNodes:
+        if node.nodeType == node.TEXT_NODE:
+            rc += node.data
+        else:
+            rc += getText(node)
+    return rc
+
+
+def readFile(location, mode="r"):
+    f = open(location, mode)
+    try:
+        return f.read()
+    except KeyboardInterrupt:
+        print(_("KeyboardInterrupt caught."), file=stderr)
+        sys.exit(1)
+    except:
+        print(_("Unable to read from: %s") % location, file=stderr)
+        sys.exit(1)
+    finally:
         f.close()
-        gotconfigxml = True
+
+
+def readConfigFile(config_text):
+    """Update settings from config XML"""
+    try:
+        settings = {}
+        configxml = xml.dom.minidom.parseString(config_text)
+        for xml_var, setting in (('pastebin', 'website'),
+                                 ('author', 'user'),
+                                 ('format', 'format'),
+                                 ('jabberid', 'jabberid')):
+            try:
+                settings[setting] = getText(
+                    configxml.getElementsByTagName(xml_var)[0])
+            except IndexError:
+                pass
+        return settings
     except KeyboardInterrupt:
-        print(_("KeyboardInterrupt caught."), file=sys.stderr)
+        print(_("KeyboardInterrupt caught."), file=stderr)
         sys.exit(1)
     except:
-        gotconfigxml = False
-
-    #Parse configuration file
-    if gotconfigxml:
+        print(_("Error parsing configuration file!"), file=stderr)
+        print(_("Please ensure that your configuration file looks "
+                "similar to the following:"), file=stderr)
+        print(config_example, file=stderr)
+        sys.exit(1)
+
+
+def getOpts(args=sys.argv[1:]):
+    try:
+        optlist, arglist = getopt.getopt(
+            args, 'hvli:f:b:a:r:j:t:m:u:p:', ('verbose',))
+    except getopt.GetoptError as e:
+        print(_("Invalid arguments: %s!" % e) + "\n", file=stderr)
+        usage(stderr)
+        sys.exit(1)
+    opts = dict(optlist)
+    opts['-i'] = [opt[1] for opt in optlist if opt[0] == "-i"] + arglist
+    return opts
+
+
+def readOptions(opts):
+    settings = {}
+
+    # Die if support information is requested
+    if "-h" in opts:
+        usage(stdout)
+        sys.exit(0)
+    if "-v" in opts:
+        print("pastebinit v" + version, file=stdout)
+        sys.exit(0)
+    if "-l" in opts:
+        print(_("Supported pastebins:"), file=stdout)
+        for pastebin in sorted(preloadPastebins()):
+            print("- %s" % pastebin, file=stdout)
+        sys.exit(0)
+
+    # Configure the environment
+    def setFromOpt(opt, label):
         try:
-            configxml = xml.dom.minidom.parseString(configtext)
-            for variable, key in (('pastebin', 'website'), ('author', 'user'),
-                                  ('format', 'format'),
-                                  ('jabberid', 'jabberid')):
-                try:
-                    value = getFirstNodeText(configxml, variable)
-                    vars()[key] = value
-                except:
-                    pass
-        except KeyboardInterrupt:
-            print(_("KeyboardInterrupt caught."), file=sys.stderr)
-            sys.exit(1)
-        except:
-            print(_("Error parsing configuration file!"), file=sys.stderr)
-            print(_("Please ensure that your configuration file looks "
-                    "similar to the following:"), file=sys.stderr)
-            print(configexample, file=sys.stderr)
-            sys.exit(1)
-
-    # Get options
+            settings[label] = opts[opt]
+        except KeyError:
+            pass
+
+    setFromOpt('-b', 'website')
+    setFromOpt('-a', 'user')
+    setFromOpt('-f', 'format')
+    setFromOpt('-j', 'jabberid')
+    setFromOpt('-t', 'title')
+    setFromOpt('-m', 'permatag')
+    setFromOpt('-u', 'username')
+    setFromOpt('-p', 'password')
+    setFromOpt('-r', 'parentpid')
+    settings['verbose'] = "--verbose" in opts
+
+    settings['contents'] = [readFile(fname, "rb") for fname in opts['-i']]
+    if not settings['contents']:
+        settings['contents'].append(stdin.read())
+    return settings
+
+
+def readPageURL(page, settings):
+    website = settings['website']
+    result_type = settings.get('result_type', None)
     try:
-        optlist, arglist = getopt.getopt(sys.argv[1:],
-                                         'hvli:f:b:a:r:j:t:m:u:p:')
-    except KeyboardInterrupt:
-        print(_("KeyboardInterrupt caught."), file=sys.stderr)
-        sys.exit(1)
-    except getopt.GetoptError:
-        print(_("Invalid arguments!\n"), file=sys.stderr)
-        Usage(sys.stderr)
-        sys.exit(1)
-
-    # Get the config
-    pastebind = preloadPastebins()
-
-    # Iterate through options
-    for opt in optlist:
-        if opt[0] == "-h":
-            Usage()
-            sys.exit(0)
-        if opt[0] == "-i":
-            filenames.append(opt[1])
-        elif opt[0] == "-f":
-            format = opt[1]
-        elif opt[0] == "-b":
-            website = opt[1]
-        elif opt[0] == "-a":
-            user = opt[1]
-        elif opt[0] == "-r":
-            parentpid = opt[1]
-        elif opt[0] == "-j":
-            jabberid = opt[1]
-        elif opt[0] == "-l":
-            print(_("Supported pastebins:"))
-            for pastebin in sorted(pastebind):
-                print("- %s" % pastebin)
-            sys.exit(0)
-        elif opt[0] == "-t":
-            title = opt[1]
-        elif opt[0] == "-m":
-            permatag = opt[1]
-        elif opt[0] == "-u":
-            username = opt[1]
-        elif opt[0] == "-p":
-            password = opt[1]
-        elif opt[0] == "-v":
-            print("pastebinit v" + version)
-            sys.exit(0)
-
-    filenames += arglist
-
-    if not filenames:
-        filenames.append("-")
-
-    contents = []
-    for filename in filenames:
-        # If - is specified as a filename read from stdin
-        # otherwise load the specified files.
-        if filename == "-":
-            content = sys.stdin.read()
+        # Check if we have to apply a regexp
+        page_url = ""
+        if result_type:
+            result = page.read().decode('utf-8')
+            result_regex = settings.get('result_regexp', None)
+            if result_regex:
+                result = re.split(result_regex, result)[1]
+
+            if result_type == 'url':
+                page_url = result.strip()
+            elif result_type == 'id':
+                page_url = settings.get('target_url', website) + result.strip()
+            else:
+                raise ValueError(_("Unsupported result_type: %s" % result_type))
         else:
-            try:
-                with open(filename, "rb") as fd:
-                    content = fd.read()
-            except KeyboardInterrupt:
-                print(_("KeyboardInterrupt caught."), file=sys.stderr)
-                sys.exit(1)
-            except:
-                print(_("Unable to read from: %s") % filename, file=sys.stderr)
-                sys.exit(1)
-
+            page_url = page.url
+
+        # You may have posted to https:, but were given an http: url
+        if website.startswith('https:'):
+            page_url = page_url.replace('http:', 'https:')
+        return page_url
+
+    except KeyboardInterrupt:
+        print(_("KeyboardInterrupt caught."), file=stderr)
+        sys.exit(1)
+    except:
+        print(_("Unable to read or parse the result page, it could be a "
+                "server timeout or a change server side, "
+                "try with another pastebin."), file=stderr)
+        sys.exit(1)
+
+
+def validateContents(settings):
+    """Check for invalid contents"""
+    for content in settings['contents']:
         if not content:
             print(_("You are trying to send an empty document, exiting."),
-                  file=sys.stderr)
-            sys.exit(1)
-
-        contents.append(content)
-
-    for content in contents:
-        # Get the parameter array
-        website, params = getParameters(website, pastebind, content, user,
-                                        jabberid, version, format, parentpid,
-                                        permatag, title, username, password)
-
-        if not website.endswith("/"):
-            website += "/"
-
-        if "sizelimit" in params:
-            if len(content) > int(params['sizelimit']):
-                print(_("The content you are trying to send exceeds "
-                        "the pastebin's size limit."), file=sys.stderr)
-                sys.exit(1)
-            else:
-                del params['sizelimit']
-
-        reLink = None
-        tmp_page = ""
-        if "page" in params:
-            website += params['page']
-            tmp_page = params['page']
-            del params["page"]
-        if "regexp" in params:
-            reLink = params['regexp']
-            del params["regexp"]
-        if "target_url" in params:
-            target_url = params["target_url"]
-            del params["target_url"]
-        else:
-            target_url = None
-        if 'post_format' in params:
-            post_format = params['post_format']
-            del params['post_format']
-        else:
-            post_format = 'standard'
-
-        url_opener = pasteURLopener()
-
-        if post_format == 'json':
-            if json:
-                params = json.dumps(params)
-                url_opener.addheader('Content-type', 'text/json')
-            else:
-                print(_("Could not find any json library."), file=sys.stderr)
-                sys.exit(1)
-        else:
-            # Convert to a format usable with the HTML POST
-            params = urlencode(params)
-
-        # Send the informations and be redirected to the final page
-        try:
-            page = url_opener.open(website, params)
-        except Exception as e:
-            print(_("Failed to contact the server: %s") % e, file=sys.stderr)
-            sys.exit(1)
-
-        try:
-            # Check if we have to apply a regexp
-            if reLink:
-                if target_url:
-                    website = target_url
-                else:
-                    website = website.replace(tmp_page, "")
-
-                if reLink == '(.*)':
-                    print(page.read().decode('utf-8').strip())
-                else:
-                    # Print the result of the regexp
-                    print(website + re.split(reLink,
-                          page.read().decode('utf-8'))[1])
-            else:
-                # Get the final page and show the url
-                print(page.url)
-        except KeyboardInterrupt:
-            print(_("KeyboardInterrupt caught."), file=sys.stderr)
-            sys.exit(1)
-        except:
-            print(_("Unable to read or parse the result page, it could be a "
-                    "server timeout or a change server side, "
-                    "try with another pastebin."), file=sys.stderr)
-            sys.exit(1)
-
-except KeyboardInterrupt:
-    print(_("KeyboardInterrupt caught."), file=sys.stderr)
-    sys.exit(1)
+                  file=stderr)
+            sys.exit(1)
+
+        if len(content) > int(settings.get('sizelimit', sys.maxsize)):
+            print(_("The content you are trying to send exceeds "
+                    "the pastebin's size limit."), file=stderr)
+            sys.exit(1)
+
+
+def main():
+    try:
+        settings = defaults()
+        settings.update(readConfigFile(readFile(config_path)))
+        settings.update(readOptions(getOpts()))
+        verbose = settings['verbose']
+
+        paste_config, settings['website'] = getPasteConfig(settings['website'])
+        params = readPasteConfig(paste_config, settings)
+
+        validateContents(settings)
+        opener = PasteURLopener(settings.get('post_format', ''))
+
+        url = settings['website'] + settings.get('page', '').lstrip("/")
+        for content in settings['contents']:
+            params[paste_config.get('format', 'content')] = content
+            page = opener.make_request(url, params)
+            print(readPageURL(page, settings), file=stdout)
+    except KeyboardInterrupt:
+        print(_("KeyboardInterrupt caught."), file=stderr)
+        sys.exit(1)
+
+if __name__ == '__main__':
+    main()
 \ No newline at end of file
 
=== added symlink 'pastebinit.py'
=== target is u'pastebinit'
=== added file 'test.py'
--- test.py	1970-01-01 00:00:00 +0000
+++ test.py	2014-12-02 02:23:09 +0000
@@ -0,0 +1,135 @@
+import unittest
+import pastebinit
+try:
+    from StringIO import StringIO
+except ImportError:
+    from io import StringIO
+
+pastebinit.stdin = StringIO('')
+pastebinit.stdout = StringIO('')
+pastebinit.stderr = StringIO('')
+
+
+class ConfigXMLTests(unittest.TestCase):
+
+    def test_config_xml_parse(self):
+        xml = """\
+        <pastebinit>
+        <pastebin>http://paste.debian.net</pastebin>
+        <author>A pastebinit user</author>
+        <jabberid>[e-mail]</jabberid>
+        <format>text</format>
+        </pastebinit>
+        """
+        out = {
+            'website': u'http://paste.debian.net',
+            'user': u'A pastebinit user',
+            'jabberid': u'[e-mail]',
+            'format': u'text',
+        }
+        assert pastebinit.readConfigFile(xml) == out, 'readConfigFile failed'
+
+    def test_config_xml_parse_missing(self):
+        xml = """\
+        <pastebinit>
+        <author>A pastebinit user</author>
+        <jabberid>[e-mail]</jabberid>
+        <format>text</format>
+        </pastebinit>
+        """
+        out = {
+            'user': u'A pastebinit user',
+            'jabberid': u'[e-mail]',
+            'format': u'text',
+        }
+        assert pastebinit.readConfigFile(xml) == out, 'readConfigFile failed'
+
+    def test_config_xml_parse_additional(self):
+        xml = """\
+        <pastebinit>
+        <author>A pastebinit user</author>
+        <jabberid>[e-mail]</jabberid>
+        <format>text</format>
+        <test>nonsense</test>
+        </pastebinit>
+        """
+        out = {
+            'user': u'A pastebinit user',
+            'jabberid': u'[e-mail]',
+            'format': u'text',
+        }
+        assert pastebinit.readConfigFile(xml) == out, 'readConfigFile failed'
+
+    def test_config_xml_parse_invalid(self):
+        try:
+            pastebinit.readConfigFile("invalid")
+            assert False, "Invalid file was parsed without error."
+        except SystemExit:
+            pass
+
+
+class GetOptsTests(unittest.TestCase):
+
+    def _check_arg(self, argname, settingname):
+        settings = pastebinit.readOptions(pastebinit.getOpts([argname, 'test']))
+        assert settings[settingname] == 'test', "Failed to read " + argname
+
+    def test_help(self):
+        try:
+            pastebinit.readOptions(['-h'])
+            assert False, "Failed to quit after help option"
+        except SystemExit:
+            pass
+
+    def test_list(self):
+        try:
+            pastebinit.readOptions(['-l'])
+            assert False, "Failed to quit after list option"
+        except SystemExit:
+            pass
+
+    def test_version(self):
+        try:
+            pastebinit.readOptions(['-v'])
+            assert False, "Failed to quit after version option"
+        except SystemExit:
+            pass
+
+    def test_website(self):
+        self._check_arg('-b', 'website')
+
+    def test_user(self):
+        self._check_arg('-a', 'user')
+
+    def test_format(self):
+        self._check_arg('-f', 'format')
+
+    def test_jabberid(self):
+        self._check_arg('-j', 'jabberid')
+
+    def test_title(self):
+        self._check_arg('-t', 'title')
+
+    def test_permatag(self):
+        self._check_arg('-m', 'permatag')
+
+    def test_username(self):
+        self._check_arg('-u', 'username')
+
+    def test_password(self):
+        self._check_arg('-p', 'password')
+
+    def test_parentpid(self):
+        self._check_arg('-r', 'parentpid')
+
+    def test_filenames(self):
+        opts = pastebinit.getOpts(['-i', 'testa',
+                                   '-i', 'testb',
+                                   '-i', 'testc',
+                                   ])
+        assert opts['-i'] == [
+            'testa','testb','testc'], "Failed to read filenames"
+
+
+if __name__ == "__main__":
+    unittest.main()
 
=== modified file 'test.sh'
--- test.sh	2014-01-06 23:23:28 +0000
+++ test.sh	2014-12-02 02:23:09 +0000
@@ -1,11 +1,16 @@
 #!/bin/sh
-teststring="blah blah blah"
+#paste.debian.net needs 3 lines
+teststring="""\
+test from pastebinit
+test from pastebinit
+test from pastebinit
+"""
 
 for interpreter in python python3; do
-    for pastebin in $($interpreter pastebinit -l | egrep "^-" | sed "s/^- //g")
+    for pastebin in $(ls pastebin.d/*.conf | sed 's~^pastebin.d/\(.*\)\.conf$~\1~')
     do
         echo "Trying http://$pastebin ($interpreter)"
-        URL=$(echo "$teststring\n$teststring\n$teststring" | $interpreter pastebinit -b http://$pastebin)
+        URL=$(echo teststring | $interpreter ./pastebinit.py -b http://$pastebin)
 
         if [ "$pastebin" = "paste.ubuntu.org.cn" ]; then
             out=$(wget -O - -q "$URL" | gzip -d | grep "$teststring")
@@ -14,9 +19,13 @@
         fi
 
         if [ -n "$out" ]; then
-            echo "PASS: http://$pastebin ($URL) ($interpreter)"
+            echo "PASS: http://$pastebin"
+            echo "      ($URL)"
+            echo "      ($interpreter)"
         else
-            echo "FAIL: http://$pastebin ($URL) ($interpreter)"
+            echo "FAIL: http://$pastebin"
+            echo "      ($URL)"
+            echo "      ($interpreter)"
         fi
         echo ""
     done