Merge lp:~cyphermox/pastebinit/dev-arbitrary-dest into lp:pastebinit

[Mathieu Trudel-Lapierre]

[Mathieu Trudel-Lapierre (cyphermox)]

Fixes bug #369795 by proposing a format for configuration files that define new personal/public pastebins.

Most pastebins already supported in the script have been factored out to config files in this branch, and commented out in the main script. The only ones kept in the main script (but also available as config files), are pastebin.com and pastebin.ca.

[Stéphane Graber (stgraber)]

Hi,

Thanks a lot for your patch.
Before merging it, may I suggest you improve the README file a bit ?
I found it quite confusing (though then reading the actual configuration of a pastebin, makes it a lot easier to understand) and contains a few typos (psatebin instead of pastebin for example).

Also, in the code, I'd suggest getting rid of the old code instead of commenting it as soon as it's converted to the new format. We have bzr if we need to retrieve older revisions.

Again, thanks for all the work, it's really appreciated.

review: Needs Fixing

[lp:~cyphermox/pastebinit/dev-arbitrary-dest]

71. By Matt Trudel <mtrudel@mtrudel-laptop> on 2009-08-14

- Cleanup useless comments.
- Rework wording for the README file.

72. By Matt Trudel <mtrudel@mtrudel-laptop> on 2009-08-14

- Spellchecking pass on README.

[Mathieu Trudel-Lapierre (cyphermox)]

Fixed the spelling and removed the commented stuff.

I've also reworked the README file and hopefully made it much clearer. If not, please let me know where things are messy or confusing, so that I can fix it :)

Thanks,

/ Matt

review: Needs Resubmitting

Unmerged revisions

=== added file 'README'
--- README	1970-01-01 00:00:00 +0000
+++ README	2009-08-13 02:56:14 +0000
@@ -0,0 +1,113 @@
+
+USING PASTEBIN.D FILES
+
+To enable support for private pastebins, first you will need to know the
+fields that are in use by the pastebin when posting data.
+
+Add a file in the /etc/pastebin.d directory with the following format:
+
+[psatebin]
+basename = (the domain name of the pastebin site, without subdomains)
+regexp = (a regular expression that identifies the base site, and any
+		subdomains that may be used)
+
+[format]
+pastebinit_field0 = site_field0
+pastebinit_fieldN = site_fieldN
+custom_field0 = site_fieldN
+custom_fieldN = site_fieldN
+
+[defaults]
+custom_field0 = value
+custom_fieldN = value
+
+
+THE PASTEBIN SECTION
+
+Under "[pastebin]", you will want to keep the very basic information to
+identify the pastebin. Only two fields are used in that section:
+
+	basename - 	the generic domain name for the pastebin.
+			this domain name should not contain
+			possible subdomains in use.
+
+	regexp -	a regular expression that matches the
+			basename, as well as any subdomains that
+			may be in use.
+
+
+THE FORMAT SECTION
+
+Under "[format]", identify the various fields in use in the pastebin you
+are trying to add support for. Identify any fields used to publish data
+on the pastebin, and add them to the configuration file.
+
+The [format] section expects keywords from the pastebinit utility, for
+which the values will be the actual name of the field in which to put the
+data for that pastebin.
+
+A number of reserved keywords can be used, but are optional:
+
+	user - 		contains the username for the user
+			calling the pastebinit utility.
+
+	content -	contains the data that will be posted.
+
+	title -		contains the title if set at the command
+			line.
+
+	version -	contains the version as set at the
+			command line.
+
+	format -	contains the format setting:
+			usually used for syntax highlighting.
+
+	parentpid -	contains the parent post ID for 
+			pastebins that use it.
+
+	permatag -	contains permatag as set at the command
+			line.
+
+	username -	contains the pastebin username if 
+			required.
+
+	password -	contains the pastebin user's password
+			if required.
+
+	jabberid -	contains the jabberid for the poster
+			as set at the command line.
+
+Two additional special parameters are available for use:
+
+	page -		used to specify a page from which to
+			post data. It is the actual URL of
+			the pastebin's form.
+
+	regexp -	used to specify a regexp to execute
+			on the resulting page after posting.
+			This is useful to deal with special
+			pastebins that don't redirect you
+			to the new post's URL.
+
+Any other fields in use for the specific pastebin you are trying to enable
+may be added to the section using a similar syntax. An easy way to deal with
+such a requirement is to repeat the same keyword, such as:
+
+[format]
+expire = expire
+
+Where 'page' can then be reused as the keyword to retrieve a static value,
+as set in the '[defaults]' section.
+
+
+THE DEFAULTS SECTION
+
+The '[defaults]' section is used to set static values for custom fields to
+retrieve. Some pastebins require setting fields such as expiry time for
+posts, or whether to use cookies. Such values would be set using a custom
+field if they are deemed necessary. In the above example, to statically
+set a page to be used in the 'page' field:
+
+[defaults]
+expire = 3600
+
 
=== added directory 'pastebin.d'
=== added file 'pastebin.d/fpaste.org.conf'
--- pastebin.d/fpaste.org.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/fpaste.org.conf	2009-08-11 02:26:03 +0000
@@ -0,0 +1,17 @@
+# Another website that doesn't work due to unusual changes.
+# this is the Fedora Pastebin, sadly.
+
+[pastebin]
+basename = fpaste.org
+regexp = "http://fpaste.org"
+
+[format]
+user = author
+content = content
+format = lang
+submit = submit
+expire_options = expire_options
+
+[defaults]
+submit = "paste it!"
+expire_options = 3600
 
=== added file 'pastebin.d/gist.github.com.conf'
--- pastebin.d/gist.github.com.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/gist.github.com.conf	2009-08-13 02:16:18 +0000
@@ -0,0 +1,13 @@
+[pastebin]
+basename = gist.github.com
+regexp = "http://gist.github.com"
+
+[format]
+page = page
+user = poster
+format = file_ext[gistfile1]
+content = file_contents[gistfile1]
+title = file_name[gistfile1]
+
+[defaults]
+page = '/gists'
 
=== added file 'pastebin.d/paste.debian.net.conf'
--- pastebin.d/paste.debian.net.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/paste.debian.net.conf	2009-08-13 02:16:18 +0000
@@ -0,0 +1,16 @@
+[pastebin]
+basename = paste.debian.net
+regexp = "http://paste.debian.net"
+
+[format]
+user = poster
+content = code
+format = syntax
+lang = lang
+remember = remember
+expire = expire
+
+[defaults]
+lang = -1
+remember = 0
+expire = 259200
 
=== added file 'pastebin.d/paste.ubuntu.com.conf'
--- pastebin.d/paste.ubuntu.com.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/paste.ubuntu.com.conf	2009-05-03 15:06:09 +0000
@@ -0,0 +1,10 @@
+[pastebin]
+basename = paste.ubuntu.com
+regexp = "http://paste.ubuntu.com"
+
+[format]
+user = poster
+content = content
+format = syntax
+
+[defaults]
 
=== added file 'pastebin.d/paste2.org.conf'
--- pastebin.d/paste2.org.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/paste2.org.conf	2009-08-13 02:16:18 +0000
@@ -0,0 +1,14 @@
+[pastebin]
+basename = paste2.org
+regexp = "http://paste2.org"
+
+[format]
+title = description
+content = code
+format = lang
+page = page
+parent = parent
+
+[defaults]
+page = '/new-paste'
+parent = 0
 
=== added file 'pastebin.d/pastebin.ca.conf'
--- pastebin.d/pastebin.ca.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/pastebin.ca.conf	2009-05-03 15:06:09 +0000
@@ -0,0 +1,17 @@
+[pastebin]
+basename = pastebin.ca
+regexp = "http://((([a-zA-Z0-9\-_\.]*)(pastebin\.ca)))"
+
+[format]
+user = name
+content = content
+type = type
+save = save
+submit = submit
+regexp = regexp
+
+[defaults]
+type = 1
+save = 0
+submit = s
+regexp = '">http://.*pastebin.ca/(.*)</a></p><p>'
 
=== added file 'pastebin.d/pastebin.com.conf'
--- pastebin.d/pastebin.com.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/pastebin.com.conf	2009-05-03 15:06:09 +0000
@@ -0,0 +1,19 @@
+[pastebin]
+basename = pastebin.com
+regexp = "http://((([a-zA-Z0-9\-_\.]*)(pastebin\.com)))"
+#FIXME: not website == "http://www.pastebin.com") or website == "http://pastebin.mozilla.org":
+
+[format]
+user = poster
+content = code2
+version = version
+parentpid = parent_pid
+format = format
+submit = paste
+remember = remember
+expiry = expiry
+
+[defaults]
+remember = f
+expiry = 1
+paste = Send
 
=== added file 'pastebin.d/pastey.net.conf'
--- pastebin.d/pastey.net.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/pastey.net.conf	2009-05-08 01:40:02 +0000
@@ -0,0 +1,18 @@
+[pastebin]
+basename = pastey.net
+regexp = "http://((([a-zA-Z0-9\-_\.]*)(pastey\.net)))"
+
+[format]
+user = author
+title = subject
+parentpid = parent
+content = text
+format = language
+paste = paste
+page = page
+regexp = regexp
+
+[defaults]
+paste = "Paste"
+page = '/submit.php'
+regexp = '">http://(?:(?:[a-zA-Z0-9\-_\.]*)(?:pastey\.net))/(.*)</a>'
 
=== added file 'pastebin.d/rafb.net.conf'
--- pastebin.d/rafb.net.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/rafb.net.conf	2009-08-11 02:26:03 +0000
@@ -0,0 +1,15 @@
+[pastebin]
+basename = rafb.net
+regexp = "http://rafb.net"
+
+[format]
+user = nick
+content = text
+page = page
+lang = lang
+cvt_tabs = cvt_tabs
+
+[defaults]
+page = "/paste/paste.php"
+lang = "Plain Text"
+cvt_tabs = "No"
 
=== added file 'pastebin.d/slexy.org.conf'
--- pastebin.d/slexy.org.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/slexy.org.conf	2009-08-11 02:26:03 +0000
@@ -0,0 +1,24 @@
+[pastebin]
+basename = slexy.org
+regexp = "http://slexy.org"
+
+[format]
+user = author
+content = raw_paste
+title = desc
+page = page
+language = language
+permissions = permissions
+linenumbers = linenumbers
+comment = comment
+submit = submit
+expire = expire
+
+[defaults]
+submit = "Submit Paste"
+page = "/index.php/submit"
+language = "text"
+permissions = 0
+comment = ""
+linenumbers = 0
+expire = 0
 
=== added file 'pastebin.d/stikked.com.conf'
--- pastebin.d/stikked.com.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/stikked.com.conf	2009-08-13 02:16:18 +0000
@@ -0,0 +1,18 @@
+[pastebin]
+basename = stikked.com
+regexp = "http://stikked.com"
+
+[format]
+user = name
+content = code
+title = title
+email = email
+lang = lang
+website = website
+submit = submit
+
+[defaults]
+email = ""
+lang = "text"
+website = ""
+submit = "submit"
 
=== added file 'pastebin.d/yourpaste.net.conf'
--- pastebin.d/yourpaste.net.conf	1970-01-01 00:00:00 +0000
+++ pastebin.d/yourpaste.net.conf	2009-08-13 02:16:18 +0000
@@ -0,0 +1,21 @@
+[pastebin]
+basename = yourpaste.net
+regexp = "http://yourpaste.net"
+
+[format]
+format = syntax
+user = name
+title = desc
+expire = expire
+content = code
+private = private
+remember = remember
+page = page
+regexp = regexp
+
+[defaults]
+expire = 0
+private = 0
+remember = 0
+page = '/paste'
+regexp = '">http://yourpaste.net(.*)</a>'
 
=== modified file 'pastebinit'
--- pastebinit	2009-03-10 05:23:09 +0000
+++ pastebinit	2009-08-13 02:11:21 +0000
@@ -20,6 +20,7 @@
 try:
     import urllib, os, sys, re, getopt, select, xml.dom.minidom, gettext
     from gettext import gettext as _
+    from configobj import ConfigObj
 
     gettext.textdomain("pastebinit")
 
@@ -32,16 +33,26 @@
         def http_error_401(self, url, fp, errcode, errmsg, headers, data=None):
             return None
 
+    def preloadPastebins():
+	confdir = '/etc/pastebin.d/'
+	confdirlist = os.listdir(confdir)
+	pastebind = {}
+	for fileitem in confdirlist:
+		bininstance = ConfigObj(confdir + fileitem)
+		basename = bininstance['pastebin']['basename']
+		pastebind[basename] = bininstance
+	return pastebind
+
     # pastey.net obfuscates parent ids for replies.  Rather than taking the
     # post ID given as the parent ID, we must handle this by going to that
     # post page and looking up what the invisible parent ID field will be
     # set to for children.
-    def pasteyParentFixup(website, parentid):
+    def doParentFixup(website, paramname, parentid):
         if parentid == "":
             return ""
         url_opener = pasteURLopener()
         page = url_opener.open(website + '/' + parentid, None)
-        matches = re.split('<input.*?name="parent".*?value="(.*?)"', page.read())
+        matches = re.split('<input.*?name="' + paramname + '".*?value="(.*?)"', page.read())
         if len(matches) <= 1 or re.match(parentid, matches[1]) == None:
             # The obfuscated version didn't begin with the partial version,
             # or unable to find the obfuscated version for some reason!
@@ -50,7 +61,7 @@
         return matches[1]
 
     #Return the parameters depending of the pastebin used
-    def getParameters(website, content, user, jabberid, version, format, parentpid, permatag, title, username, password):
+    def getParameters(website, pastebind, content, user, jabberid, version, format, parentpid, permatag, title, username, password):
         "Return the parameters array for the selected pastebin"
         params={}
         # pastebin.com v0.50
@@ -84,81 +95,113 @@
 #            params['username'] = username
 #            params['password'] = password
 #            params['version'] = version
-        elif website == "http://rafb.net":
-            params['page'] = "/paste/paste.php"
-            params['nick'] = user
-            params['text'] = content
-            params['lang'] = "Plain Text" #The format, for syntax hilighting
-            params['cvt_tabs'] = "No"
-        elif website == "http://stikked.com":
-            params['name'] = user
-            params['title'] = title
-            params['email'] = ""
-            params['lang'] = "text" #The format, for syntax hilighting
-            params['code'] = content
-            params['website'] = ""
-            params['submit'] = "submit"
-        elif website == "http://slexy.org":
-            params['page'] = "/index.php/submit"
-            params['raw_paste'] = content
-            params['comment'] = ""
-            params['author'] = user
-            params['language'] = "text" #The format, for syntax hilighting
-            params['permissions'] = "0"
-            params['desc'] = title
-            params['linenumbers'] = "0"
-            params['expire'] = "0"
-            params['submit'] = "Submit Paste"
-        elif website == "http://fpaste.org":
-            params['lang'] = format #The format, for syntax hilighting
-            params['page'] = "/paste/save"
-            params['content'] = content
-            params['submit'] = "pastebinit"
-        elif website == "http://paste2.org":
-            params['page'] = "/new-paste"
-            params['description'] = title
-            params['lang'] = format
-            params['code'] = content
-            params['parent'] = "0"
-        elif re.search("http://((([a-zA-Z0-9\-_\.]*)(pastey\.net)))", website):    
-            params['author'] = user
-            params['subject'] = title
-            params['parent'] = pasteyParentFixup(website, parentpid)
-            params['text'] = content
-            params['language'] = format # File format, as a string like "cpp" or "lua"
-            params['paste'] = "Paste"
-            params['page'] = '/submit.php'
-            params['regexp'] = '">http://(?:(?:[a-zA-Z0-9\-_\.]*)(?:pastey\.net))(/.*)</a>'
-        elif website == "http://yourpaste.net":
-            params['syntax'] = format
-            params['name'] = user
-            params['desc'] = title
-            params['expire'] = "0" # Forever
-            params['code'] = content
-            params['private'] = "0" # It's not a private post
-            params['remember'] = "0" # Cookies? ;)
-            params['page'] = "/paste"
-            params['regexp'] = '">http://yourpaste.net(.*)/</a>'
-        elif website == "http://gist.github.com":
-            params['page'] = "/gists"
-            params['poster'] = user
-            params['file_ext[gistfile1]'] = format #The format, for syntax hilighting
-            params['file_contents[gistfile1]'] = content
-            params['file_name[gistfile1]'] = title
-        elif website == "http://paste.ubuntu.com":
-            params['poster'] = user
-            params['syntax'] = format #The format, for syntax hilighting
-            params['content'] = content
-        elif website == "http://paste.debian.net":
-            params['poster'] = user
-            params['lang'] = "-1" #The format, for syntax hilighting, default to plain text
-            params['syntax'] = format #The format, for syntax hilighting
-            params['code'] = content
-            params['remember'] = "0" #Do you want a cookie ?
-            params['expire'] = "259200" # expire in 72h
+#        elif website == "http://rafb.net":
+#        if website == "http://rafb.net":
+#            params['page'] = "/paste/paste.php"
+#            params['nick'] = user
+#            params['text'] = content
+#            params['lang'] = "Plain Text" #The format, for syntax hilighting
+#            params['cvt_tabs'] = "No"
+#        elif website == "http://stikked.com":
+#            params['name'] = user
+#            params['title'] = title
+#            params['email'] = ""
+#            params['lang'] = "text" #The format, for syntax hilighting
+#            params['code'] = content
+#            params['website'] = ""
+#            params['submit'] = "submit"
+#        elif website == "http://slexy.org":
+#        if website == "http://slexy.org":
+#            params['page'] = "/index.php/submit"
+#            params['raw_paste'] = content
+#            params['comment'] = ""
+#            params['author'] = user
+#            params['language'] = "text" #The format, for syntax hilighting
+#            params['permissions'] = "0"
+#            params['desc'] = title
+#            params['linenumbers'] = "0"
+#            params['expire'] = "0"
+#            params['submit'] = "Submit Paste"
+#        elif website == "http://fpaste.org":
+#            params['lang'] = format #The format, for syntax hilighting
+#            params['page'] = "/paste/save"
+#            params['content'] = content
+#            params['submit'] = "pastebinit"
+#        elif website == "http://paste2.org":
+#            params['page'] = "/new-paste"
+#            params['description'] = title
+#            params['lang'] = format
+#            params['code'] = content
+#            params['parent'] = "0"
+#        elif re.search("http://((([a-zA-Z0-9\-_\.]*)(pastey\.net)))", website):    
+#            params['author'] = user
+#            params['subject'] = title
+#            params['parent'] = pasteyParentFixup(website, parentpid)
+#            params['text'] = content
+#            params['language'] = format # File format, as a string like "cpp" or "lua"
+#            params['paste'] = "Paste"
+#            params['page'] = '/submit.php'
+#            params['regexp'] = '">http://(?:(?:[a-zA-Z0-9\-_\.]*)(?:pastey\.net))(/.*)</a>'
+#        elif website == "http://yourpaste.net":
+#            params['syntax'] = format
+#            params['name'] = user
+#            params['desc'] = title
+#            params['expire'] = "0" # Forever
+#            params['code'] = content
+#            params['private'] = "0" # It's not a private post
+#            params['remember'] = "0" # Cookies? ;)
+#            params['page'] = "/paste"
+#            params['regexp'] = '">http://yourpaste.net(.*)/</a>'
+#        elif website == "http://gist.github.com":
+#            params['page'] = "/gists"
+#            params['poster'] = user
+#            params['file_ext[gistfile1]'] = format #The format, for syntax hilighting
+#            params['file_contents[gistfile1]'] = content
+#            params['file_name[gistfile1]'] = title
+#        elif website == "http://paste.ubuntu.com":
+#            params['poster'] = user
+#            params['syntax'] = format #The format, for syntax hilighting
+#            params['content'] = content
+#        elif website == "http://paste.debian.net":
+#            params['poster'] = user
+#            params['lang'] = "-1" #The format, for syntax hilighting, default to plain text
+#            params['syntax'] = format #The format, for syntax hilighting
+#            params['code'] = content
+#            params['remember'] = "0" #Do you want a cookie ?
+#            params['expire'] = "259200" # expire in 72h
         else:
-            sys.exit(_("Unknown website, please post a bugreport to request this pastebin to be added (%s)") % website)
-        return params
+		for pastebin in pastebind:
+			if re.search( pastebind[pastebin]['pastebin']['regexp'], website ):
+				for param in pastebind[pastebin]['format'].keys():
+					paramname = pastebind[pastebin]['format'][param]
+					if param == 'user':
+						params[paramname] = user
+					elif param == 'content':
+						params[paramname] = content			
+					elif param == 'title':
+						params[paramname] = title
+					elif param == 'version':
+						params[paramname] = version
+					elif param == 'format':
+						params[paramname] = format
+					elif param == 'parentpid':
+						#params[paramname] = parentpid
+            					params[paramname] = doParentFixup(website, paramname, parentpid)
+					elif param == 'permatag':
+						params[paramname] = parmatag
+					elif param == 'username':
+						params[paramname] = username
+					elif param == 'password':
+						params[paramname] = password
+					elif param == 'jabberid':
+						params[paramname] = jabberid
+					else:
+						params[paramname] = pastebind[pastebin]['defaults'][paramname]
+				#print params
+	if params:
+        	return params
+	else:
+		sys.exit(_("Unknown website, please post a bugreport to request this pastebin to be added (%s)") % website)
 
     #XML Handling methods
     def getText(nodelist):
@@ -297,7 +340,8 @@
             sys.exit(_("Unable to read from: %s") % filename)
     if not content:
         sys.exit(_("You are trying to send an empty document, exiting.")) 
-    params = getParameters(website, content, user, jabberid, version, format, parentpid, permatag, title, username, password) #Get the parameters array
+    pastebind = preloadPastebins() #get the config from /etc/pastebin.d/
+    params = getParameters(website, pastebind, content, user, jabberid, version, format, parentpid, permatag, title, username, password) #Get the parameters array
 
     if not re.search(".*/", website):
         website += "/"