Merge lp:~marco-gallotta/ibid/contest into lp:~ibid-core/ibid/old-trunk-1.6

[marcog]

[marcog (marco-gallotta)]

First iteration of the contest plugin, with USACO scriping pretty much done.

[Stefano Rivera (stefanor)]

Run pyflakes. Otherwise you are simply asking me to do it for you:

ibid/plugins/contest.py:1: 'codecs' imported but unused
ibid/plugins/contest.py:4: 'ElementTree' imported but unused
ibid/plugins/contest.py:8: 'ibid' imported but unused
ibid/plugins/contest.py:10: 'eagerload' imported but unused
ibid/plugins/contest.py:70: local variable 'etree' is assigned to but never used

The name "contest" is too board. It should be called programming_contest, if anything.
Also, it's really esoteric, so it should have autoload = False on all the Processors.

> priority = -20
why? Put a comment in explaining please.

> urlencode({u'NAME': user, u'PASSWORD': password})
URLencode doesn't encode to utf-8, so utf-8 encode the parameters yourself, and give it bytestrings.

Applies to all your urlencodes()

> if font.text and font.text.find('Please try again') != -1:
More pythonic:
if font.text and u'Please try again' in font.text:

This crops up again in: division = [b.text for b in etree.getiterator(u'b') if b.text and b.text.find(usaco_user) != -1][0]

> class UsacoException(Exception):
> pass

I believe that if you are creating your own exception to carry an exception message it should do so and not rely on Exception, as Python 3's Exception doesn't have a message. We don't support Python 3 yet, but it's good practice.
> .filter(and_(
> Identity.identity == user,
> Identity.source == event.source)) \

You can save a line by doing:
 .filter(Identity.identity == user) \
 .filter(Identity.source == event.source) \

> usaco_account = [attr.value for attr in account.attributes if attr.name == 'usaco_account']

If you are going to do that, you do want to eager load attributes.

> __redacted__
I'd prefer [redacted]. It looks nicer :)

> def setup(self):
> pass

Be aware that if lp:~stefanor/ibid/ibid-plugin-507489 gets in first, it'll clash.

review: Needs Fixing

[lp:~marco-gallotta/ibid/contest]

851. By marcog on 2010-01-17

Remove unused imports

852. By marcog on 2010-01-17

Handle usaco user not found

853. By marcog on 2010-01-17

Collection of minor changes: autoload=False, utf-8 encode params and other less important changes

854. By marcog on 2010-01-17

Make UsacoException have a msg so that it works in Python3

855. By marcog on 2010-01-17

Eagerload attributes and shave a line with filters

856. By marcog on 2010-01-17

__redacted__ -> [redacted], to make stefano happy :)

[marcog (marco-gallotta)]

> The name "contest" is too board. It should be called programming_contest, if
> anything.

We still need to agree on a name. I agree that contest is too general. I think programming_contest is too long (20 characters and the longest current name is 12); if we do go with this, I think we should drop the _ to follow the style of other plugins (gameservers, buildbot). Vhata suggested progcomp, but as he said that's ugly. There's also the possibility that maths contests are added: what then?

> Run pyflakes. Otherwise you are simply asking me to do it for you:
>
> ibid/plugins/contest.py:1: 'codecs' imported but unused
> ibid/plugins/contest.py:4: 'ElementTree' imported but unused
> ibid/plugins/contest.py:8: 'ibid' imported but unused
> ibid/plugins/contest.py:10: 'eagerload' imported but unused
> ibid/plugins/contest.py:70: local variable 'etree' is assigned to but never
> used
>
> Also, it's really esoteric, so it should have autoload = False on all the
> Processors.
>
> > priority = -20
> why? Put a comment in explaining please.
>
> > urlencode({u'NAME': user, u'PASSWORD': password})
> URLencode doesn't encode to utf-8, so utf-8 encode the parameters yourself,
> and give it bytestrings.
>
> Applies to all your urlencodes()
>
> > if font.text and font.text.find('Please try again') != -1:
> More pythonic:
> if font.text and u'Please try again' in font.text:
>
> This crops up again in: division = [b.text for b in etree.getiterator(u'b') if
> b.text and b.text.find(usaco_user) != -1][0]
>
> > class UsacoException(Exception):
> > pass
>
> I believe that if you are creating your own exception to carry an exception
> message it should do so and not rely on Exception, as Python 3's Exception
> doesn't have a message. We don't support Python 3 yet, but it's good practice.
> > .filter(and_(
> > Identity.identity == user,
> > Identity.source == event.source)) \
>
> You can save a line by doing:
> .filter(Identity.identity == user) \
> .filter(Identity.source == event.source) \
>
> > usaco_account = [attr.value for attr in account.attributes if attr.name ==
> 'usaco_account']
>
> If you are going to do that, you do want to eager load attributes.
>
> > __redacted__
> I'd prefer [redacted]. It looks nicer :)

Done

> > def setup(self):
> > pass
>
> Be aware that if lp:~stefanor/ibid/ibid-plugin-507489 gets in first, it'll
> clash.

Hopefully someone notices :)

[lp:~marco-gallotta/ibid/contest]

857. By marcog on 2010-01-17

Fix some unicode stuff

858. By marcog on 2010-01-17

Saving another and_

859. By marcog on 2010-01-17

Fix some minor bugs

860. By marcog on 2010-01-17

Put autoload=False back

[Stefano Rivera (stefanor)]

ibid/plugins/contest.py:7: 'and_' imported but unused

And the name needs to be sorted. But otherwise I approve

review: Approve

[lp:~marco-gallotta/ibid/contest]

861. By marcog on 2010-01-17

Handle updates to usaco account

[Stefano Rivera (stefanor)]

Can you add a copyright header to the top (see any other file in current trunk)

review: Needs Fixing

[marcog (marco-gallotta)]

> Can you add a copyright header to the top (see any other file in current
> trunk)

Done

[lp:~marco-gallotta/ibid/contest]

862. By marcog on 2010-01-19

Copyright header

863. By marcog on 2010-01-21

Search for usaco results also by name and usaco account when not linked

864. By marcog on 2010-01-21

Add timeout for fetching USACO results and suggest that the site might be down

865. By marcog on 2010-01-21

Forbid giving passwords in public

866. By marcog on 2010-01-21

Allow admins to link usaco accounts without a usaco password

867. By marcog on 2010-01-21

Tell the user when usaco site appears dead

868. By marcog on 2010-01-21

Change from looking up Identities to Accounts table to find users not on the current source

869. By marcog on 2010-01-21

Improve the 'you need an account' msg

870. By marcog on 2010-01-21

Put division inline if only one person is listed in usaco results

871. By marcog on 2010-01-25

Correct the test for 'not a teacher account'

872. By marcog on 2010-01-25

Rename contest->codecontest

873. By marcog on 2010-01-25

Small tidy-up

Unmerged revisions

=== added file 'ibid/plugins/contest.py'
--- ibid/plugins/contest.py	1970-01-01 00:00:00 +0000
+++ ibid/plugins/contest.py	2010-01-19 13:45:24 +0000
@@ -0,0 +1,259 @@
+# Copyright (c) 2010, Marco Gallotta
+# Released under terms of the MIT/X/Expat Licence. See COPYING for details.
+
+import re
+from urllib2 import HTTPError
+
+from urllib import urlencode
+
+from ibid.config import Option
+from ibid.db import eagerload
+from ibid.db.models import Account, Attribute, Identity
+from ibid.plugins import Processor, match
+from ibid.utils import cacheable_download
+from ibid.utils.html import get_html_parse_tree
+
+help = {u'usaco': u'Query USACO sections, divisions and more. Since this info is private, users are required to provide their USACO password when linking their USACO account to their ibid account and only linked accounts can be queried. Your password is used only to confirm that the account is yours and is discarded immediately.'}
+
+class UsacoException(Exception):
+    def __init__(self, msg):
+        self.msg = msg
+
+    def __unicode__(self):
+        return unicode(self.msg)
+
+class Usaco(Processor):
+    """usaco <section|division> for <user>
+    usaco <contest> results [for <user>]
+    i am <usaco_username> on usaco password <usaco_password>"""
+
+    admin_user = Option('admin_user', 'Admin user on USACO', None)
+    admin_password = Option('admin_password', 'Admin password on USACO', None)
+
+    feature = 'usaco'
+    # Clashes with identity, so lower our priority since if we match, then
+    # this is the better match
+    priority = -20
+    autoload = False
+
+    def _login(self, user, password):
+        params = urlencode({'NAME': user.encode('utf-8'), 'PASSWORD': password.encode('utf-8')})
+        etree = get_html_parse_tree(u'http://ace.delos.com/usacogate', data=params, treetype=u'etree')
+        for font in etree.getiterator(u'font'):
+            if font.text and u'Please try again' in font.text:
+                return None
+        return etree
+
+    def _check_login(self, user, password):
+        return self._login(user, password) is not None
+
+    def _get_section(self, monitor_url, usaco_user, user):
+        etree = get_html_parse_tree(monitor_url, treetype=u'etree')
+        usaco_user = usaco_user.lower()
+        header = True
+        for tr in etree.getiterator(u'tr'):
+            if header:
+                header = False
+                continue
+            tds = [t.text for t in tr.getiterator(u'td')]
+            section = u'is on section %s' % tds[5]
+            if tds[5] == u'DONE':
+                section = u'has completed USACO training'
+            if tds[0] and tds[0].lower() == usaco_user:
+                return u'%(user)s (%(usaco_user)s on USACO) %(section)s and last logged in %(days)s ago' % {
+                    'user': user,
+                    'usaco_user': usaco_user,
+                    'days': tds[3],
+                    'section': section,
+                }
+
+        return None
+
+    def _add_user(self, monitor_url, user):
+        matches = re.search(r'a=(.+)&', monitor_url)
+        auth = matches.group(1)
+        params = urlencode({'STUDENTID': user.encode('utf-8'), 'ADD': 'ADD STUDENT',
+            'a': auth.encode('utf-8'), 'monitor': '1'})
+        etree = get_html_parse_tree(monitor_url, treetype=u'etree', data=params)
+        for font in etree.getiterator(u'font'):
+            if font.text and u'No STATUS file for' in font.text:
+                raise UsacoException(u'Sorry, user %s not found' % user)
+
+    def _get_monitor_url(self):
+        if self.admin_user is None or self.admin_password is None:
+            raise UsacoException(u'Sorry, you need to configure a USACO admin account')
+            return
+        etree = self._login(self.admin_user, self.admin_password)
+        if etree is None:
+            raise UsacoException(u'Sorry, the configured USACO admin account is invalid')
+
+        urls = [a.get(u'href') for a in etree.getiterator(u'a')]
+        monitor_url = [url for url in urls if u'monitor' in url][0]
+        if len(monitor_url) == 0:
+            raise UsacoException(u'USACO admin account does not have teacher status')
+
+        return monitor_url
+
+    def _get_usaco_user(self, event, user):
+        account = event.session.query(Account) \
+            .options(eagerload('attributes')) \
+            .filter(Account.username == user) \
+            .first()
+        if account is None:
+            account = event.session.query(Account) \
+                .options(eagerload('attributes')) \
+                .join('identities') \
+                .filter(Identity.identity == user) \
+                .filter(Identity.source == event.source) \
+                .first()
+            if account is None:
+                raise UsacoException(u'Sorry, %s has not been linked to a USACO account yet' % user)
+
+        usaco_account = [attr.value for attr in account.attributes if attr.name == 'usaco_account']
+        if len(usaco_account) == 0:
+            raise UsacoException(u'Sorry, %s has not been linked to a USACO account yet' % user)
+        return usaco_account[0]
+
+    def _get_usaco_users(self, event):
+        accounts = event.session.query(Identity) \
+            .join(['account', 'attributes']) \
+            .add_entity(Attribute) \
+            .filter(Attribute.name == u'usaco_account') \
+            .filter(Identity.source == event.source) \
+            .all()
+
+        users = {}
+        for a in accounts:
+            users[a[1].value] = a[0].identity
+        return users
+
+    @match(r'^usaco\s+section\s+(?:for\s+)?(.+)$')
+    def get_section(self, event, user):
+        try:
+            usaco_user = self._get_usaco_user(event, user)
+            monitor_url = self._get_monitor_url()
+        except UsacoException, e:
+            event.addresponse(e)
+            return
+
+        section = self._get_section(monitor_url, usaco_user, user)
+        if section:
+            event.addresponse(section)
+            return
+
+        try:
+            self._add_user(monitor_url, user)
+        except UsacoException, e:
+            event.addresponse(e)
+            return
+
+        event.addresponse(self._get_section(monitor_url, usaco_user, user))
+
+    @match(r'^usaco\s+division\s+(?:for\s+)?(.+)$')
+    def get_division(self, event, user):
+        try:
+            usaco_user = self._get_usaco_user(event, user)
+        except UsacoException, e:
+            event.addresponse(e)
+            return
+
+        params = urlencode({'id': usaco_user.encode('utf-8'), 'search': 'SEARCH'})
+        etree = get_html_parse_tree(u'http://ace.delos.com/showdiv', data=params, treetype=u'etree')
+        division = [b.text for b in etree.getiterator(u'b') if b.text and usaco_user in b.text][0]
+        if division.find(u'would compete') != -1:
+            event.addresponse(u'%(user)s (%(usaco_user)s on USACO) has not competed in a USACO before',
+                    {u'user': user, u'usaco_user': usaco_user})
+        matches = re.search(r'(\w+) Division', division)
+        division = matches.group(1).lower()
+        event.addresponse(u'%(user)s (%(usaco_user)s on USACO) is in the %(division)s division',
+                {u'user': user, u'usaco_user': usaco_user, u'division': division})
+
+    def _redact(self, event, term):
+        for type in ['raw', 'deaddressed', 'clean', 'stripped']:
+            event['message'][type] = re.sub(r'(.*)(%s)' % re.escape(term), r'\1[redacted]', event['message'][type])
+
+    @match(r'^i\s+am\s+(\S+)\s+on\s+usaco\s+password\s+(\S+)$')
+    def usaco_account(self, event, user, password):
+        self._redact(event, password)
+
+        if not self._check_login(user, password):
+            event.addresponse(u'Sorry, that account is invalid')
+            return
+        if not event.account:
+            event.addresponse(u'Sorry, you need to create an account first')
+            return
+
+        try:
+            monitor_url = self._get_monitor_url()
+        except UsacoException, e:
+            event.addresponse(e)
+            return
+
+        self._add_user(monitor_url, user)
+
+        account = event.session.query(Account).get(event.account)
+        usaco_account = [attr for attr in account.attributes if attr.name == u'usaco_account']
+        if usaco_account:
+            usaco_account[0].value = user
+        else:
+            account.attributes.append(Attribute('usaco_account', user))
+        event.session.save_or_update(account)
+        event.session.commit()
+
+        event.addresponse(u'Done')
+
+    @match(r'^usaco\s+(\S+)\s+results(?:\s+for\s+(\S+))?$')
+    def usaco_results(self, event, contest, user):
+        if user is not None:
+            try:
+                usaco_user = self._get_usaco_user(event, user)
+            except UsacoException, e:
+                event.addresponse(e)
+                return
+
+        url = u'http://ace.delos.com/%sresults' % contest.upper()
+        try:
+            filename = cacheable_download(url, u'usaco/results_%s' % contest.upper())
+        except HTTPError:
+            event.addresponse(u"Sorry, the results for %s aren't released yet", contest)
+
+        if user is not None:
+            users = {usaco_user: user}
+        else:
+            users = self._get_usaco_users(event)
+
+        text = open(filename, 'r').read().decode('ISO-8859-2')
+        divisions = [u'gold', u'silver', u'bronze']
+        results = [[], [], []]
+        division = None
+        for line in text.splitlines():
+            for index, d in enumerate(divisions):
+                if d in line.lower():
+                    division = index
+            # Example results line:
+            #            2010 POL Jakub Pachocki      meret1      ***** ***** 270 ***** ***** * 396 ***** ***** ** 324 1000
+            matches = re.match(r'^\s*(\d{4})\s+([A-Z]{3})\s+(.+?)\s+(\S+\d)\s+([\*xts\.e0-9 ]+?)\s+(\d+)\s*$', line)
+            if matches:
+                year = matches.group(1)
+                country = matches.group(2)
+                name = matches.group(3)
+                usaco_user = matches.group(4)
+                scores = matches.group(5)
+                total = matches.group(6)
+                if usaco_user in users.keys():
+                    results[division].append((year, country, name, usaco_user, scores, total))
+
+        response = []
+        for i, division in enumerate(divisions):
+            if results[i]:
+                response.append(u'%s division results:' % division.title())
+            for result in results[i]:
+                response.append(u'%(user)s (%(usaco_user)s on USACO) scored %(total)s (%(scores)s)' % {
+                    u'user': users[result[3]],
+                    u'usaco_user': result[3],
+                    u'total': result[5],
+                    u'scores': result[4],
+                })
+        event.addresponse(u'\n'.join(response), conflate=False)
+
+# vi: set et sta sw=4 ts=4:
 
=== modified file 'scripts/ibid-plugin'
--- scripts/ibid-plugin	2010-01-18 22:13:23 +0000
+++ scripts/ibid-plugin	2010-01-19 13:45:25 +0000
@@ -65,6 +65,9 @@
     permissions = []
     supports = ('action', 'multiline', 'notice')
 
+    def setup(self):
+        pass
+
     def logging_name(self, name):
         return name