Storm

Merge lp:~therve/storm/binary-and into lp:storm

  1. binary-and
  2. Merge into trunk
Proposed by Thomas Herve
Status: Work in progress
Proposed branch: lp:~therve/storm/binary-and
Merge into: lp:storm
Diff against target: 305 lines (has conflicts) 
Text conflict in storm/databases/mysql.py
Text conflict in storm/databases/postgres.py
Text conflict in tests/databases/base.py
To merge this branch: bzr merge lp:~therve/storm/binary-and
Reviewer Review Type Date Requested Status
James Henstridge Approve
Gustavo Niemeyer Approve
Review via email: mp+387@code.launchpad.net
To post a comment you must log in.
lp:~therve/storm/binary-and updated
222. By Thomas Herve

 * Fix precedence management. It should now do the right thing under Postgres
 * Rename the XOR operator to BitwiseXOr

223. By Thomas Herve

Make explicit compilation of XOR for postgres backend.

224. By Thomas Herve

Re-add precedence in compile_python.

Revision history for this message
Gustavo Niemeyer (niemeyer) wrote :

Looks great! +1!

review: Approve
Reply
Revision history for this message
James Henstridge (jamesh) wrote :

Looks good, but the compile_python precedences look wrong still. You have:

+compile_python.set_precedence(40, BitwiseNot, BitwiseOr, BitwiseXOr, BitwiseAnd)

While according to http://docs.python.org/ref/summary.html, the 4 operators have different precedences.

According to http://www.postgresql.org/docs/8.3/static/sql-syntax-lexical.html#SQL-PRECEDENCE it looks like the defaults you've got for compile are okay (it puts shifts at the same precendence, but that doesn't matter too much).

review: Approve (tweak)
Reply
Revision history for this message
James Henstridge (jamesh) wrote :

As an example of the python precedence problem, consider the following:

    >>> (1 | 6) & 4
    4

    >>> from storm.expr import compile_python, BitwiseAnd, BitwiseOr
    >>> expr = BitwiseAnd(BitwiseOr(1, 6), 4)
    >>> compile_python(expr)
    '1|6&3'
    >>> eval(compile_python(expr))
    5

More URLs for operator precedence:

    http://www.sqlite.org/lang_expr.html
    http://dev.mysql.com/doc/refman/5.0/en/operator-precedence.html

Reply

Unmerged revisions

224. By Thomas Herve

Re-add precedence in compile_python.

223. By Thomas Herve

Make explicit compilation of XOR for postgres backend.

222. By Thomas Herve

 * Fix precedence management. It should now do the right thing under Postgres
 * Rename the XOR operator to BitwiseXOr

221. By Thomas Herve <therve@twisted>

Add more tests for operator precedence.

220. By Thomas Herve <therve@twisted>

Merge forward

219. By Thomas Herve <therve@twisted>

Use clean way to emulate XOR on sqlite, and fix BitwiseNot precedence for
sqlite < 3.5.3

218. By Thomas Herve <therve@twisted>

Fix precedence of bitwise operators

217. By Thomas Herve <therve@twisted>

s/BitWise/Bitwise/g

216. By Thomas Herve <therve@twisted>

Add XOR support for postgres and sqlite.

215. By Thomas Herve <therve@twisted>

Rename to bitwise, add other operators for standard cases.

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
=== modified file 'storm/databases/mysql.py'
--- storm/databases/mysql.py 2008-08-16 07:47:33 +0000
+++ storm/databases/mysql.py 2009-08-13 20:21:01 +0000
@@ -30,9 +30,15 @@
30except ImportError:30except ImportError:
31 MySQLdb = dummy31 MySQLdb = dummy
3232
33<<<<<<< TREE
33from storm.expr import (34from storm.expr import (
34 compile, Insert, Select, compile_select, Undef, And, Eq,35 compile, Insert, Select, compile_select, Undef, And, Eq,
35 SQLRaw, SQLToken, is_safe_token)36 SQLRaw, SQLToken, is_safe_token)
37=======
38from storm.expr import (compile, Select, compile_select, Undef, And, Eq,
39 SQLRaw, SQLToken, is_safe_token, BitwiseOr, BitwiseAnd,
40 BitwiseXOr)
41>>>>>>> MERGE-SOURCE
36from storm.variables import Variable42from storm.variables import Variable
37from storm.database import Database, Connection, Result43from storm.database import Database, Connection, Result
38from storm.exceptions import (44from storm.exceptions import (
@@ -58,6 +64,10 @@
58 return expr64 return expr
59 return '`%s`' % expr.replace('`', '``')65 return '`%s`' % expr.replace('`', '``')
6066
67compile.set_precedence(51, BitwiseOr)
68compile.set_precedence(52, BitwiseXOr)
69compile.set_precedence(53, BitwiseAnd)
70
6171
62class MySQLResult(Result):72class MySQLResult(Result):
6373
6474
=== modified file 'storm/databases/postgres.py'
--- storm/databases/postgres.py 2009-07-07 02:06:59 +0000
+++ storm/databases/postgres.py 2009-08-13 20:21:01 +0000
@@ -42,8 +42,13 @@
42 Undef, Expr, SetExpr, Select, Insert, Alias, And, Eq, FuncExpr, SQLRaw,42 Undef, Expr, SetExpr, Select, Insert, Alias, And, Eq, FuncExpr, SQLRaw,
43 Sequence, Like, SQLToken, COLUMN, COLUMN_NAME, COLUMN_PREFIX, TABLE,43 Sequence, Like, SQLToken, COLUMN, COLUMN_NAME, COLUMN_PREFIX, TABLE,
44 compile, compile_select, compile_insert, compile_set_expr, compile_like,44 compile, compile_select, compile_insert, compile_set_expr, compile_like,
45<<<<<<< TREE
45 compile_sql_token)46 compile_sql_token)
46from storm.variables import Variable, ListVariable47from storm.variables import Variable, ListVariable
48=======
49 compile_sql_token, BitwiseXOr)
50from storm.variables import Variable, ListVariable, RawStrVariable
51>>>>>>> MERGE-SOURCE
47from storm.database import Database, Connection, Result52from storm.database import Database, Connection, Result
48from storm.exceptions import (53from storm.exceptions import (
49 install_exceptions, DatabaseError, DatabaseModuleError, InterfaceError,54 install_exceptions, DatabaseError, DatabaseModuleError, InterfaceError,
@@ -202,6 +207,10 @@
202 for subexpr in expr.split("."))207 for subexpr in expr.split("."))
203 return compile_sql_token(compile, expr, state)208 return compile_sql_token(compile, expr, state)
204209
210@compile.when(BitwiseXOr)
211def compile_bitwise_xor_postgres(compile, expr, state):
212 return "%s%s%s" % (compile(expr.expr1, state), "#",
213 compile(expr.expr2, state))
205214
206class PostgresResult(Result):215class PostgresResult(Result):
207216
208217
=== modified file 'storm/databases/sqlite.py'
--- storm/databases/sqlite.py 2008-06-27 07:43:45 +0000
+++ storm/databases/sqlite.py 2009-08-13 20:21:01 +0000
@@ -37,7 +37,8 @@
37from storm.exceptions import install_exceptions, DatabaseModuleError37from storm.exceptions import install_exceptions, DatabaseModuleError
38from storm.expr import (38from storm.expr import (
39 Insert, Select, SELECT, Undef, SQLRaw, Union, Except, Intersect,39 Insert, Select, SELECT, Undef, SQLRaw, Union, Except, Intersect,
40 compile, compile_insert, compile_select)40 compile, compile_insert, compile_select, BitwiseXOr, BitwiseNot,
41 BitwiseAnd, BitwiseOr, compile_binary_oper)
4142
4243
43install_exceptions(sqlite)44install_exceptions(sqlite)
@@ -70,6 +71,19 @@
70 return compile_insert(compile, insert, state)71 return compile_insert(compile, insert, state)
7172
7273
74@compile.when(BitwiseXOr)
75def compile_bitwise_xor_sqlite(compile, expr, state):
76 # XOR doesn't exist in SQLite, so we have to simulate it
77 return compile(BitwiseAnd(BitwiseNot(BitwiseAnd(expr.expr1, expr.expr2)),
78 BitwiseOr(expr.expr1, expr.expr2)),
79 state)
80
81if sqlite is not dummy and sqlite.sqlite_version_info < (3, 5, 3):
82 # ~ precedence was fixed in 3.5.3, but before we have to force it to have
83 # a lowest precedence than other bitwise operations
84 compile.set_precedence(55, BitwiseNot)
85
86
73class SQLiteResult(Result):87class SQLiteResult(Result):
7488
75 def get_insert_identity(self, primary_key, primary_variables):89 def get_insert_identity(self, primary_key, primary_variables):
7690
=== modified file 'storm/expr.py'
--- storm/expr.py 2009-07-31 01:53:08 +0000
+++ storm/expr.py 2009-08-13 20:21:01 +0000
@@ -1012,6 +1012,14 @@
1012 __slots__ = ()1012 __slots__ = ()
1013 oper = "<<"1013 oper = "<<"
10141014
1015class BitwiseAnd(NonAssocBinaryOper):
1016 oper = "&"
1017
1018class BitwiseOr(NonAssocBinaryOper):
1019 oper = "|"
1020
1021class BitwiseXOr(NonAssocBinaryOper):
1022 oper = "^"
10151023
1016class Like(BinaryOper):1024class Like(BinaryOper):
1017 __slots__ = ("escape", "case_sensitive")1025 __slots__ = ("escape", "case_sensitive")
@@ -1300,6 +1308,10 @@
1300 __slots__ = ()1308 __slots__ = ()
1301 suffix = "DESC"1309 suffix = "DESC"
13021310
1311class BitwiseNot(PrefixExpr):
1312 prefix = "~"
1313
1314
13031315
1304# --------------------------------------------------------------------1316# --------------------------------------------------------------------
1305# Plain SQL expressions.1317# Plain SQL expressions.
@@ -1434,16 +1446,18 @@
1434compile.set_precedence(30, Or)1446compile.set_precedence(30, Or)
1435compile.set_precedence(40, And)1447compile.set_precedence(40, And)
1436compile.set_precedence(50, Eq, Ne, Gt, Ge, Lt, Le, Like, In)1448compile.set_precedence(50, Eq, Ne, Gt, Ge, Lt, Le, Like, In)
1437compile.set_precedence(60, LShift, RShift)1449compile.set_precedence(60, BitwiseNot, BitwiseOr, BitwiseXOr, BitwiseAnd)
1438compile.set_precedence(70, Add, Sub)1450compile.set_precedence(70, LShift, RShift)
1439compile.set_precedence(80, Mul, Div, Mod)1451compile.set_precedence(80, Add, Sub)
1452compile.set_precedence(90, Mul, Div, Mod)
14401453
1441compile_python.set_precedence(10, Or)1454compile_python.set_precedence(10, Or)
1442compile_python.set_precedence(20, And)1455compile_python.set_precedence(20, And)
1443compile_python.set_precedence(30, Eq, Ne, Gt, Ge, Lt, Le, Like, In)1456compile_python.set_precedence(30, Eq, Ne, Gt, Ge, Lt, Le, Like, In)
1444compile_python.set_precedence(40, LShift, RShift)1457compile_python.set_precedence(40, BitwiseNot, BitwiseOr, BitwiseXOr, BitwiseAnd)
1445compile_python.set_precedence(50, Add, Sub)1458compile_python.set_precedence(50, LShift, RShift)
1446compile_python.set_precedence(60, Mul, Div, Mod)1459compile_python.set_precedence(60, Add, Sub)
1460compile_python.set_precedence(70, Mul, Div, Mod)
14471461
14481462
1449# --------------------------------------------------------------------1463# --------------------------------------------------------------------
14501464
=== modified file 'tests/databases/base.py'
--- tests/databases/base.py 2009-07-30 06:19:27 +0000
+++ tests/databases/base.py 2009-08-13 20:21:01 +0000
@@ -26,7 +26,12 @@
26import os26import os
2727
28from storm.uri import URI28from storm.uri import URI
29<<<<<<< TREE
29from storm.expr import Select, Column, SQLToken, SQLRaw, Count, Alias30from storm.expr import Select, Column, SQLToken, SQLRaw, Count, Alias
31=======
32from storm.expr import (Select, Column, SQLToken, SQLRaw, Count, Alias,
33 BitwiseXOr, BitwiseOr, BitwiseAnd, BitwiseNot, Add)
34>>>>>>> MERGE-SOURCE
30from storm.variables import (Variable, PickleVariable, RawStrVariable,35from storm.variables import (Variable, PickleVariable, RawStrVariable,
31 DecimalVariable, DateTimeVariable, DateVariable,36 DecimalVariable, DateTimeVariable, DateVariable,
32 TimeVariable, TimeDeltaVariable)37 TimeVariable, TimeDeltaVariable)
@@ -378,6 +383,7 @@
378 result.from_database = self.from_database383 result.from_database = self.from_database
379 self.assertEquals(iter(result).next(), (2, 3))384 self.assertEquals(iter(result).next(), (2, 3))
380385
386<<<<<<< TREE
381 def test_rowcount_insert(self):387 def test_rowcount_insert(self):
382 # All supported backends support rowcount, so far.388 # All supported backends support rowcount, so far.
383 result = self.connection.execute(389 result = self.connection.execute(
@@ -395,6 +401,55 @@
395 "UPDATE test SET title='whatever'")401 "UPDATE test SET title='whatever'")
396 self.assertEquals(result.rowcount, 2)402 self.assertEquals(result.rowcount, 2)
397403
404=======
405 def test_bitwise_xor_support(self):
406 expr = Select(BitwiseXOr(Variable(1), Variable(2)))
407 result = self.connection.execute(expr)
408 self.assertEquals(result.get_all(), [(3,)])
409
410 def test_bitwise_precedence(self):
411 expr = Select(BitwiseOr(BitwiseAnd(Variable(1), Variable(3)),
412 BitwiseAnd(Variable(1), Variable(2))))
413 result = self.connection.execute(expr)
414 self.assertEquals(result.get_all(), [(1,)])
415
416 expr = Select(BitwiseAnd(BitwiseOr(Variable(2), Variable(3)),
417 BitwiseOr(Variable(1), Variable(3))))
418 result = self.connection.execute(expr)
419 self.assertEquals(result.get_all(), [(3,)])
420
421 expr = Select(BitwiseAnd(BitwiseOr(BitwiseNot(Variable(2)),
422 Variable(3)),
423 Variable(1)))
424 result = self.connection.execute(expr)
425 self.assertEquals(result.get_all(), [(1,)])
426
427 def test_bitwise_precedence_python(self):
428 expr = Select(Add(BitwiseNot(2), 3))
429 result = self.connection.execute(expr)
430 self.assertEquals(result.get_all(), [(0,)])
431
432 expr = Select(BitwiseOr(BitwiseAnd(1, 3),
433 BitwiseAnd(1, 2)))
434 result = self.connection.execute(expr)
435 self.assertEquals(result.get_all(), [(1,)])
436
437 expr = Select(BitwiseAnd(BitwiseOr(2, 3),
438 BitwiseOr(1, 3)))
439 result = self.connection.execute(expr)
440 self.assertEquals(result.get_all(), [(3,)])
441
442 expr = Select(BitwiseAnd(BitwiseOr(BitwiseNot(2),
443 3),
444 1))
445 result = self.connection.execute(expr)
446 self.assertEquals(result.get_all(), [(1,)])
447
448 expr = Select(Add(BitwiseNot(2), 3))
449 result = self.connection.execute(expr)
450 self.assertEquals(result.get_all(), [(0,)])
451
452>>>>>>> MERGE-SOURCE
398453
399class UnsupportedDatabaseTest(object):454class UnsupportedDatabaseTest(object):
400 455
401456
=== modified file 'tests/databases/postgres.py'
=== modified file 'tests/databases/sqlite.py'
--- tests/databases/sqlite.py 2008-06-27 07:49:40 +0000
+++ tests/databases/sqlite.py 2009-08-13 20:21:01 +0000
@@ -18,9 +18,7 @@
18# You should have received a copy of the GNU Lesser General Public License18# You should have received a copy of the GNU Lesser General Public License
19# along with this program. If not, see <http://www.gnu.org/licenses/>.19# along with this program. If not, see <http://www.gnu.org/licenses/>.
20#20#
21from datetime import timedelta
22import time21import time
23import os
2422
25from storm.exceptions import OperationalError23from storm.exceptions import OperationalError
26from storm.databases.sqlite import SQLite24from storm.databases.sqlite import SQLite
2725
=== modified file 'tests/expr.py'
--- tests/expr.py 2009-07-23 22:47:10 +0000
+++ tests/expr.py 2009-08-13 20:21:01 +0000
@@ -1052,6 +1052,34 @@
1052 self.assertEquals(statement, "func1()>>?")1052 self.assertEquals(statement, "func1()>>?")
1053 self.assertVariablesEqual(state.parameters, [Variable("value")])1053 self.assertVariablesEqual(state.parameters, [Variable("value")])
10541054
1055 def test_bitwise_and(self):
1056 expr = BitwiseAnd(Func1(), Func2())
1057 state = State()
1058 statement = compile(expr, state)
1059 self.assertEquals(statement, "func1()&func2()")
1060 self.assertEquals(state.parameters, [])
1061
1062 def test_bitwise_or(self):
1063 expr = BitwiseOr(Func1(), Func2())
1064 state = State()
1065 statement = compile(expr, state)
1066 self.assertEquals(statement, "func1()|func2()")
1067 self.assertEquals(state.parameters, [])
1068
1069 def test_bitwise_xor(self):
1070 expr = BitwiseXOr(Func1(), Func2())
1071 state = State()
1072 statement = compile(expr, state)
1073 self.assertEquals(statement, "func1()^func2()")
1074 self.assertEquals(state.parameters, [])
1075
1076 def test_bitwise_not(self):
1077 expr = BitwiseNot(Func1())
1078 state = State()
1079 statement = compile(expr, state)
1080 self.assertEquals(statement, "~ func1()")
1081 self.assertEquals(state.parameters, [])
1082
1055 def test_like(self):1083 def test_like(self):
1056 expr = Like(Func1(), "value")1084 expr = Like(Func1(), "value")
1057 state = State()1085 state = State()
@@ -2112,6 +2140,21 @@
2112 self.assertEquals(py_expr, "_0>>_1")2140 self.assertEquals(py_expr, "_0>>_1")
2113 self.assertEquals(state.parameters, [1, 2])2141 self.assertEquals(state.parameters, [1, 2])
21142142
2143 def test_bitwise_and(self):
2144 expr = BitwiseAnd(Variable(1), Variable(2))
2145 py_expr = compile_python(expr)
2146 self.assertEquals(py_expr, "1&2")
2147
2148 def test_bitwise_or(self):
2149 expr = BitwiseOr(Variable(1), Variable(2))
2150 py_expr = compile_python(expr)
2151 self.assertEquals(py_expr, "1|2")
2152
2153 def test_bitwise_xor(self):
2154 expr = BitwiseXOr(Variable(1), Variable(2))
2155 py_expr = compile_python(expr)
2156 self.assertEquals(py_expr, "1^2")
2157
2115 def test_in(self):2158 def test_in(self):
2116 expr = In(Variable(1), Variable(2))2159 expr = In(Variable(1), Variable(2))
2117 state = State()2160 state = State()

Subscribers

People subscribed via source and target branches

to status/vote changes: