1
=== added file 'applications/examples/views/default/cron.html'
2
--- applications/examples/views/default/cron.html	1970-01-01 00:00:00 +0000
3
+++ applications/examples/views/default/cron.html	2009-03-05 12:32:00 +0000
4
@@ -0,0 +1,87 @@
5
1
{{extend 'layout.html'}}
6
2
{{import os}}
7
3
8
4
<h1>web2py<sup style="font-size:0.5em;">TM</sup> cron</h1>
9
5
10
6
The web2py cron provides the ability for applications to execute tasks at preset times, in a platform independent manner (tested on Windows XP, Linux and MacOS X, but should work on any python 2.5 supported platform). 
11
7
12
8
Cron functionality is defined by a crontab file (in regular <A href="http://en.wikipedia.org/wiki/Cron#crontab_syntax">crontab syntax</A>) in your application's cron directory. This also means that every application can have a separate cron configuration and that cron config can be changed from within web2py without affecting the host OS itself.
13
9
<BR>
14
10
<BR>
15
11
Example:
16
12
{{=CODE(""" 0-59/1  *       *       *       *       root python /path/to/python/script.py
17
13
*/30    *       *       *       *       root *applications/admin/cron/expire_sessions.py
18
14
-1      *       *       *       *       root *mycontroller/myfunction""")}}
19
15
20
16
As you might have noticed, some of the lines use <A href="#extensions">extensions</A> to regular cron syntax to provide additional web2py functionality.
21
17
22
18
<BR>
23
19
<h2>Modes of use</h2>
24
20
25
21
Depending on how you are invoking web2py, there are four modes of operation for web2py cron.
26
22
27
23
<UL>
28
24
<LI><A href="#soft">Soft cron</A>, available under all execution modes</LI>
29
25
<LI><A href="#hard">Hard cron</A>, available if using the built-in web server (either directly or via Apache mod_proxy)</LI>
30
26
<LI><A href="#external">External cron</A>, available if you have access to the systems own cron service</LI>
31
27
<LI><A href="#no">No cron</A></LI>
32
28
</UL>
33
29
34
30
The default is hard cron if you are using the built-in web server, in all other cases the default is soft-cron.
35
31
36
32
37
33
<h3 id="soft">Soft cron</h3>
38
34
39
35
Soft cron is the default if you are using CGI, FASTCGI or WSGI. Your tasks will be executed in the first call (page load) to web2py after the time specified in crontab (but AFTER processing the page, so no delay to the user is visible). Obviously, there is some uncertainty exactly when the task will be executed depending on the traffic the site receives. Also, the cron task may get interrupted if the web server has a page load timeout set. If these limitations are not acceptable, see <A href="#external">external cron</A>. Soft cron is a reasonable last resort, but if your web server allows other cron methods, they should be preferred over soft cron.
40
36
41
37
<h3 id="hard">Hard cron</h3>
42
38
43
39
Hard cron is the default if you are using the built-in web server (either directly or via Apache mod_proxy). Hard cron is executed in a parallel thread, so unlike soft cron there are no limitations with regard to run time or execution time precision.
44
40
45
41
<h3 id="external">External cron</h3>
46
42
47
43
External cron is not default in any scenarios, but requires you to have access to the system cron facilities. It runs in a parallel process, so none of the limitations of soft cron apply. This is the recommended way of using cron under WSGI or FASTCGI. 
48
44
<BR>
49
45
<BR>
50
46
51
47
Example of line to add to the system crontab, (usually /etc/crontab):
52
48
{{=CODE("""0-59/1 * * * * web2py cd /var/www/web2py/ && python web2py.py -C -D 1 >> /tmp/cron.output 2>&1""")}}
53
49
54
50
If you are running external cron, make sure you add the -N command line parameter to your web2py startup script or config so there is no collision of multiple types of cron.
55
51
56
52
<h3 id="no">No cron</h3>
57
53
58
54
In case you do not need any cron functionality, you can use the -N command line parameter to disable it. Note that this will disable some maintenance tasks (like the automatic cleaning of session dirs).
59
55
60
56
<BR>
61
57
<h2 id="extensions">Cron extensions</h2>
62
58
63
59
Web2py cron has a some extra syntax to support web2py application specifics.
64
60
65
61
<h3>Calling scripts in the web2py environment</h3>
66
62
67
63
If the task/script is prefixed with an asterisk and ends with ".py", it will be executed in the web2py environment. This means you will have all the controllers and models at your disposal. Warning: be careful how you use models. While the execution happens in a separate process, database locks have to be taken into account in order to avoid pages waiting for cron tasks that be blocking the database.
68
64
69
65
<BR>
70
66
<BR>
71
67
Example:
72
68
{{=CODE("""*/30    *       *       *       *       root *applications/admin/cron/expire_sessions.py""")}}
73
69
74
70
<h3>Calling controller functions</h3>
75
71
76
72
Same as above, but a function from a controller is executed instead of a separate script file.
77
73
<BR>
78
74
<BR>
79
75
80
76
Example:
81
77
{{=CODE("""*/30    *       *       *       *       root *mycontroller/myfunction""")}}
82
78
83
79
<h3>Application initialization</h3>
84
80
85
81
If you specify -1 as minutes in the crontab file, the given task will be executed only ONCE, on web2py startup. You can use this feature if you want to precache, check or initialize data for an application on web2py startup.
86
82
87
83
Example:
88
84
{{=CODE("""
89
85
-1      *       *       *       *       root *mycontroller/myfunction
90
86
""")}}
91
87
92
0
88
93
=== removed file 'applications/examples/views/default/cron.html'
94
--- applications/examples/views/default/cron.html	2009-03-01 19:22:48 +0000
95
+++ applications/examples/views/default/cron.html	1970-01-01 00:00:00 +0000
96
@@ -1,87 +0,0 @@
97
1
{{extend 'layout.html'}}
98
2
{{import os}}
99
3
100
4
<h1>web2py<sup style="font-size:0.5em;">TM</sup> cron</h1>
101
5
102
6
The web2py cron provides the ability for applications to execute tasks at preset times, in a platform independent manner (tested on Windows XP, Linux and MacOS X, but should work on any python 2.5 supported platform). 
103
7
104
8
Cron functionality is defined by a crontab file (in regular <A href="http://en.wikipedia.org/wiki/Cron#crontab_syntax">crontab syntax</A>) in your application's cron directory. This also means that every application can have a separate cron configuration and that cron config can be changed from within web2py without affecting the host OS itself.
105
9
<BR>
106
10
<BR>
107
11
Example:
108
12
{{=CODE(""" 0-59/1  *       *       *       *       root python /path/to/python/script.py
109
13
*/30    *       *       *       *       root *applications/admin/cron/expire_sessions.py
110
14
-1      *       *       *       *       root *mycontroller/myfunction""")}}
111
15
112
16
As you might have noticed, some of the lines use <A href="#extensions">extensions</A> to regular cron syntax to provide additional web2py functionality.
113
17
114
18
<BR>
115
19
<h2>Modes of use</h2>
116
20
117
21
Depending on how you are invoking web2py, there are four modes of operation for web2py cron.
118
22
119
23
<UL>
120
24
<LI><A href="#soft">Soft cron</A>, available under all execution modes</LI>
121
25
<LI><A href="#hard">Hard cron</A>, available if using the built-in web server (either directly or via Apache mod_proxy)</LI>
122
26
<LI><A href="#external">External cron</A>, available if you have access to the systems own cron service</LI>
123
27
<LI><A href="#no">No cron</A></LI>
124
28
</UL>
125
29
126
30
The default is hard cron if you are using the built-in web server, in all other cases the default is soft-cron.
127
31
128
32
129
33
<h3 id="soft">Soft cron</h3>
130
34
131
35
Soft cron is the default if you are using CGI, FASTCGI or WSGI. Your tasks will be executed in the first call (page load) to web2py after the time specified in crontab (but AFTER processing the page, so no delay to the user is visible). Obviously, there is some uncertainty exactly when the task will be executed depending on the traffic the site receives. Also, the cron task may get interrupted if the web server has a page load timeout set. If these limitations are not acceptable, see <A href="#external">external cron</A>. Soft cron is a reasonable last resort, but if your web server allows other cron methods, they should be preferred over soft cron.
132
36
133
37
<h3 id="hard">Hard cron</h3>
134
38
135
39
Hard cron is the default if you are using the built-in web server (either directly or via Apache mod_proxy). Hard cron is executed in a parallel thread, so unlike soft cron there are no limitations with regard to run time or execution time precision.
136
40
137
41
<h3 id="external">External cron</h3>
138
42
139
43
External cron is not default in any scenarios, but requires you to have access to the system cron facilities. It runs in a parallel process, so none of the limitations of soft cron apply. This is the recommended way of using cron under WSGI or FASTCGI. 
140
44
<BR>
141
45
<BR>
142
46
143
47
Example of line to add to the system crontab, (usually /etc/crontab):
144
48
{{=CODE("""0-59/1 * * * * web2py cd /var/www/web2py/ && touch applications/admin/cron/cron.master && python web2py.py -C -D 1 >> /tmp/cron.output 2>&1""")}}
145
49
146
50
If you are running external cron, make sure you add the -N command line parameter to your web2py startup script or config so there is no collision of multiple types of cron.
147
51
148
52
<h3 id="no">No cron</h3>
149
53
150
54
In case you do not need any cron functionality, you can use the -N command line parameter to disable it. Note that this will disable some maintenance tasks (like the automatic cleaning of session dirs).
151
55
152
56
<BR>
153
57
<h2 id="extensions">Cron extensions</h2>
154
58
155
59
Web2py cron has a some extra syntax to support web2py application specifics.
156
60
157
61
<h3>Calling scripts in the web2py environment</h3>
158
62
159
63
If the task/script is prefixed with an asterisk and ends with ".py", it will be executed in the web2py environment. This means you will have all the controllers and models at your disposal. Warning: be careful how you use models. While the execution happens in a separate process, database locks have to be taken into account in order to avoid pages waiting for cron tasks that be blocking the database.
160
64
161
65
<BR>
162
66
<BR>
163
67
Example:
164
68
{{=CODE("""*/30    *       *       *       *       root *applications/admin/cron/expire_sessions.py""")}}
165
69
166
70
<h3>Calling controller functions</h3>
167
71
168
72
Same as above, but a function from a controller is executed instead of a separate script file.
169
73
<BR>
170
74
<BR>
171
75
172
76
Example:
173
77
{{=CODE("""*/30    *       *       *       *       root *mycontroller/myfunction""")}}
174
78
175
79
<h3>Application initialization</h3>
176
80
177
81
If you specify -1 as minutes in the crontab file, the given task will be executed only ONCE, on web2py startup. You can use this feature if you want to precache, check or initialize data for an application on web2py startup.
178
82
179
83
Example:
180
84
{{=CODE("""
181
85
-1      *       *       *       *       root *mycontroller/myfunction
182
86
""")}}
183
87
184
88
0
185
=== modified file 'gluon/contrib/cron.py'
186
--- gluon/contrib/cron.py	2009-02-22 15:32:13 +0000
187
+++ gluon/contrib/cron.py	2009-03-05 12:32:00 +0000
188
@@ -31,7 +31,9 @@
189
31
31
190
32
    def run(self):
32
    def run(self):
191
33
        logging.debug('External cron invocation')
33
        logging.debug('External cron invocation')
193
34
        crondance(apppath({'web2py_path': self.basedir}), 'ext')
34
        if tokenmaster(os.path.join(self.basedir, 'applications', 'admin', 'cron')):
194
35
            crondance(apppath({'web2py_path': self.basedir}), 'ext')
195
36
            tokenmaster(os.path.join(self.basedir, 'applications', 'admin', 'cron'), action = 'release')
196
35
37
197
36
38
198
37
class hardcron(threading.Thread):
39
class hardcron(threading.Thread):
199
@@ -41,6 +43,12 @@
200
41
        self.setDaemon(True)
43
        self.setDaemon(True)
201
42
        self.basedir = os.getcwd()
44
        self.basedir = os.getcwd()
202
43
45
203
46
    def launch(self):
204
47
        path = apppath({'web2py_path': self.basedir})
205
48
        if tokenmaster(os.path.join(path, 'admin', 'cron')):
206
49
            crondance(path, 'hard')
207
50
            tokenmaster(os.path.join(path, 'admin', 'cron'), action = 'release')
208
51
        
209
44
    def run(self):
52
    def run(self):
210
45
        global crontype
53
        global crontype
211
46
        crontype = 'Hard'
54
        crontype = 'Hard'
212
@@ -48,8 +56,7 @@
213
48
        logging.info('Hard cron daemon started')
56
        logging.info('Hard cron daemon started')
214
49
        while True:
57
        while True:
215
50
            now = time.time()
58
            now = time.time()
218
51
            s.enter(60 - now % 60, 1, crondance,
59
            s.enter(60 - now % 60, 1, self.launch, ())
217
52
                    (apppath({'web2py_path' : self.basedir}), 'hard'))
219
53
            s.run()
60
            s.run()
220
54
61
221
55
class softcron(threading.Thread):
62
class softcron(threading.Thread):
222
@@ -62,33 +69,63 @@
223
62
69
224
63
    def run(self):
70
    def run(self):
225
64
        path = apppath(self.env)
71
        path = apppath(self.env)
226
65
        marker = os.path.join(path, 'admin/cron/cron.master')  # location of Chronos, Master of All Time !
227
66
        if not os.path.exists(marker):  # cron master missing, try to recreate one
228
67
            logging.warning('WEB2PY CRON: cron.master not found at %s. Trying to recreate.'
229
68
                             % marker)
230
69
            mfile = open(marker, 'wb')  # touch cron marker
231
70
            mfile.close()
232
71
233
72
        now = time.time()
72
        now = time.time()
235
73
        if 60 > now - self.cronmaster:  # our own thread did a cron check less than a minute ago, don't even bother checking the file
73
        if self.cronmaster and 60 > now - self.cronmaster:  # our own thread did a cron check less than a minute ago, don't even bother checking the file
236
74
            logging.debug("Don't bother with cron.master, it's only %s s old"
74
            logging.debug("Don't bother with cron.master, it's only %s s old"
237
75
                           % (now - self.cronmaster))
75
                           % (now - self.cronmaster))
238
76
            return
76
            return
239
77
77
240
78
        try:
241
79
            self.cronmaster = os.stat(marker).st_mtime  # get last_modified timestamp of cron.master file
242
80
        except Exception, e:
243
81
            self.cronmaster = 0
244
82
            logging.warning('cron.master trouble: %s' % e)
245
83
246
84
        logging.debug('Cronmaster stamp: %s, Now: %s'
78
        logging.debug('Cronmaster stamp: %s, Now: %s'
247
85
                       % (self.cronmaster, now))
79
                       % (self.cronmaster, now))
248
86
        if 60 <= now - self.cronmaster:  # new minute, do the cron dance
80
        if 60 <= now - self.cronmaster:  # new minute, do the cron dance
254
87
            mfile = open(marker, 'wb')  # touch cron marker
81
            self.cronmaster = tokenmaster(os.path.join(path, 'admin', 'cron'))
255
88
            mfile.close()
82
            if self.cronmaster:
256
89
            crondance(path, 'soft')
83
                crondance(path, 'soft')
257
90
84
                self.cronmaster = tokenmaster(os.path.join(path, 'admin', 'cron'), action = 'release')
258
91
85
259
86
def tokenmaster(path, db = None, action = 'claim'):
260
87
    token = os.path.join(path, 'cron.master')  
261
88
    tokeninuse = os.path.join(path, 'cron.running')  
262
89
263
90
    if action == 'release':
264
91
        logging.debug('WEB2PY CRON: Releasing cron lock')
265
92
        os.unlink(tokeninuse)
266
93
        return time.time()
267
94
268
95
    tokentime = os.stat(token).st_mtime
269
96
    if tokentime - ( tokentime % 60) + 60 > time.time():                # already ran in this minute
270
97
         return 0
271
98
272
99
    if os.path.exists(tokeninuse):                                      # running now
273
100
        logging.warning('alreadyrunning')
274
101
        if os.stat(tokeninuse).st_mtime + 60 < time.time():             # check if stale, just in case
275
102
            logging.warning('WEB2PY CRON: Stale cron.master detected')
276
103
            os.unlink(tokeninuse)
277
104
278
105
    if not (os.path.exists(token) or os.path.exists(tokeninuse)):       # no tokens, new install ? Need to regenerate anyho
279
106
        logging.warning("WEB2PY CRON: cron.master not found at %s. Trying to re-create." % token)
280
107
        try:
281
108
            mfile = open(token, 'wb')
282
109
            mfile.close()
283
110
        except:
284
111
            logging.error('WEB2PY CRON: Unable to re-create cron.master, cron functionality likely not available')
285
112
286
113
    if os.path.exists(token) and not os.path.exists(tokeninuse):        # has unclaimed token and not running
287
114
        logging.debug('WEB2PY CRON: Trying to acquire lock')
288
115
        try:
289
116
            os.rename(token, tokeninuse)
290
117
            mfile = open(token, 'wb')                                   # can't must recreate and not rename as we need a correct claim time
291
118
            mfile.close()
292
119
            logging.debug('WEB2PY CRON: Locked')
293
120
            return os.stat(token).st_mtime
294
121
            
295
122
        except:
296
123
            logging.info('WEB2PY CRON: Failed to claim %s' % token)
297
124
            return 0
298
125
299
126
    logging.debug('WEB2PY CRON: already started from another process')
300
127
    return 0
301
128
    
302
92
def apppath(env=None):
129
def apppath(env=None):
303
93
    try:
130
    try:
304
94
        apppath = os.path.join(env.get('web2py_path'), 'applications')
131
        apppath = os.path.join(env.get('web2py_path'), 'applications')
Status:	Needs review
Proposed branch:	lp:~achipa/web2py/cron
Merge into:	lp:~mdipierro/web2py/devel
Diff against target:	None lines
To merge this branch:	bzr merge lp:~achipa/web2py/cron
Related bugs:	Link a bug report
Related blueprints:	Cron and asynchronous task execution (Undefined)
Reviewer	Review Type	Date Requested	Status
Massimo		2009-03-05	Pending
Review via email: mp+4193@code.launchpad.net