Merge lp:~blake-rouse/maas/largefile-model into lp:maas/trunk

Proposed by Blake Rouse on 2014-08-07
Status: Merged
Merged at revision: 2662
Proposed branch: lp:~blake-rouse/maas/largefile-model
Merge into: lp:maas/trunk
Prerequisite: lp:~blake-rouse/maas/large-object-store
Diff against target: 650 lines (+575/-4)
5 files modified
src/maasserver/migrations/0097_add_largefile_model.py (+344/-0)
src/maasserver/models/__init__.py (+5/-3)
src/maasserver/models/largefile.py (+108/-0)
src/maasserver/models/tests/test_largefile.py (+92/-0)
src/maasserver/testing/factory.py (+26/-1)
To merge this branch: bzr merge lp:~blake-rouse/maas/largefile-model
Reviewer Review Type Date Requested Status
Graham Binns (community) Approve on 2014-08-08
Jason Hobbs (community) 2014-08-07 Needs Information on 2014-08-07
Review via email: mp+229949@code.launchpad.net

This proposal supersedes a proposal from 2014-08-07.

Commit message

Added LargeFile model, that will be used by the incoming BootResource model.

Description of the change

This will be used by the BootResource models to hold the data for the boot images. This design allows multiple files from simplestreams to reference the same content, which is almost always the case for "boot-kernel" and "di-kernel".

Also provides the ability to get the current progress of the file being stored. This will allow the UI to show progress will downloading the boot images.

To post a comment you must log in.
Jason Hobbs (jason-hobbs) wrote :

This mostly looks good - I just have a few questions inline, and some minor things to change. If you have good answers, I'll approve.

review: Needs Information
Blake Rouse (blake-rouse) :
Julian Edwards (julian-edwards) wrote :

On Thursday 07 Aug 2014 14:37:22 you wrote:
> Also provides the ability to get the current progress of the file being
> stored. This will allow the UI to show progress will downloading the boot
> images.

Just a small point, but see src/maasserver/models/downloadprogress.py and
consider whether you need to use that or deprecate it (it's not complete
IIRC).

Jason Hobbs (jason-hobbs) wrote :

On Thu, Aug 7, 2014 at 9:51 PM, Blake Rouse <email address hidden>
wrote:

> '\0' terminating character.
>
> > + ('sha256', self.gf('django.db.models.fields.CharField')(unique=True,
> max_length=65)),
>

What '\0' terminating character? This isn't C...

Jason

Graham Binns (gmb) wrote :

Thanks for this branch Blake; couple of nitpicks from me, but no blockers.

review: Approve
Jason Hobbs (jason-hobbs) wrote :
Download full text (41.0 KiB)

This branch should not land with the sha256sum field size 65. It should
definitely be 64.

On Fri, Aug 8, 2014 at 12:46 PM, Graham Binns <email address hidden>
wrote:

> Review: Approve
>
> Thanks for this branch Blake; couple of nitpicks from me, but no blockers.
>
> Diff comments:
>
> > === added file 'src/maasserver/migrations/0097_add_largefile_model.py'
> > --- src/maasserver/migrations/0097_add_largefile_model.py 1970-01-01
> 00:00:00 +0000
> > +++ src/maasserver/migrations/0097_add_largefile_model.py 2014-08-07
> 19:58:22 +0000
> > @@ -0,0 +1,344 @@
> > +from django.db import models
> > +from south.db import db
> > +# -*- coding: utf-8 -*-
> > +from south.utils import datetime_utils as datetime
> > +from south.v2 import SchemaMigration
> > +
> > +
> > +class Migration(SchemaMigration):
> > +
> > + def forwards(self, orm):
> > + # Adding model 'LargeFile'
> > + db.create_table(u'maasserver_largefile', (
> > + (u'id', self.gf
> ('django.db.models.fields.AutoField')(primary_key=True)),
> > + ('created', self.gf
> ('django.db.models.fields.DateTimeField')()),
> > + ('updated', self.gf
> ('django.db.models.fields.DateTimeField')()),
> > + ('sha256', self.gf('django.db.models.fields.CharField')(unique=True,
> max_length=65)),
> > + ('total_size', self.gf
> ('django.db.models.fields.IntegerField')()),
> > + ('content', self.gf
> ('maasserver.fields.LargeObjectField')()),
> > + ))
> > + db.send_create_signal(u'maasserver', ['LargeFile'])
> > +
> > +
> > + def backwards(self, orm):
> > + # Deleting model 'LargeFile'
> > + db.delete_table(u'maasserver_largefile')
> > +
> > +
> > + models = {
> > + u'auth.group': {
> > + 'Meta': {'object_name': 'Group'},
> > + u'id': ('django.db.models.fields.AutoField', [],
> {'primary_key': 'True'}),
> > + 'name': ('django.db.models.fields.CharField', [],
> {'unique': 'True', 'max_length': '80'}),
> > + 'permissions':
> ('django.db.models.fields.related.ManyToManyField', [], {'to':
> u"orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'})
> > + },
> > + u'auth.permission': {
> > + 'Meta': {'ordering': "(u'content_type__app_label',
> u'content_type__model', u'codename')", 'unique_together':
> "((u'content_type', u'codename'),)", 'object_name': 'Permission'},
> > + 'codename': ('django.db.models.fields.CharField', [],
> {'max_length': '100'}),
> > + 'content_type':
> ('django.db.models.fields.related.ForeignKey', [], {'to':
> u"orm['contenttypes.ContentType']"}),
> > + u'id': ('django.db.models.fields.AutoField', [],
> {'primary_key': 'True'}),
> > + 'name': ('django.db.models.fields.CharField', [],
> {'max_length': '50'})
> > + },
> > + u'auth.user': {
> > + 'Meta': {'object_name': 'User'},
> > + 'date_joined': ('django.db.models.fields.DateTimeField',
> [], {'default': 'datetime.datetime.now'}),
> > + 'email': ('django.db.models.fields.EmailField', [],
> {'unique': 'True', 'max_length': '75', 'blank': 'Tr...

Blake Rouse (blake-rouse) wrote :

Jason,

Changed sha256 field to 64 length.

This command reports it as 65, 'sha256sum setup.py | awk '{print $1}' | wc'. But you are correct python does not need the '\0'.

Jason Hobbs (jason-hobbs) wrote :

thanks!

On Fri, Aug 8, 2014 at 2:19 PM, Blake Rouse <email address hidden>
wrote:

> Jason,
>
> Changed sha256 field to 64 length.
>
> This command reports it as 65, 'sha256sum setup.py | awk '{print $1}' |
> wc'. But you are correct python does not need the '\0'.
> --
> https://code.launchpad.net/~blake-rouse/maas/largefile-model/+merge/229949
> You are reviewing the proposed merge of
> lp:~blake-rouse/maas/largefile-model into lp:maas.
>

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== added file 'src/maasserver/migrations/0097_add_largefile_model.py'
2--- src/maasserver/migrations/0097_add_largefile_model.py 1970-01-01 00:00:00 +0000
3+++ src/maasserver/migrations/0097_add_largefile_model.py 2014-08-08 12:16:36 +0000
4@@ -0,0 +1,344 @@
5+from django.db import models
6+from south.db import db
7+# -*- coding: utf-8 -*-
8+from south.utils import datetime_utils as datetime
9+from south.v2 import SchemaMigration
10+
11+
12+class Migration(SchemaMigration):
13+
14+ def forwards(self, orm):
15+ # Adding model 'LargeFile'
16+ db.create_table(u'maasserver_largefile', (
17+ (u'id', self.gf('django.db.models.fields.AutoField')(primary_key=True)),
18+ ('created', self.gf('django.db.models.fields.DateTimeField')()),
19+ ('updated', self.gf('django.db.models.fields.DateTimeField')()),
20+ ('sha256', self.gf('django.db.models.fields.CharField')(unique=True, max_length=64)),
21+ ('total_size', self.gf('django.db.models.fields.IntegerField')()),
22+ ('content', self.gf('maasserver.fields.LargeObjectField')()),
23+ ))
24+ db.send_create_signal(u'maasserver', ['LargeFile'])
25+
26+
27+ def backwards(self, orm):
28+ # Deleting model 'LargeFile'
29+ db.delete_table(u'maasserver_largefile')
30+
31+
32+ models = {
33+ u'auth.group': {
34+ 'Meta': {'object_name': 'Group'},
35+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
36+ 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '80'}),
37+ 'permissions': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['auth.Permission']", 'symmetrical': 'False', 'blank': 'True'})
38+ },
39+ u'auth.permission': {
40+ 'Meta': {'ordering': "(u'content_type__app_label', u'content_type__model', u'codename')", 'unique_together': "((u'content_type', u'codename'),)", 'object_name': 'Permission'},
41+ 'codename': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
42+ 'content_type': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['contenttypes.ContentType']"}),
43+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
44+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '50'})
45+ },
46+ u'auth.user': {
47+ 'Meta': {'object_name': 'User'},
48+ 'date_joined': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}),
49+ 'email': ('django.db.models.fields.EmailField', [], {'unique': 'True', 'max_length': '75', 'blank': 'True'}),
50+ 'first_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}),
51+ 'groups': ('django.db.models.fields.related.ManyToManyField', [], {'symmetrical': 'False', 'related_name': "u'user_set'", 'blank': 'True', 'to': u"orm['auth.Group']"}),
52+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
53+ 'is_active': ('django.db.models.fields.BooleanField', [], {'default': 'True'}),
54+ 'is_staff': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
55+ 'is_superuser': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
56+ 'last_login': ('django.db.models.fields.DateTimeField', [], {'default': 'datetime.datetime.now'}),
57+ 'last_name': ('django.db.models.fields.CharField', [], {'max_length': '30', 'blank': 'True'}),
58+ 'password': ('django.db.models.fields.CharField', [], {'max_length': '128'}),
59+ 'user_permissions': ('django.db.models.fields.related.ManyToManyField', [], {'symmetrical': 'False', 'related_name': "u'user_set'", 'blank': 'True', 'to': u"orm['auth.Permission']"}),
60+ 'username': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '30'})
61+ },
62+ u'contenttypes.contenttype': {
63+ 'Meta': {'ordering': "('name',)", 'unique_together': "(('app_label', 'model'),)", 'object_name': 'ContentType', 'db_table': "'django_content_type'"},
64+ 'app_label': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
65+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
66+ 'model': ('django.db.models.fields.CharField', [], {'max_length': '100'}),
67+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '100'})
68+ },
69+ u'maasserver.bootimage': {
70+ 'Meta': {'unique_together': "((u'nodegroup', u'osystem', u'architecture', u'subarchitecture', u'release', u'purpose', u'label'),)", 'object_name': 'BootImage'},
71+ 'architecture': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
72+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
73+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
74+ 'label': ('django.db.models.fields.CharField', [], {'default': "u'release'", 'max_length': '255'}),
75+ 'nodegroup': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.NodeGroup']"}),
76+ 'osystem': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
77+ 'purpose': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
78+ 'release': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
79+ 'subarchitecture': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
80+ 'supported_subarches': ('django.db.models.fields.CharField', [], {'max_length': '255', 'blank': 'True'}),
81+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
82+ 'xinstall_path': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'null': 'True', 'blank': 'True'}),
83+ 'xinstall_type': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '30', 'null': 'True', 'blank': 'True'})
84+ },
85+ u'maasserver.bootsource': {
86+ 'Meta': {'object_name': 'BootSource'},
87+ 'cluster': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.NodeGroup']", 'null': 'True'}),
88+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
89+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
90+ 'keyring_data': ('maasserver.fields.EditableBinaryField', [], {'blank': 'True'}),
91+ 'keyring_filename': ('django.db.models.fields.FilePathField', [], {'max_length': '100', 'blank': 'True'}),
92+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
93+ 'url': ('django.db.models.fields.URLField', [], {'max_length': '200'})
94+ },
95+ u'maasserver.bootsourceselection': {
96+ 'Meta': {'object_name': 'BootSourceSelection'},
97+ 'arches': ('djorm_pgarray.fields.ArrayField', [], {'default': 'None', 'dbtype': "u'text'", 'null': 'True', 'blank': 'True'}),
98+ 'boot_source': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.BootSource']"}),
99+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
100+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
101+ 'labels': ('djorm_pgarray.fields.ArrayField', [], {'default': 'None', 'dbtype': "u'text'", 'null': 'True', 'blank': 'True'}),
102+ 'release': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '20', 'blank': 'True'}),
103+ 'subarches': ('djorm_pgarray.fields.ArrayField', [], {'default': 'None', 'dbtype': "u'text'", 'null': 'True', 'blank': 'True'}),
104+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
105+ },
106+ u'maasserver.componenterror': {
107+ 'Meta': {'object_name': 'ComponentError'},
108+ 'component': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '40'}),
109+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
110+ 'error': ('django.db.models.fields.CharField', [], {'max_length': '1000'}),
111+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
112+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
113+ },
114+ u'maasserver.config': {
115+ 'Meta': {'object_name': 'Config'},
116+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
117+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
118+ 'value': ('maasserver.fields.JSONObjectField', [], {'null': 'True'})
119+ },
120+ u'maasserver.dhcplease': {
121+ 'Meta': {'object_name': 'DHCPLease'},
122+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
123+ 'ip': ('maasserver.fields.MAASIPAddressField', [], {'unique': 'True', 'max_length': '39'}),
124+ 'mac': ('maasserver.fields.MACAddressField', [], {}),
125+ 'nodegroup': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.NodeGroup']"})
126+ },
127+ u'maasserver.downloadprogress': {
128+ 'Meta': {'object_name': 'DownloadProgress'},
129+ 'bytes_downloaded': ('django.db.models.fields.IntegerField', [], {'null': 'True', 'blank': 'True'}),
130+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
131+ 'error': ('django.db.models.fields.CharField', [], {'max_length': '1000', 'blank': 'True'}),
132+ 'filename': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
133+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
134+ 'nodegroup': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.NodeGroup']"}),
135+ 'size': ('django.db.models.fields.IntegerField', [], {'null': 'True', 'blank': 'True'}),
136+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
137+ },
138+ u'maasserver.event': {
139+ 'Meta': {'object_name': 'Event'},
140+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
141+ 'description': ('django.db.models.fields.TextField', [], {'default': "u''", 'blank': 'True'}),
142+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
143+ 'node': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.Node']"}),
144+ 'type': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.EventType']"}),
145+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
146+ },
147+ u'maasserver.eventtype': {
148+ 'Meta': {'object_name': 'EventType'},
149+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
150+ 'description': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
151+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
152+ 'level': ('django.db.models.fields.IntegerField', [], {}),
153+ 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '255'}),
154+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
155+ },
156+ u'maasserver.filestorage': {
157+ 'Meta': {'unique_together': "((u'filename', u'owner'),)", 'object_name': 'FileStorage'},
158+ 'content': ('metadataserver.fields.BinaryField', [], {'blank': 'True'}),
159+ 'filename': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
160+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
161+ 'key': ('django.db.models.fields.CharField', [], {'default': "u'aa7f5dec-1e3c-11e4-87da-bcee7b78dc5b'", 'unique': 'True', 'max_length': '36'}),
162+ 'owner': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'to': u"orm['auth.User']", 'null': 'True', 'blank': 'True'})
163+ },
164+ u'maasserver.largefile': {
165+ 'Meta': {'object_name': 'LargeFile'},
166+ 'content': ('maasserver.fields.LargeObjectField', [], {}),
167+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
168+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
169+ 'sha256': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '64'}),
170+ 'total_size': ('django.db.models.fields.IntegerField', [], {}),
171+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
172+ },
173+ u'maasserver.licensekey': {
174+ 'Meta': {'unique_together': "((u'osystem', u'distro_series'),)", 'object_name': 'LicenseKey'},
175+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
176+ 'distro_series': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
177+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
178+ 'license_key': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
179+ 'osystem': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
180+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
181+ },
182+ u'maasserver.macaddress': {
183+ 'Meta': {'ordering': "(u'created',)", 'object_name': 'MACAddress'},
184+ 'cluster_interface': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'to': u"orm['maasserver.NodeGroupInterface']", 'null': 'True', 'blank': 'True'}),
185+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
186+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
187+ 'ip_addresses': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['maasserver.StaticIPAddress']", 'symmetrical': 'False', 'through': u"orm['maasserver.MACStaticIPAddressLink']", 'blank': 'True'}),
188+ 'mac_address': ('maasserver.fields.MACAddressField', [], {'unique': 'True'}),
189+ 'networks': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['maasserver.Network']", 'symmetrical': 'False', 'blank': 'True'}),
190+ 'node': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.Node']"}),
191+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
192+ },
193+ u'maasserver.macstaticipaddresslink': {
194+ 'Meta': {'unique_together': "((u'ip_address', u'mac_address'),)", 'object_name': 'MACStaticIPAddressLink'},
195+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
196+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
197+ 'ip_address': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.StaticIPAddress']", 'unique': 'True'}),
198+ 'mac_address': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.MACAddress']"}),
199+ 'nic_alias': ('django.db.models.fields.IntegerField', [], {'default': 'None', 'null': 'True', 'blank': 'True'}),
200+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
201+ },
202+ u'maasserver.network': {
203+ 'Meta': {'object_name': 'Network'},
204+ 'description': ('django.db.models.fields.TextField', [], {'blank': 'True'}),
205+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
206+ 'ip': ('maasserver.fields.MAASIPAddressField', [], {'unique': 'True', 'max_length': '39'}),
207+ 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '255'}),
208+ 'netmask': ('maasserver.fields.MAASIPAddressField', [], {'max_length': '39'}),
209+ 'vlan_tag': ('django.db.models.fields.PositiveSmallIntegerField', [], {'unique': 'True', 'null': 'True', 'blank': 'True'})
210+ },
211+ u'maasserver.node': {
212+ 'Meta': {'object_name': 'Node'},
213+ 'agent_name': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'null': 'True', 'blank': 'True'}),
214+ 'architecture': ('django.db.models.fields.CharField', [], {'max_length': '31'}),
215+ 'boot_type': ('django.db.models.fields.CharField', [], {'default': "u'fastpath'", 'max_length': '20'}),
216+ 'cpu_count': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
217+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
218+ 'distro_series': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '20', 'blank': 'True'}),
219+ 'error': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
220+ 'error_description': ('django.db.models.fields.TextField', [], {'default': "u''", 'blank': 'True'}),
221+ 'hostname': ('django.db.models.fields.CharField', [], {'default': "u''", 'unique': 'True', 'max_length': '255', 'blank': 'True'}),
222+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
223+ 'license_key': ('django.db.models.fields.CharField', [], {'max_length': '30', 'null': 'True', 'blank': 'True'}),
224+ 'memory': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
225+ 'netboot': ('django.db.models.fields.BooleanField', [], {'default': 'True'}),
226+ 'nodegroup': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.NodeGroup']", 'null': 'True'}),
227+ 'osystem': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '20', 'blank': 'True'}),
228+ 'owner': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'to': u"orm['auth.User']", 'null': 'True', 'blank': 'True'}),
229+ 'power_parameters': ('maasserver.fields.JSONObjectField', [], {'default': "u''", 'blank': 'True'}),
230+ 'power_state': ('django.db.models.fields.CharField', [], {'default': "u'unknown'", 'max_length': '10'}),
231+ 'power_type': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '10', 'blank': 'True'}),
232+ 'routers': ('djorm_pgarray.fields.ArrayField', [], {'default': 'None', 'dbtype': "u'macaddr'", 'null': 'True', 'blank': 'True'}),
233+ 'status': ('django.db.models.fields.IntegerField', [], {'default': '0', 'max_length': '10'}),
234+ 'storage': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
235+ 'system_id': ('django.db.models.fields.CharField', [], {'default': "u'node-aa7fb684-1e3c-11e4-87da-bcee7b78dc5b'", 'unique': 'True', 'max_length': '41'}),
236+ 'tags': ('django.db.models.fields.related.ManyToManyField', [], {'to': u"orm['maasserver.Tag']", 'symmetrical': 'False'}),
237+ 'token': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['piston.Token']", 'null': 'True'}),
238+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
239+ 'zone': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.Zone']", 'on_delete': 'models.SET_DEFAULT'})
240+ },
241+ u'maasserver.nodegroup': {
242+ 'Meta': {'object_name': 'NodeGroup'},
243+ 'api_key': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '18'}),
244+ 'api_token': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['piston.Token']", 'unique': 'True'}),
245+ 'cluster_name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '100', 'blank': 'True'}),
246+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
247+ 'dhcp_key': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
248+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
249+ 'maas_url': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
250+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '80', 'blank': 'True'}),
251+ 'status': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
252+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
253+ 'uuid': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '36'})
254+ },
255+ u'maasserver.nodegroupinterface': {
256+ 'Meta': {'unique_together': "((u'nodegroup', u'name'),)", 'object_name': 'NodeGroupInterface'},
257+ 'broadcast_ip': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
258+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
259+ 'foreign_dhcp_ip': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
260+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
261+ 'interface': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
262+ 'ip': ('maasserver.fields.MAASIPAddressField', [], {'max_length': '39'}),
263+ 'ip_range_high': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
264+ 'ip_range_low': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
265+ 'management': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
266+ 'name': ('django.db.models.fields.CharField', [], {'default': "u''", 'max_length': '255', 'blank': 'True'}),
267+ 'nodegroup': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['maasserver.NodeGroup']"}),
268+ 'router_ip': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
269+ 'static_ip_range_high': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
270+ 'static_ip_range_low': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
271+ 'subnet_mask': ('maasserver.fields.MAASIPAddressField', [], {'default': 'None', 'max_length': '39', 'null': 'True', 'blank': 'True'}),
272+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
273+ },
274+ u'maasserver.sshkey': {
275+ 'Meta': {'unique_together': "((u'user', u'key'),)", 'object_name': 'SSHKey'},
276+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
277+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
278+ 'key': ('django.db.models.fields.TextField', [], {}),
279+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
280+ 'user': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['auth.User']"})
281+ },
282+ u'maasserver.sslkey': {
283+ 'Meta': {'unique_together': "((u'user', u'key'),)", 'object_name': 'SSLKey'},
284+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
285+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
286+ 'key': ('django.db.models.fields.TextField', [], {}),
287+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
288+ 'user': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['auth.User']"})
289+ },
290+ u'maasserver.staticipaddress': {
291+ 'Meta': {'object_name': 'StaticIPAddress'},
292+ 'alloc_type': ('django.db.models.fields.IntegerField', [], {'default': '0'}),
293+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
294+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
295+ 'ip': ('maasserver.fields.MAASIPAddressField', [], {'unique': 'True', 'max_length': '39'}),
296+ 'updated': ('django.db.models.fields.DateTimeField', [], {}),
297+ 'user': ('django.db.models.fields.related.ForeignKey', [], {'default': 'None', 'to': u"orm['auth.User']", 'null': 'True', 'blank': 'True'})
298+ },
299+ u'maasserver.tag': {
300+ 'Meta': {'object_name': 'Tag'},
301+ 'comment': ('django.db.models.fields.TextField', [], {'blank': 'True'}),
302+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
303+ 'definition': ('django.db.models.fields.TextField', [], {'blank': 'True'}),
304+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
305+ 'kernel_opts': ('django.db.models.fields.TextField', [], {'null': 'True', 'blank': 'True'}),
306+ 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '256'}),
307+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
308+ },
309+ u'maasserver.userprofile': {
310+ 'Meta': {'object_name': 'UserProfile'},
311+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
312+ 'user': ('django.db.models.fields.related.OneToOneField', [], {'to': u"orm['auth.User']", 'unique': 'True'})
313+ },
314+ u'maasserver.zone': {
315+ 'Meta': {'ordering': "[u'name']", 'object_name': 'Zone'},
316+ 'created': ('django.db.models.fields.DateTimeField', [], {}),
317+ 'description': ('django.db.models.fields.TextField', [], {'blank': 'True'}),
318+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
319+ 'name': ('django.db.models.fields.CharField', [], {'unique': 'True', 'max_length': '256'}),
320+ 'updated': ('django.db.models.fields.DateTimeField', [], {})
321+ },
322+ u'piston.consumer': {
323+ 'Meta': {'object_name': 'Consumer'},
324+ 'description': ('django.db.models.fields.TextField', [], {}),
325+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
326+ 'key': ('django.db.models.fields.CharField', [], {'max_length': '18'}),
327+ 'name': ('django.db.models.fields.CharField', [], {'max_length': '255'}),
328+ 'secret': ('django.db.models.fields.CharField', [], {'max_length': '32'}),
329+ 'status': ('django.db.models.fields.CharField', [], {'default': "'pending'", 'max_length': '16'}),
330+ 'user': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'consumers'", 'null': 'True', 'to': u"orm['auth.User']"})
331+ },
332+ u'piston.token': {
333+ 'Meta': {'object_name': 'Token'},
334+ 'callback': ('django.db.models.fields.CharField', [], {'max_length': '255', 'null': 'True', 'blank': 'True'}),
335+ 'callback_confirmed': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
336+ 'consumer': ('django.db.models.fields.related.ForeignKey', [], {'to': u"orm['piston.Consumer']"}),
337+ u'id': ('django.db.models.fields.AutoField', [], {'primary_key': 'True'}),
338+ 'is_approved': ('django.db.models.fields.BooleanField', [], {'default': 'False'}),
339+ 'key': ('django.db.models.fields.CharField', [], {'max_length': '18'}),
340+ 'secret': ('django.db.models.fields.CharField', [], {'max_length': '32'}),
341+ 'timestamp': ('django.db.models.fields.IntegerField', [], {'default': '1407420663L'}),
342+ 'token_type': ('django.db.models.fields.IntegerField', [], {}),
343+ 'user': ('django.db.models.fields.related.ForeignKey', [], {'blank': 'True', 'related_name': "'tokens'", 'null': 'True', 'to': u"orm['auth.User']"}),
344+ 'verifier': ('django.db.models.fields.CharField', [], {'max_length': '10'})
345+ }
346+ }
347+
348+ complete_apps = ['maasserver']
349
350=== modified file 'src/maasserver/models/__init__.py'
351--- src/maasserver/models/__init__.py 2014-07-18 15:45:22 +0000
352+++ src/maasserver/models/__init__.py 2014-08-08 12:16:36 +0000
353@@ -22,6 +22,7 @@
354 'DownloadProgress',
355 'Event',
356 'FileStorage',
357+ 'LargeFile',
358 'LicenseKey',
359 'logger',
360 'MACAddress',
361@@ -56,6 +57,7 @@
362 from maasserver.models.event import Event
363 from maasserver.models.eventtype import EventType
364 from maasserver.models.filestorage import FileStorage
365+from maasserver.models.largefile import LargeFile
366 from maasserver.models.licensekey import LicenseKey
367 from maasserver.models.macaddress import MACAddress
368 from maasserver.models.macipaddresslink import MACStaticIPAddressLink
369@@ -77,9 +79,9 @@
370 # export in __all__.
371 ignore_unused(
372 BootImage, ComponentError, Config, DHCPLease, DownloadProgress,
373- Event, EventType, FileStorage, LicenseKey, StaticIPAddress, MACAddress,
374- MACStaticIPAddressLink, Network, NodeGroup, SSHKey, Tag, UserProfile,
375- NodeGroupInterface, Zone, logger)
376+ Event, EventType, FileStorage, LargeFile, LicenseKey, StaticIPAddress,
377+ MACAddress, MACStaticIPAddressLink, Network, NodeGroup, SSHKey, Tag,
378+ UserProfile, NodeGroupInterface, Zone, logger)
379
380
381 # Connect the 'create_user' method to the post save signal of User.
382
383=== added file 'src/maasserver/models/largefile.py'
384--- src/maasserver/models/largefile.py 1970-01-01 00:00:00 +0000
385+++ src/maasserver/models/largefile.py 2014-08-08 12:16:36 +0000
386@@ -0,0 +1,108 @@
387+# Copyright 2014 Canonical Ltd. This software is licensed under the
388+# GNU Affero General Public License version 3 (see the file LICENSE).
389+
390+"""Large file storage."""
391+
392+from __future__ import (
393+ absolute_import,
394+ print_function,
395+ unicode_literals,
396+ )
397+
398+str = None
399+
400+__metaclass__ = type
401+__all__ = [
402+ 'LargeFile',
403+ ]
404+
405+import os
406+
407+from django.db.models import (
408+ BigIntegerField,
409+ CharField,
410+ Manager,
411+ )
412+from django.db.models.signals import post_delete
413+from django.dispatch import receiver
414+from maasserver import DefaultMeta
415+from maasserver.fields import LargeObjectField
416+from maasserver.models.cleansave import CleanSave
417+from maasserver.models.timestampedmodel import TimestampedModel
418+from maasserver.utils.orm import get_one
419+
420+
421+class FileStorageManager(Manager):
422+ """Manager for `LargeFile` objects."""
423+
424+ def has_file(self, sha256):
425+ """True if file with sha256 value exists."""
426+ return self.filter(sha256=sha256).exists()
427+
428+ def get_file(self, sha256):
429+ """Return file based on SHA256 value."""
430+ return get_one(self.filter(sha256=sha256))
431+
432+
433+class LargeFile(CleanSave, TimestampedModel):
434+ """Files that are stored in the large object storage.
435+
436+ Only unique files are stored in the database, as only one sha256 value
437+ can exist per file. This provides data deduplication on the file level.
438+
439+ Currently only used by `BootResourceFile`. This speeds up the import
440+ process by only saving unique files.
441+
442+ :ivar sha256: Calculated SHA256 value of `content`.
443+ :ivar total_size: Final size of `content`. The data might currently
444+ be saving, so total_size could be larger than `size`. `size` should
445+ never be larger than `total_size`.
446+ :ivar content: File data.
447+ """
448+
449+ class Meta(DefaultMeta):
450+ """Needed for South to recognize this model."""
451+
452+ objects = FileStorageManager()
453+
454+ sha256 = CharField(max_length=64, unique=True, editable=False)
455+
456+ total_size = BigIntegerField(editable=False)
457+
458+ # content is stored directly in the database, in the large object storage.
459+ # Max file storage size is 4TB.
460+ content = LargeObjectField()
461+
462+ @property
463+ def size(self):
464+ """Size of content."""
465+ with self.content.open('rb') as stream:
466+ stream.seek(0, os.SEEK_END)
467+ size = stream.tell()
468+ return size
469+
470+ @property
471+ def progress(self):
472+ """Precentage of `content` saved."""
473+ if self.size <= 0:
474+ # Handle division of zero
475+ return 0
476+ return self.total_size / float(self.size)
477+
478+ @property
479+ def complete(self):
480+ """`content` has been completely saved."""
481+ return (self.total_size == self.size)
482+
483+
484+@receiver(post_delete)
485+def delete_large_object(sender, instance, **kwargs):
486+ """Delete the large object when the `LargeFile` is deleted.
487+
488+ This is done using the `post_delete` signal instead of overriding delete
489+ on `LargeFile`, so it works correctly for both the model and
490+ `QuerySet`.
491+ """
492+ if sender == LargeFile:
493+ if instance.content is not None:
494+ instance.content.unlink()
495
496=== added file 'src/maasserver/models/tests/test_largefile.py'
497--- src/maasserver/models/tests/test_largefile.py 1970-01-01 00:00:00 +0000
498+++ src/maasserver/models/tests/test_largefile.py 2014-08-08 12:16:36 +0000
499@@ -0,0 +1,92 @@
500+# Copyright 2014 Canonical Ltd. This software is licensed under the
501+# GNU Affero General Public License version 3 (see the file LICENSE).
502+
503+"""Tests for :class:`LargeFile`."""
504+
505+from __future__ import (
506+ absolute_import,
507+ print_function,
508+ unicode_literals,
509+ )
510+
511+str = None
512+
513+__metaclass__ = type
514+__all__ = []
515+
516+from random import randint
517+
518+from maasserver.models.largefile import LargeFile
519+from maasserver.testing.factory import factory
520+from maasserver.testing.testcase import MAASServerTestCase
521+from maastesting.matchers import MockCalledOnceWith
522+
523+
524+class TestLargeFileManager(MAASServerTestCase):
525+
526+ def test_has_file(self):
527+ largefile = factory.make_large_file()
528+ self.assertTrue(LargeFile.objects.has_file(largefile.sha256))
529+
530+ def test_get_file(self):
531+ largefile = factory.make_large_file()
532+ obj = LargeFile.objects.get_file(largefile.sha256)
533+ self.assertEqual(largefile, obj)
534+
535+
536+class TestLargeFile(MAASServerTestCase):
537+
538+ def test_content(self):
539+ size = randint(512, 1024)
540+ content = factory.make_string(size=size)
541+ largefile = factory.make_large_file(content, size=size)
542+ with largefile.content.open('rb') as stream:
543+ data = stream.read()
544+ self.assertEqual(content, data)
545+
546+ def test_empty_content(self):
547+ size = 0
548+ content = ""
549+ largefile = factory.make_large_file(content, size=size)
550+ with largefile.content.open('rb') as stream:
551+ data = stream.read()
552+ self.assertEqual(content, data)
553+
554+ def test_size(self):
555+ size = randint(512, 1024)
556+ total_size = randint(1025, 2048)
557+ content = factory.make_string(size=size)
558+ largefile = factory.make_large_file(content, size=total_size)
559+ self.assertEqual(size, largefile.size)
560+
561+ def test_progress(self):
562+ size = randint(512, 1024)
563+ total_size = randint(1025, 2048)
564+ content = factory.make_string(size=size)
565+ largefile = factory.make_large_file(content, size=total_size)
566+ self.assertEqual(total_size / float(size), largefile.progress)
567+
568+ def test_progress_of_empty_file(self):
569+ size = 0
570+ content = ""
571+ largefile = factory.make_large_file(content, size=size)
572+ self.assertEqual(0, largefile.progress)
573+
574+ def test_complete_returns_False_when_content_incomplete(self):
575+ size = randint(512, 1024)
576+ total_size = randint(1025, 2048)
577+ content = factory.make_string(size=size)
578+ largefile = factory.make_large_file(content, size=total_size)
579+ self.assertFalse(largefile.complete)
580+
581+ def test_complete_returns_True_when_content_is_complete(self):
582+ largefile = factory.make_large_file()
583+ self.assertTrue(largefile.complete)
584+
585+ def test_delete_calls_unlink_on_content(self):
586+ largefile = factory.make_large_file()
587+ content = largefile.content
588+ self.addCleanup(content.unlink)
589+ unlink_mock = self.patch(content, 'unlink')
590+ largefile.delete()
591+ self.assertThat(unlink_mock, MockCalledOnceWith())
592
593=== modified file 'src/maasserver/testing/factory.py'
594--- src/maasserver/testing/factory.py 2014-08-01 11:19:44 +0000
595+++ src/maasserver/testing/factory.py 2014-08-08 12:16:36 +0000
596@@ -17,6 +17,7 @@
597 "Messages",
598 ]
599
600+import hashlib
601 from io import BytesIO
602 import logging
603 import random
604@@ -32,7 +33,10 @@
605 NODEGROUPINTERFACE_MANAGEMENT,
606 POWER_STATE,
607 )
608-from maasserver.fields import MAC
609+from maasserver.fields import (
610+ LargeObjectFile,
611+ MAC,
612+ )
613 from maasserver.models import (
614 BootImage,
615 BootSource,
616@@ -42,6 +46,7 @@
617 Event,
618 EventType,
619 FileStorage,
620+ LargeFile,
621 LicenseKey,
622 MACAddress,
623 MACStaticIPAddressLink,
624@@ -950,6 +955,26 @@
625 type = self.make_event_type()
626 return Event.objects.create(node=node, type=type)
627
628+ def make_large_file(self, content=None, size=512):
629+ """Create `LargeFile`.
630+
631+ :param content: Data to store in large file object.
632+ :param size: Size of `content`. If `content` is None
633+ then it will be a random string of this size. If content is
634+ provided and `size` is not the same length, then it will
635+ be an inprogress file.
636+ """
637+ if content is None:
638+ content = factory.make_string(size=size)
639+ sha256 = hashlib.sha256()
640+ sha256.update(content)
641+ sha256 = sha256.hexdigest()
642+ largeobject = LargeObjectFile()
643+ with largeobject.open('wb') as stream:
644+ stream.write(content)
645+ return LargeFile.objects.create(
646+ sha256=sha256, total_size=size, content=largeobject)
647+
648
649 # Create factory singleton.
650 factory = Factory()