Merge lp:~mabac/linaro-image-tools/snowball-support into lp:linaro-image-tools/11.11
- snowball-support
- Merge into trunk
Status: | Merged | ||||
---|---|---|---|---|---|
Approved by: | Mattias Backman | ||||
Approved revision: | 353 | ||||
Merged at revision: | 339 | ||||
Proposed branch: | lp:~mabac/linaro-image-tools/snowball-support | ||||
Merge into: | lp:linaro-image-tools/11.11 | ||||
Diff against target: |
220 lines (+178/-0) 2 files modified
linaro_image_tools/media_create/boards.py (+146/-0) linaro_image_tools/media_create/tests/test_media_create.py (+32/-0) |
||||
To merge this branch: | bzr merge lp:~mabac/linaro-image-tools/snowball-support | ||||
Related bugs: |
|
Reviewer | Review Type | Date Requested | Status |
---|---|---|---|
Tony Mansson (community) | Approve | ||
Hans Odeberg (community) | Approve | ||
Loïc Minier (community) | Approve | ||
Review via email: mp+61582@code.launchpad.net |
This proposal supersedes a proposal from 2011-05-16.
Commit message
Description of the change
Hi,
This branch is is based on the initial Snowball support written by Tony Månsson and Hans Odeberg @ STE. I have made as non-intrusive changes as possible while taking in the information posted in several previous merge proposals:
https:/
https:/
https:/
This change introduces two board configurations, namely:
SnowballSdC
SnowballEmm
There is still some work to be done before this should be merged:
Assert that TOC does not overflow hardcoded boundary of 512 bytes. DONE
Other checks to secure that files are written in a controlled manner.
Write unit tests for the new code.
Thanks,
Mattias
Mattias Backman (mabac) wrote : Posted in a previous version of this proposal | # |
Hans Odeberg (hans-odeberg) wrote : Posted in a previous version of this proposal | # |
"Move functions install_
Did you rerun the unit tests after this? I had to move these functions to a global scope when adding unit tests, as otherwise the test framework would not stub them, and they would try to install non-existing files.
Mattias Backman (mabac) wrote : Posted in a previous version of this proposal | # |
On Tue, May 17, 2011 at 10:01 AM, Hans Odeberg
<email address hidden> wrote:
> "Move functions install_
> Did you rerun the unit tests after this? I had to move these functions to a global scope when adding unit tests, as otherwise the test framework would not stub them, and they would try to install non-existing files.
Yes, we need to create other ways to test these functions, possibly by
explicitly mocking them. Imo they belong to the SnowballConfig class
and if they do, they shouldn't be global just for the sake of testing.
Guilherme Salgado (salgado) wrote : Posted in a previous version of this proposal | # |
Yeah, we should have some unit tests for get_file_info(), create_toc() and install_
Hans Odeberg (hans-odeberg) wrote : Posted in a previous version of this proposal | # |
OK, I will put writing the additional unit tests on my desk. If I cannot figure out how to do the mocking, I will get back to you with the hope you don't mock me...
Guilherme Salgado (salgado) wrote : Posted in a previous version of this proposal | # |
On Wed, 2011-05-18 at 08:48 +0000, Hans Odeberg wrote:
> OK, I will put writing the additional unit tests on my desk. If I cannot figure out how to do the mocking, I will get back to you with the hope you don't mock me...
I certainly wouldn't do that, and I'd be happy to work with you on
writing those tests (maybe using http://
if Mattias doesn't beat us to it :)
Loïc Minier (lool) wrote : | # |
Looks good to me; the linaro-media-create flag is missing as well.
I think it would be ok to merge with:
- the flag to set the file implemented
- a note in the code to add the sanity checks
- a bug to remind ourselves to add tests
- a change to hwpackv2 spec to support this stuff
Mattias Backman (mabac) wrote : | # |
> Looks good to me; the linaro-media-create flag is missing as well.
>
> I think it would be ok to merge with:
> - the flag to set the file implemented
Actually I think we realized that the file that describes the "startup files" has to be in the hwpack. This discussion might have been f2f or in STE email, there's been a lot of parallell threads so I forgot about that.
The reason for keeping the description file with the hwpack is that the hwpack contains the startup files and whoever creates the hwpack can make sure that the description file matches the contents.
This would make it possible to skip the flag to linaro-media-create since the description file can have a fixed file name. Of course I need to make it relative to wherever the file would be unpacked from the hwpack.
> - a note in the code to add the sanity checks
> - a bug to remind ourselves to add tests
> - a change to hwpackv2 spec to support this stuff
Sure, that's a good idea.
Note to self: remove snowball_toc.txt from the tree before merging.
Hans Odeberg (hans-odeberg) wrote : | # |
Looks good to me, so far (with the limitations on tests and sanity checks you mentioned above).
Just a question on SNOWBALL_
Mattias Backman (mabac) wrote : | # |
On Thu, May 19, 2011 at 6:11 PM, Hans Odeberg
<email address hidden> wrote:
> Looks good to me, so far (with the limitations on tests and sanity checks you mentioned above).
>
> Just a question on SNOWBALL_
Yes, I just added that as an example of the format I expect the file
to be in. But it will be removed from the source tree before the
change is ready. I would like to have a complete hwpack with the file
in it for testing before I'm satisfied with this.
Mattias Backman (mabac) wrote : | # |
> I think it would be ok to merge with:
> - the flag to set the file implemented
It's assumed that the file is packaged with the startup files in the hwpack instead. Is that ok?
> - a note in the code to add the sanity checks
Done.
> - a bug to remind ourselves to add tests
> - a change to hwpackv2 spec to support this stuff
I'll take care of these today.
Mattias Backman (mabac) wrote : | # |
I have tested this by creating a "raw image" for the board like this:
./linaro-
and writing the image to the board like this:
sudo riff -f snowball.bin
The board boots to prompt with the nano image. I hope STE (Tony?) will verify this more thoroughly while reviewing.
Note that the riff debs where taken from an internal share at STE and the hwpack was taken from Tony's network share. Can I redistribute these within Linaro? The hwpack does not contain the startfiles.cfg file yet btw.
Thanks,
Mattias
Loïc Minier (lool) wrote : | # |
On Fri, May 20, 2011, Mattias Backman wrote:
> It's assumed that the file is packaged with the startup files in the
> hwpack instead. Is that ok?
Yup, that's also ok
--
Loïc Minier
Loïc Minier (lool) wrote : | # |
Branch looks good to me; since you removed the sample toc file and since I don't think we have support in linaro-
===
hwpacks for use with --dev snowball-emmc need a file called toc.cfg at the root of the tarball, but this isn't yet supported by linaro-
<sample TOC file>
===
or file a bug to remind ourselves about this lack of support for toc files in linaro-
Loïc Minier (lool) : | # |
Hans Odeberg (hans-odeberg) wrote : | # |
> Branch looks good to me; since you removed the sample toc file and since I
> don't think we have support in linaro-
> you add some explanation under doc/ on how the toc file and on repacking
> hwpacks with such a file; e.g. a doc/snowball-
> like that which would say something like:
> ===
> hwpacks for use with --dev snowball-emmc need a file called toc.cfg at the
> root of the tarball, but this isn't yet supported by linaro-
> repack the hwpack by unpacking the tarball, adding the toc file and the files
> it references at the root of the tarball and repack the tarball. The TOC file
> has this format:
> <sample TOC file>
> ===
>
> or file a bug to remind ourselves about this lack of support for toc files in
> linaro-
Do we need to add support for toc files in linaro-
Mattias Backman (mabac) wrote : | # |
About hwpack-create; I'll add the documentation if needed of course. I can see that the STE binaries are included in the hwpack as a deb package: startupfiles_
startupfiles_
boot/
u-boot.bin
...
Hans Odeberg (hans-odeberg) : | # |
Tony Mansson (tony-mansson) wrote : | # |
I am OK with this as "good enough". For future enhancement I'd like to note:
1) The bootloader environment in 'bootargs' and 'bootcmd' are never used currently, as we have a separate u-boot-env.bin file with these. I am unsure of where LMC puts its hard-coded data and I don't see the advantage of having it here. It does however no harm other than confuse. OK?
2) For the final version we need to have two separate packages containing startup files as the licenses are different (u-boot is GPL and the others are proprietary). as I understand this implementation there is only room for one SNOWBALL_
Mattias Backman (mabac) wrote : | # |
> I am OK with this as "good enough". For future enhancement I'd like to note:
Great, thank you.
>
> 1) The bootloader environment in 'bootargs' and 'bootcmd' are never used
> currently, as we have a separate u-boot-env.bin file with these. I am unsure
> of where LMC puts its hard-coded data and I don't see the advantage of having
> it here. It does however no harm other than confuse. OK?
I think we can handle that in the hwpack v2 work which is about taking away the hard coded data and let hwpacks handle it all. I'm going to start expanding this spec next week to take into account the discussions we had at LDS.
https:/
>
> 2) For the final version we need to have two separate packages containing
> startup files as the licenses are different (u-boot is GPL and the others are
> proprietary). as I understand this implementation there is only room for one
> SNOWBALL_
> of both its own plus an other package which is not pretty. A general solution
> would e.g loop over all packages and find one cfg file for each package. The
> current solution is however legally OK and works for now.
Then we'll go with this if it works for you and let's change it as soon as that new packaging scheme is fixed.
Hans Odeberg (hans-odeberg) wrote : | # |
> 2) For the final version we need to have two separate packages containing
> startup files as the licenses are different (u-boot is GPL and the others are
> proprietary). as I understand this implementation there is only room for one
> SNOWBALL_
> of both its own plus an other package which is not pretty. A general solution
> would e.g loop over all packages and find one cfg file for each package. The
> current solution is however legally OK and works for now.
My opinion is that the HW pack should be responsible for placing one configuration file in the /boot directory. This could be done by:
1) Putting the config file in the startup_files package. Simple, but ugly, as the file will then refer to a file in another package.
2) Putting the config file in a separate package, which then has dependencies on a specific version of the startup file and u-boot packages. Sort of like how a devel package depends on a specific version of a binary package.
3) Putting one config file fragment in each of the startup_files and u-boot packages, have the u-boot package depend on the startup_files package, and then include an install script in the u-boot package which combines the two fragments into a config file.
I'm sure more complicated schemes could be invented, too. The end result should however look the same to LMC: a config file in the /boot directory.
Preview Diff
1 | === modified file 'linaro_image_tools/media_create/boards.py' |
2 | --- linaro_image_tools/media_create/boards.py 2011-04-26 16:37:19 +0000 |
3 | +++ linaro_image_tools/media_create/boards.py 2011-05-20 08:27:14 +0000 |
4 | @@ -495,6 +495,150 @@ |
5 | make_boot_script(boot_env, boot_script_path) |
6 | |
7 | |
8 | +class SnowballSdConfig(Ux500Config): |
9 | + '''Use only with --mmc option. Creates the standard vfat and ext2 |
10 | + partitions for kernel and rootfs on an SD card. |
11 | + Note that the Snowball board needs a loader partition on the |
12 | + internal eMMC flash to boot. That partition is created with |
13 | + the SnowballConfigImage configuration.''' |
14 | + |
15 | + @classmethod |
16 | + def _make_boot_files(cls, boot_env, chroot_dir, boot_dir, |
17 | + boot_device_or_file, k_img_data, i_img_data, |
18 | + d_img_data): |
19 | + make_uImage(cls.load_addr, k_img_data, boot_dir) |
20 | + boot_script_path = os.path.join(boot_dir, cls.boot_script) |
21 | + make_boot_script(boot_env, boot_script_path) |
22 | + |
23 | + |
24 | +class SnowballEmmcConfig(SnowballSdConfig): |
25 | + '''Use only with --image option. Creates a raw image which contains an |
26 | + additional (raw) loader partition, containing some boot stages |
27 | + and u-boot.''' |
28 | + # Boot ROM looks for a boot table of contents (TOC) at 0x20000 |
29 | + # Actually, it first looks at address 0, but that's where l-m-c |
30 | + # puts the MBR, so the boot loader skips that address. |
31 | + SNOWBALL_LOADER_START_S = (128 * 1024) / SECTOR_SIZE |
32 | + SNOWBALL_STARTUP_FILES_CONFIG = 'startfiles.cfg' |
33 | + TOC_SIZE = 512 |
34 | + |
35 | + @classmethod |
36 | + def get_sfdisk_cmd(cls, should_align_boot_part=None): |
37 | + """Return the sfdisk command to partition the media. |
38 | + |
39 | + :param should_align_boot_part: Ignored. |
40 | + |
41 | + The Snowball partitioning scheme depends on whether the target is |
42 | + a raw image or an SD card. Both targets have the normal |
43 | + FAT 32 boot partition and EXT? root partition. |
44 | + The raw image prepends these two partitions with a raw loader partition, |
45 | + containing HW-dependent boot stages up to and including u-boot. |
46 | + This is done since the boot rom always boots off the internal memory; |
47 | + there simply is no point to having a loader partition on SD card. |
48 | + """ |
49 | + # boot ROM expects bootloader at 0x20000, which is sector 0x100 |
50 | + # with the usual SECTOR_SIZE of 0x200. |
51 | + # (sector 0 is MBR / partition table) |
52 | + loader_start, loader_end, loader_len = align_partition( |
53 | + SnowballEmmcConfig.SNOWBALL_LOADER_START_S, |
54 | + LOADER_MIN_SIZE_S, 1, PART_ALIGN_S) |
55 | + |
56 | + boot_start, boot_end, boot_len = align_partition( |
57 | + loader_end + 1, BOOT_MIN_SIZE_S, PART_ALIGN_S, PART_ALIGN_S) |
58 | + # we ignore _root_end / _root_len and return an sfdisk command to |
59 | + # instruct the use of all remaining space; XXX if we had some root size |
60 | + # config, we could do something more sensible |
61 | + root_start, _root_end, _root_len = align_partition( |
62 | + boot_end + 1, ROOT_MIN_SIZE_S, PART_ALIGN_S, PART_ALIGN_S) |
63 | + |
64 | + return '%s,%s,0xDA\n%s,%s,0x0C,*\n%s,,,-' % ( |
65 | + loader_start, loader_len, boot_start, boot_len, root_start) |
66 | + |
67 | + @classmethod |
68 | + def _make_boot_files(cls, boot_env, chroot_dir, boot_dir, |
69 | + boot_device_or_file, k_img_data, i_img_data, |
70 | + d_img_data): |
71 | + make_uImage(cls.load_addr, k_img_data, boot_dir) |
72 | + boot_script_path = os.path.join(boot_dir, cls.boot_script) |
73 | + make_boot_script(boot_env, boot_script_path) |
74 | + _, toc_filename = tempfile.mkstemp() |
75 | + atexit.register(os.unlink, toc_filename) |
76 | + config_files_path = os.path.join(chroot_dir, 'boot') |
77 | + new_files = cls.get_file_info(config_files_path) |
78 | + with open(toc_filename, 'wb') as toc: |
79 | + cls.create_toc(toc, new_files) |
80 | + cls.install_snowball_boot_loader(toc_filename, new_files, |
81 | + boot_device_or_file, |
82 | + cls.SNOWBALL_LOADER_START_S) |
83 | + |
84 | + @classmethod |
85 | + def install_snowball_boot_loader(cls, toc_file_name, files, |
86 | + boot_device_or_file, start_sector): |
87 | + ''' Copies TOC and boot files into the boot partition. |
88 | + A sector size of 1 is used for some files, as they do not |
89 | + necessarily start on an even address. ''' |
90 | + assert os.path.getsize(toc_file_name) <= cls.TOC_SIZE |
91 | + _dd(toc_file_name, boot_device_or_file, seek=start_sector) |
92 | + |
93 | + for file in files: |
94 | + # XXX We need checks that these files do not overwrite each |
95 | + # other. This code assumes that offset and file sizes are ok. |
96 | + if (file['offset'] % SECTOR_SIZE) != 0: |
97 | + seek_bytes = start_sector * SECTOR_SIZE + file['offset'] |
98 | + _dd(file['filename'], boot_device_or_file, block_size=1, |
99 | + seek=seek_bytes) |
100 | + else: |
101 | + seek_sectors = start_sector + file['offset']/SECTOR_SIZE |
102 | + _dd(file['filename'], boot_device_or_file, seek=seek_sectors) |
103 | + |
104 | + @classmethod |
105 | + def create_toc(cls, f, files): |
106 | + ''' Writes a table of contents of the boot binaries. |
107 | + Boot rom searches this table to find the binaries.''' |
108 | + for file in files: |
109 | + # Format string means: < little endian, |
110 | + # I; unsigned int; offset, |
111 | + # I; unsigned int; size, |
112 | + # I; unsigned int; flags, |
113 | + # i; int; align, |
114 | + # i; int; load_address, |
115 | + # 12s; string of char; name |
116 | + # http://igloocommunity.org/support/index.php/ConfigPartitionOverview |
117 | + flags = 0 |
118 | + load_adress = file['align'] |
119 | + data = struct.pack('<IIIii12s', file['offset'], file['size'], |
120 | + flags, file['align'], load_adress, |
121 | + file['section_name']) |
122 | + f.write(data) |
123 | + |
124 | + @classmethod |
125 | + def get_file_info(cls, bin_dir): |
126 | + ''' Fills in the offsets of files that are located in |
127 | + non-absolute memory locations depending on their sizes.' |
128 | + Also fills in file sizes''' |
129 | + ofs = cls.TOC_SIZE |
130 | + files = [] |
131 | + with open(os.path.join(bin_dir, cls.SNOWBALL_STARTUP_FILES_CONFIG), |
132 | + 'r') as info_file: |
133 | + for line in info_file: |
134 | + file_data = line.split() |
135 | + if file_data[0][0] == '#': |
136 | + continue |
137 | + filename = os.path.join(bin_dir, file_data[1]) |
138 | + address = long(file_data[3], 16) |
139 | + if address != 0: |
140 | + ofs = address |
141 | + size = os.path.getsize(filename) |
142 | + files.append({'section_name': file_data[0], |
143 | + 'filename': filename, |
144 | + 'align': int(file_data[2]), |
145 | + 'offset': ofs, |
146 | + 'size': size, |
147 | + 'load_adress': file_data[4]}) |
148 | + ofs += size |
149 | + return files |
150 | + |
151 | + |
152 | class Mx5Config(BoardConfig): |
153 | serial_tty = 'ttymxc0' |
154 | extra_serial_opts = 'console=tty0 console=%s,115200n8' % serial_tty |
155 | @@ -687,6 +831,8 @@ |
156 | 'panda': PandaConfig, |
157 | 'vexpress': VexpressConfig, |
158 | 'ux500': Ux500Config, |
159 | + 'snowball_sd': SnowballSdConfig, |
160 | + 'snowball_emmc': SnowballEmmcConfig, |
161 | 'efikamx': EfikamxConfig, |
162 | 'efikasb': EfikasbConfig, |
163 | 'mx51evk': Mx51evkConfig, |
164 | |
165 | === modified file 'linaro_image_tools/media_create/tests/test_media_create.py' |
166 | --- linaro_image_tools/media_create/tests/test_media_create.py 2011-05-10 15:05:57 +0000 |
167 | +++ linaro_image_tools/media_create/tests/test_media_create.py 2011-05-20 08:27:14 +0000 |
168 | @@ -213,6 +213,11 @@ |
169 | expected = ['make_uImage', 'make_uInitrd', 'make_boot_script'] |
170 | self.assertEqual(expected, self.funcs_calls) |
171 | |
172 | + def test_snowball_sd_steps(self): |
173 | + self.make_boot_files(boards.SnowballSdConfig) |
174 | + expected = ['make_uImage', 'make_boot_script'] |
175 | + self.assertEqual(expected, self.funcs_calls) |
176 | + |
177 | def test_panda_steps(self): |
178 | self.mock_set_appropriate_serial_tty(boards.PandaConfig) |
179 | self.make_boot_files(boards.PandaConfig) |
180 | @@ -330,6 +335,16 @@ |
181 | '1,8191,0xDA\n8192,106496,0x0C,*\n114688,,,-', |
182 | boards.Mx5Config.get_sfdisk_cmd()) |
183 | |
184 | + def test_snowball_sd(self): |
185 | + self.assertEqual( |
186 | + '63,106432,0x0C,*\n106496,,,-', |
187 | + boards.SnowballSdConfig.get_sfdisk_cmd()) |
188 | + |
189 | + def test_snowball_emmc(self): |
190 | + self.assertEqual( |
191 | + '256,7936,0xDA\n8192,106496,0x0C,*\n114688,,,-', |
192 | + boards.SnowballEmmcConfig.get_sfdisk_cmd()) |
193 | + |
194 | def test_smdkv310(self): |
195 | self.assertEquals( |
196 | '1,8191,0xDA\n8192,106496,0x0C,*\n114688,,,-', |
197 | @@ -393,6 +408,23 @@ |
198 | 'bootm 0x00100000 0x08000000'} |
199 | self.assertEqual(expected, boot_commands) |
200 | |
201 | + def test_snowball_emmc(self): |
202 | + boot_commands = board_configs['snowball_emmc']._get_boot_env( |
203 | + is_live=False, is_lowmem=False, consoles=[], |
204 | + rootfs_uuid="deadbeef", d_img_data=None) |
205 | + expected = { |
206 | + 'bootargs': 'console=tty0 console=ttyAMA2,115200n8 ' |
207 | + 'root=UUID=deadbeef rootwait ro earlyprintk ' |
208 | + 'rootdelay=1 fixrtc nocompcache mem=96M@0 ' |
209 | + 'mem_modem=32M@96M mem=44M@128M pmem=22M@172M ' |
210 | + 'mem=30M@194M mem_mali=32M@224M pmem_hwb=54M@256M ' |
211 | + 'hwmem=48M@302M mem=152M@360M', |
212 | + 'bootcmd': 'fatload mmc 1:1 0x00100000 uImage; ' |
213 | + 'fatload mmc 1:1 0x08000000 uInitrd; ' |
214 | + 'bootm 0x00100000 0x08000000'} |
215 | + self.assertEqual(expected, boot_commands) |
216 | + |
217 | + |
218 | def test_panda(self): |
219 | # XXX: To fix bug 697824 we have to change class attributes of our |
220 | # OMAP board configs, and some tests do that so to make sure they |
There should be a hwpack at https:/ /github. com/Winterland/ linaro- hwpack which can be combined with https:/ /github. com/Winterland/ startup to build a new .deb to test with. I get only 404 though, possible lacking permissions.