Merge lp:~laurynas-biveinis/percona-xtrabackup/bug1049174-2.0 into lp:percona-xtrabackup/2.0

Proposed by Laurynas Biveinis on 2012-09-14
Status: Merged
Approved by: Alexey Kopytov on 2012-09-19
Approved revision: 467
Merged at revision: 469
Proposed branch: lp:~laurynas-biveinis/percona-xtrabackup/bug1049174-2.0
Merge into: lp:percona-xtrabackup/2.0
Diff against target: 304 lines (+139/-56)
1 file modified
src/xtrabackup.c (+139/-56)
To merge this branch: bzr merge lp:~laurynas-biveinis/percona-xtrabackup/bug1049174-2.0
Reviewer Review Type Date Requested Status
Alexey Kopytov (community) 2012-09-14 Approve on 2012-09-19
Review via email: mp+124411@code.launchpad.net

Description of the change

Fix handling of compressed tablespaces, created between the last full
or incremental and the next incremental backup.

Bugs fixed:
bug 1049174 (Redundant zip_size handling at delta application
time),
bug 1044398 (Handling of compressed tablespaces with compressed page
size == server page size broken),
bug 1043762 (Incremental backups may take up to 5x more space after
the fix for bug #1022562).

In order not to include the first 64KB of a tablespace unconditionally
in every delta (bug 1043762), we need to be able to create a
tablespace on disk without relying on the presence of the first pages
of a newly-created tablespace in the delta. These pages, in the
absence of any later writes to them, get a modification LSN of 0,
causing them to be omitted from the delta.

Thus we create new tablespaces from scratch by a new function
xb_delta_create_space_file() that is similar to
fil_create_new_single_table_tablespace() in InnoDB. For this function
to work, it needs a tablespace flags value, and a default value of 0
(which assumes a regular uncompressed tablespace) will cause asserts
in trying to apply log records to it later, if the tablespace is
actually compressed and its header page was not overwritten by the
delta due to reasons above. Thus, we need the actual flags value.

For this purpose we extend the incremental backup delta metadata with
a new field zip_size. On backup preperation the presence of this flag
is optional, to keep the ability to restore from the old backups and
because its value is required only in the case describe above (a new
compressed tablespace created between the backups, no log records
touching its 1st page).

Always pass zero to fil_space_create() instead of the tablespace flags
in xb_delta_open_matching_space(). Push the flags calculation between
fil_space_create() and xb_delta_create_space_file(). Calculate these
flags for all XtraBackup flavours by conditionally defining the
required constants, to minimize the conditionally-compiled code. If
there was no zip_size in the metadata, assume flags to be zero.

Revert the changes in xtrabackup_copy_datafile() made for bug 1022562,
that is, do not store the first 64KB of every tablespace
unconditionally.

In xtrabackup_apply_delta(), remove the assumption that zip_size is
equal to page_size whenever page_size < UNIV_PAGE_SIZE, fixing bug
10443398
.

No testsuite changes, as this is non-functional case. The regular
incremental compressed backups are tested by the
xb_incremental_compressed test, an the corner case of a new compressed
tablespace that does not have its 1st page in the delta is tested by
bug1028949 test.

Jenkins: http://jenkins.percona.com/job/percona-xtrabackup-2.0-param/261/

Issue #16274

To post a comment you must log in.
Alexey Kopytov (akopytov) :
review: Approve

Preview Diff

[H/L] Next/Prev Comment, [J/K] Next/Prev File, [N/P] Next/Prev Hunk
1=== modified file 'src/xtrabackup.c'
2--- src/xtrabackup.c 2012-09-05 08:32:02 +0000
3+++ src/xtrabackup.c 2012-09-14 12:55:19 +0000
4@@ -105,6 +105,10 @@
5 #define MACH_READ_64 mach_read_from_8
6 #define MACH_WRITE_64 mach_write_to_8
7 #define OS_MUTEX_CREATE() os_mutex_create(NULL)
8+#define PAGE_ZIP_MIN_SIZE_SHIFT 10
9+#define DICT_TF_ZSSIZE_SHIFT 1
10+#define DICT_TF_FORMAT_ZIP 1
11+#define DICT_TF_FORMAT_SHIFT 5
12 #else
13 #define IB_INT64 ib_int64_t
14 #define LSN64 ib_uint64_t
15@@ -655,6 +659,7 @@
16
17 typedef struct {
18 ulint page_size;
19+ ulint zip_size;
20 ulint space_id;
21 } xb_delta_info_t;
22
23@@ -2855,6 +2860,7 @@
24
25 /* set defaults */
26 info->page_size = ULINT_UNDEFINED;
27+ info->zip_size = ULINT_UNDEFINED;
28 info->space_id = ULINT_UNDEFINED;
29
30 fp = fopen(filepath, "r");
31@@ -2867,6 +2873,8 @@
32 if (fscanf(fp, "%50s = %50s\n", key, value) == 2) {
33 if (strcmp(key, "page_size") == 0) {
34 info->page_size = strtoul(value, NULL, 10);
35+ } else if (strcmp(key, "zip_size") == 0) {
36+ info->zip_size = strtoul(value, NULL, 10);
37 } else if (strcmp(key, "space_id") == 0) {
38 info->space_id = strtoul(value, NULL, 10);
39 }
40@@ -2904,8 +2912,10 @@
41 MY_STAT mystat;
42
43 snprintf(buf, sizeof(buf),
44- "page_size = %lu\nspace_id = %lu\n",
45- info->page_size, info->space_id);
46+ "page_size = %lu\n"
47+ "zip_size = %lu\n"
48+ "space_id = %lu\n",
49+ info->page_size, info->zip_size, info->space_id);
50 len = strlen(buf);
51
52 mystat.st_size = len;
53@@ -3103,14 +3113,12 @@
54 byte* incremental_buffer_base = NULL;
55 ulint page_size;
56 ulint page_size_shift;
57-#ifdef INNODB_VERSION_SHORT
58- ulint zip_size;
59-#endif
60 xb_delta_info_t info;
61 datasink_t *ds = ds_ctxt->datasink;
62 ds_file_t *dstfile = NULL;
63
64 info.page_size = 0;
65+ info.zip_size = 0;
66 info.space_id = 0;
67
68 #ifdef XTRADB_BASED
69@@ -3257,11 +3265,11 @@
70 page_size = UNIV_PAGE_SIZE;
71 page_size_shift = UNIV_PAGE_SIZE_SHIFT;
72 #else
73- zip_size = xb_get_zip_size(src_file);
74- if (zip_size == ULINT_UNDEFINED) {
75+ info.zip_size = xb_get_zip_size(src_file);
76+ if (info.zip_size == ULINT_UNDEFINED) {
77 goto skip;
78- } else if (zip_size) {
79- page_size = zip_size;
80+ } else if (info.zip_size) {
81+ page_size = info.zip_size;
82 page_size_shift = get_bit_shift(page_size);
83 msg("[%02u] %s is compressed with page size = "
84 "%lu bytes\n", thread_n, node->name, page_size);
85@@ -3367,7 +3375,8 @@
86 #ifndef INNODB_VERSION_SHORT
87 if (buf_page_is_corrupted(page + chunk_offset))
88 #else
89- if (buf_page_is_corrupted(page + chunk_offset, zip_size))
90+ if (buf_page_is_corrupted(page + chunk_offset,
91+ info.zip_size))
92 #endif
93 {
94 if (
95@@ -3418,18 +3427,11 @@
96 if (xtrabackup_incremental) {
97 for (chunk_offset = 0; chunk_offset < chunk; chunk_offset += page_size) {
98 /* newer page */
99- /* This condition may be OK for header, ibuf and fsp
100- We always copy the 1st
101- FIL_IBD_FILE_INITIAL_SIZE * UNIV_PAGE_SIZE
102- bytes of any tablespace, regardless of the LSNs
103- of the pages there. These pages are guaranteed
104- to be flushed upon tablespace create and they
105- are required for InnoDB to recognize the
106- tablespace. */
107+ /* This condition may be OK for header, ibuf
108+ and fsp. */
109 if (ut_dulint_cmp(incremental_lsn,
110- MACH_READ_64(page + chunk_offset + FIL_PAGE_LSN)) < 0
111- || (offset + chunk_offset
112- < FIL_IBD_FILE_INITIAL_SIZE * UNIV_PAGE_SIZE)) {
113+ MACH_READ_64(page + chunk_offset
114+ + FIL_PAGE_LSN)) < 0) {
115 /* ========================================= */
116 IB_INT64 offset_on_page;
117
118@@ -5652,12 +5654,107 @@
119 return TRUE;
120 }
121
122+/****************************************************************//**
123+Create a new tablespace on disk and return the handle to its opened
124+file. Code adopted from fil_create_new_single_table_tablespace with
125+the main difference that only disk file is created without updating
126+the InnoDB in-memory dictionary data structures.
127+
128+@return TRUE on success, FALSE on error. */
129+static
130+ibool
131+xb_delta_create_space_file(
132+/*=======================*/
133+ const char* path, /*!<in: path to tablespace */
134+ ulint space_id, /*!<in: space id */
135+ ulint flags __attribute__((unused)),/*!<in: tablespace
136+ flags */
137+ os_file_t* file) /*!<out: file handle */
138+{
139+ ibool ret;
140+ byte* buf;
141+ byte* page;
142+
143+ *file = xb_file_create_no_error_handling(path, OS_FILE_CREATE,
144+ OS_FILE_READ_WRITE, &ret);
145+ if (!ret) {
146+ msg("xtrabackup: cannot create file %s\n", path);
147+ return ret;
148+ }
149+
150+ ret = os_file_set_size(path, *file,
151+ FIL_IBD_FILE_INITIAL_SIZE * UNIV_PAGE_SIZE, 0);
152+ if (!ret) {
153+ msg("xtrabackup: cannot set size for file %s\n", path);
154+ os_file_close(*file);
155+ os_file_delete(path);
156+ return ret;
157+ }
158+
159+ buf = ut_malloc(3 * UNIV_PAGE_SIZE);
160+ /* Align the memory for file i/o if we might have O_DIRECT set */
161+ page = ut_align(buf, UNIV_PAGE_SIZE);
162+
163+ memset(page, '\0', UNIV_PAGE_SIZE);
164+
165+#ifdef INNODB_VERSION_SHORT
166+ fsp_header_init_fields(page, space_id, flags);
167+ mach_write_to_4(page + FIL_PAGE_ARCH_LOG_NO_OR_SPACE_ID, space_id);
168+
169+ if (!(flags & DICT_TF_ZSSIZE_MASK)) {
170+ buf_flush_init_for_writing(page, NULL, 0);
171+
172+ ret = os_file_write(path, *file, page, 0, 0, UNIV_PAGE_SIZE);
173+ }
174+ else {
175+ page_zip_des_t page_zip;
176+ ulint zip_size;
177+
178+ zip_size = (PAGE_ZIP_MIN_SIZE >> 1)
179+ << ((flags & DICT_TF_ZSSIZE_MASK)
180+ >> DICT_TF_ZSSIZE_SHIFT);
181+ page_zip_set_size(&page_zip, zip_size);
182+ page_zip.data = page + UNIV_PAGE_SIZE;
183+ fprintf(stderr, "zip_size = %lu\n", zip_size);
184+
185+#ifdef UNIV_DEBUG
186+ page_zip.m_start =
187+#endif /* UNIV_DEBUG */
188+ page_zip.m_end = page_zip.m_nonempty =
189+ page_zip.n_blobs = 0;
190+
191+ buf_flush_init_for_writing(page, &page_zip, 0);
192+
193+ ret = os_file_write(path, *file, page_zip.data, 0, 0,
194+ zip_size);
195+ }
196+#else
197+ fsp_header_write_space_id(page, space_id);
198+
199+ buf_flush_init_for_writing(page, ut_dulint_zero, space_id, 0);
200+
201+ ret = os_file_write(path, *file, page, 0, 0, UNIV_PAGE_SIZE);
202+#endif
203+
204+ ut_free(buf);
205+
206+ if (!ret) {
207+ msg("xtrabackup: could not write the first page to %s\n",
208+ path);
209+ os_file_close(*file);
210+ os_file_delete(path);
211+ return ret;
212+ }
213+
214+ return TRUE;
215+}
216+
217 /***********************************************************************
218 Searches for matching tablespace file for given .delta file and space_id
219 in given directory. When matching tablespace found, renames it to match the
220 name of .delta file. If there was a tablespace with matching name and
221 mismatching ID, renames it to xtrabackup_tmp_#ID.ibd. If there was no
222-matching file, creates a placeholder for the new tablespace.
223+matching file, creates a new tablespace.
224 @return file handle of matched or created file */
225 static
226 os_file_t
227@@ -5758,19 +5855,20 @@
228 goto found;
229 }
230
231- /* No matching space found. create the new one. Note that this is not
232- a full-fledged tablespace create, as done by
233- fil_create_new_single_table_tablespace(): the minumum tablespace size
234- is not ensured and the 1st page fields are not set. We rely on backup
235- delta to contain the 1st FIL_IBD_FILE_INITIAL_SIZE * UNIV_PAGE_SIZE
236- to ensure the correct tablespace image. */
237+ /* No matching space found. create the new one. */
238
239 #ifdef INNODB_VERSION_SHORT
240- /* Calculate correct tablespace flags for compressed tablespaces. Do
241- not bother to set all flags correctly, just enough for
242- fil_space_create() to work. The full flags will be restored from the
243- delta later. */
244- if (!zip_size) {
245+ if (!fil_space_create(dest_space_name, space_id, 0, FIL_TABLESPACE)) {
246+#else
247+ if (!fil_space_create(dest_space_name, space_id, FIL_TABLESPACE)) {
248+#endif
249+ msg("xtrabackup: Cannot create tablespace %s\n",
250+ dest_space_name);
251+ goto exit;
252+ }
253+
254+ /* Calculate correct tablespace flags for compressed tablespaces. */
255+ if (!zip_size || zip_size == ULINT_UNDEFINED) {
256 tablespace_flags = 0;
257 }
258 else {
259@@ -5780,28 +5878,14 @@
260 << DICT_TF_ZSSIZE_SHIFT)
261 | DICT_TF_COMPACT
262 | (DICT_TF_FORMAT_ZIP << DICT_TF_FORMAT_SHIFT);
263- }
264- ut_a(dict_table_flags_to_zip_size(tablespace_flags) == zip_size);
265- if (!fil_space_create(dest_space_name, space_id, tablespace_flags,
266- FIL_TABLESPACE)) {
267-#else
268- if (!fil_space_create(dest_space_name, space_id,
269- FIL_TABLESPACE)) {
270+#ifdef INNODB_VERSION_SHORT
271+ ut_a(dict_table_flags_to_zip_size(tablespace_flags)
272+ == zip_size);
273 #endif
274- msg("xtrabackup: Cannot create tablespace %s\n",
275- dest_space_name);
276- goto exit;
277- }
278-
279- file = xb_file_create_no_error_handling(real_name, OS_FILE_CREATE,
280- OS_FILE_READ_WRITE,
281- &ok);
282-
283- if (ok) {
284- *success = TRUE;
285- } else {
286- msg("xtrabackup: Cannot open file %s\n", real_name);
287- }
288+ }
289+
290+ *success = xb_delta_create_space_file(real_name, space_id,
291+ tablespace_flags, &file);
292
293 goto exit;
294
295@@ -5911,8 +5995,7 @@
296 xb_file_set_nocache(src_file, src_path, "OPEN");
297
298 dst_file = xb_delta_open_matching_space(
299- dbname, space_name, info.space_id,
300- info.page_size == UNIV_PAGE_SIZE ? 0 : info.page_size,
301+ dbname, space_name, info.space_id, info.zip_size,
302 dst_path, sizeof(dst_path), &success);
303 if (!success) {
304 msg("xtrabackup: error: cannot open %s\n", dst_path);

Subscribers

People subscribed via source and target branches