Merge lp:~widelands-dev/widelands/campaign_data into lp:widelands

[Benedikt Straub]

Adds two Lua functions save_campaign_data() and read_campaign_data() as functions of EditorGameBase.
These functions allow a campaign to store information in a file at the end of one scenario, and read it again in another mission.
Campaign data is stored in ".widelands/campaigns/campaign_name/scenario_name.wcd".

A script can call these functions like this:
  wl.Game().save_campaign_data("frisians", "fri03", "Hello World")
  local text = wl.Game().read_campaign_data("frisians", "fri03") --> "Hello World"
  local text = wl.Game().read_campaign_data("non-existant", "xyz") --> nil

[GunChleoc (gunchleoc)]

I think this would benefit greatly from having proper data structures rather than using a string

https://wl.widelands.org/forum/topic/4228/?page=2#post-24594

Let me know if you need any help with the Lua tables in the C++ backend - they are a bit tricky and I usually find a similar function and copy/paste myself.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3403. State: failed. Details: https://travis-ci.org/widelands/widelands/builds/369988831.
Appveyor build 3209. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3209.

[hessenfarmer (stephan-lutz)]

Thinking as scenario designer I would prefer to have the string solution. Because if there is a fixed format and I Need just one single value in the next Scenario I would have to write a lot of Zeros to the file, additionally there is more information in a Scenario than is contained in the lua table.
For example you could take the decisions contained in empire 4 (How to deal with the monastry) or fri02 (peace Option or war Option) you could easily use these decisions in follow on scenarios as well and they are just local variables.

[Benedikt Straub (nordfriese)]

I could try to implement the table solution without fixed formats. So you pass a table to the save function, but it is up to you what keys to specify, and the .wcd file will contain only the keys you chose to use, with the values you assigned to them.
For example, you could call
  local decision = ...
  wl.Game().save_campaign_data("empiretut", "emp04", {
    conquered_monastry = decision
  })
and then you´d get in the next scenario
  local cdtable = wl.Game().read_campaign_data("empiretut", "emp04")
  --> value of cdtable is {conquered_monastry=true}
  local decision = cdtable.conquered_monastry
So you´d save and get only the information you really want.

[GunChleoc (gunchleoc)]

> I would have to write a lot of Zeros to the file

Why do you think that? If you want 1 value, write 1 value.

> For example you could take the decisions contained in empire 4 (How to deal with the monastry) or fri02 (peace Option or war Option) you could easily use these decisions in follow on scenarios as well and they are just local variables.

We can have a "general" section here like Nordfriese suggested, with as many string_key = string_value pairs as you want. Or maybe even better, a section "booleans" or "decisions" for decisions, and section "strings" for general strings? At the moment, I don't wee he need for strings though.

Using just 1 long string that you then need to parse in Lua sounds pretty hacky to me. Since this is a new design, we should take the time to design it properly. This way, we won't have to change it later and write compatibility code or break savegame compatibility. It will take longer now, but experience with this project has shown me that we will benefit greatly from a proper data structure.

[hessenfarmer (stephan-lutz)]

Ok now I understand the thing better.
Of course Organising the file in different tables containing variables by their type, would make sense.
However as I can't imagine yet what things could be useful to save in the future, the Format should be able to cope with this. So it should provide some flexibility as well. Probably this will be given if there would be a table for every Kind of variable available.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3411. State: failed. Details: https://travis-ci.org/widelands/widelands/builds/371025848.
Appveyor build 3217. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3217.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3413. State: failed. Details: https://travis-ci.org/widelands/widelands/builds/371121421.
Appveyor build 3219. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3219.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3415. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/371521832.
Appveyor build 3221. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3221.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3439. State: errored. Details: https://travis-ci.org/widelands/widelands/builds/373880326.
Appveyor build 3244. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3244.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3526. State: errored. Details: https://travis-ci.org/widelands/widelands/builds/380097569.
Appveyor build 3331. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3331.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3527. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/380202280.
Appveyor build 3332. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3332.

[Klaus Halfmann (klaus-halfmann)]

Looks we lost trac on this one.

But as it should not break anything on the road
to R20 we can include it and start using in with R21.

If I play a scenario twice with different outcome,
the data will still be inside the different save-files, ok?

One comment inline.

If this compiles and the test pass I will tell bunnybot to merge.

review: Approve

[Klaus Halfmann (klaus-halfmann)]

Oops this crashes on OSX when running the regression tests:

test/maps/plain.wmf/scripting/test_campaign_data.lua ...
  Running Widelands ... FAIL

READ of size 3 at 0x7ffeeb29c1c1 thread T0
    #0 0x109307c34 in wrap_strlen (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x15c34)
    #1 0x1049e3d04 in boost::range_detail::length(char const*) as_literal.hpp:38
    #2 0x1049e3bf9 in boost::iterator_range<char const*> boost::range_detail::make_range<char const>(char const*, bool) as_literal.hpp:86
    #3 0x106e5794e in boost::iterator_range<boost::range_iterator<char const* const, void>::type> boost::as_literal<char const*>(char const* const&) as_literal.hpp:109
    #4 0x106e58028 in bool boost::algorithm::equals<char const*, char const*, boost::algorithm::is_iequal>(char const* const&, char const* const&, boost::algorithm::is_iequal) predicate.hpp:290
    #5 0x106e49ab6 in bool boost::algorithm::iequals<char const*, char const*>(char const* const&, char const* const&, std::__1::locale const&) predicate.hpp:346
    #6 0x106e496da in Section::has_val(char const*) const profile.cc:216
    #7 0x106a5a3d2 in LuaBases::push_table_recursively(lua_State*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, Section*, Section*, Section*, Section*) lua_bases.cc:431
    #8 0x106a56d06 in LuaBases::LuaEditorGameBase::read_campaign_data(lua_State*) lua_bases.cc:488
    #9 0x106bf85ca in int method_dispatch<LuaRoot::LuaGame, LuaBases::LuaEditorGameBase>(lua_State*) luna_impl.h:175
..

  This frame has 7 object(s):
    [32, 56) 'ref.tmp' (line 429) <== Memory access at offset 33 is inside this variable
    [96, 120) 'ref.tmp2' (line 429)
    [160, 184) 'ref.tmp3' (line 429)
...

SUMMARY: AddressSanitizer: stack-use-after-scope (libclang_rt.asan_osx_dynamic.dylib:x86_64h+0x15c34) in wrap_strlen

Benedikt, can you confirm / fix it with this information?
Otherwise I will have to setup a longer debugger session.

review: Needs Fixing (regresson test)

[GunChleoc (gunchleoc)]

I get an error too, from ASAN. There is some illegal data passed to Profile by that line.

=================================================================
==10374==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffe6d2c9300 at pc 0x7f9ba515b66e bp 0x7ffe6d2c8d80 sp 0x7ffe6d2c8528
READ of size 3 at 0x7ffe6d2c9300 thread T0
    #0 0x7f9ba515b66d (/usr/lib/x86_64-linux-gnu/libasan.so.4+0x5166d)
    #1 0x55b57ce0c51d in boost::range_detail::length(char const*) /usr/include/boost/range/as_literal.hpp:38
    #2 0x55b57ce2093e in boost::iterator_range<char const*> boost::range_detail::make_range<char const>(char const*, bool) /usr/include/boost/range/as_literal.hpp:86
    #3 0x55b57cee0d81 in boost::iterator_range<boost::range_iterator<char const* const, void>::type> boost::as_literal<char const*>(char const* const&) /usr/include/boost/range/as_literal.hpp:109
    #4 0x55b57dd78425 in bool boost::algorithm::equals<char const*, char const*, boost::algorithm::is_iequal>(char const* const&, char const* const&, boost::algorithm::is_iequal) /usr/include/boost/algorithm/string/predicate.hpp:290
    #5 0x55b57dd77345 in bool boost::algorithm::iequals<char const*, char const*>(char const* const&, char const* const&, std::locale const&) /usr/include/boost/algorithm/string/predicate.hpp:346
    #6 0x55b57dd71a84 in Section::has_val(char const*) const src/profile/profile.cc:216
    #7 0x55b57db5209d in LuaBases::push_table_recursively(lua_State*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, Section*, Section*, Section*, Section*) src/scripting/lua_bases.cc:431
    #8 0x55b57db5330f in LuaBases::LuaEditorGameBase::read_campaign_data(lua_State*) src/scripting/lua_bases.cc:488

[Benedikt Straub (nordfriese)]

Diff comment: I don´t think a recursion depth limit is needed, as the documentation explicitly forbids cyclic dependencies. It would be easy to implement though – what would be a sensible limit here?

> If I play a scenario twice with different outcome,
> the data will still be inside the different save-files, ok?

Every time you play the scenario, the old .wcd file is overwritten. So, the read method always returns the last data that was saved.

Regarding the ASAN error – I have no idea why this happens. It puzzles me that it works correctly despite ASAN´s complaint. Any ideas how to fix this?

[GunChleoc (gunchleoc)]

I would surround the offending line with some log outputs to try to find out which value is causing this.

[Klaus Halfmann (klaus-halfmann)]

Trying to debug this is not easy, I still do not understand that code:

I run:
> ./regression_test.py -b ./widelands -n -k -r test_campaign_data
I checked the outout in
> /var/folders/bx/q32c1w1965g5clyfbfdykcb40000gp/T/widelands_regression_test/WidelandsTestCase/stdout_00.txt
I had to tweak the file to replace \n by real newline to make it readable.

'b'Trying to run: map:scripting/init.lua: Forcing flag at (10, 11)
'b'Forcing flag at (23, 26)
'b'Message: adding warehouse for player 1 at (22, 25)
'b'done
'b'Trying to run: test/maps/plain.wmf/scripting/test_campaign_data.lua: done
'b'Writing campaign data...
'b'Done.
'b'Reading campaign data...
'b'=================================================================
'b'==17300==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffee08c4261 at pc 0x000113d4bc35 bp 0x7ffee08c3990 sp 0x7ffee08c3138
'b'READ of size 3 at 0x7ffee08c4261 thread T0
...
'b' #6 0x11184300a in Section::has_val(char const*) const profile.cc:216
'b' #7 0x11141d4e2 in LuaBases::push_table_recursively(lua_State*, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, Section*, Section*, Section*, Section*) lua_bases.cc:431
'b' #8 0x111419e16 in LuaBases::LuaEditorGameBase::read_campaign_data(lua_State*) lua_bases.cc:488

So that code reads some variable from a stack that has already been freed.

Benedikt:
* Please add more comments to push_table_recursively(). What is pushed from where?
* I could not locate the file campaigns/test_suite_campaign_data/test_suite_campaign_data.wcd
  where should I find this?

I assume profile.get_section("xxxx") returns a pointer to a local variable,
which is always a bad idea. I found this function with more then one section
at a time is used only by your code. I had expected that the compiler will give
a warning, but Clang does not?

Can anybody give me some more hints how this Profile class should be used?

[Klaus Halfmann (klaus-halfmann)]

Mhh, that Profile/Section concept is used more often then I first found it,
but its not broken. I think I now undestood the idea how you want to make
the recursive table into a flat .ini like file.

What ist the best way to make a quick and dirty test for this particular function?

[Klaus Halfmann (klaus-halfmann)]

This is _not_ UseAfterReturn as I assumed before but UseAfterScope:
Here the examples from the ASAN Docs:
https://github.com/google/sanitizers/wiki/AddressSanitizerExampleUseAfterScope

I did not dind a direkt scopes with brakets { .... } but an expression may be a scope, too.

What about this one?
    const char* key_key = (depth + "_" + std::to_string(i)).c_str();

http://www.cplusplus.com/reference/string/string/c_str/ states:
The pointer returned may be invalidated by further calls to
other member functions that modify the object.

Bu the temporary std:string you build here, is destructed right after that call,
as it goes out of scope. Will try o fix this now ... done.

Benedikt/Gun pleas take a look at my commit, (give me a few minutes)

[Benedikt Straub (nordfriese)]

I added some comments and logs.
Normally, the campaign data file should be in "~/.widelands/campaigns/campaign_name/scenario_name.wcd".
I believe the error may be caused by Section::has_val(). The Profile class is used in many places but the has_val() function was unused until now.
I´m unable to run with ASAN for some reason, it compiles fine but I get this error on startup:
==25087==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
Can you please trigger the error again with the additional logs and provide the output, and possibly the contents of the .wcd file (if it exists)?

The easiest way to test is to put the following code in the scripting/init.lua for any map, and load it as a singleplayer scenario:

include "scripting/coroutine.lua"
function thread()
game = wl.Game ()
save()
read()
end
function save()
print("Starting to WRITE...")
game:save_campaign_data("test_campaign", "test_scenario", {
   x = 5,
   y = "Hello",
   z = {
      nil,
      10,
      12,
      14,
      16,
      "A",
      "B",
      "C",
      {
         a = 1,
         b = 2,
      },
      "D",
   }
})
print("Done.")
end
function print_table(t, depth)
   print(string.rep(" ", depth - 1) .. "#=" .. #t)
   for k,v in pairs(t) do
      local string = string.rep(" ", depth) .. k .. "="
      if v == nil then
         print ( string .. "nil" )
      elseif type(v) == "table" then
         print ( string .. "{" )
         print_table(v, depth + 2)
         print (string.rep(" ", depth) .. "}")
      elseif type(v) == "boolean" then
         if v then print ( string .. "true" ) else print ( string .. "false" ) end
      else
         print ( string .. v )
      end
   end
end
function read()
print("Starting to READ...")
local result = game:read_campaign_data("test_campaign", "test_scenario")
print("Returned:")
if result == nil then
   print (" <nil>")
else
   print_table(result, 2)
end
print("Done.")
end
run(thread)

This saves a campaign data file, then reads it and prints the result to stdout. You can then compare the init.lua, the wcd file and the output. Without ASAN, this works fine for me.

[Klaus Halfmann (klaus-halfmann)]

Commited, feel free to merge

review: Approve (compile, testrun)

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3557. State: failed. Details: https://travis-ci.org/widelands/widelands/builds/384390395.
Appveyor build 3361. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3361.

[GunChleoc (gunchleoc)]

We still have a codecheck error:

src/scripting/lua_bases.cc:461: Trailing whitespace at end of line

> I had to tweak the file to replace \n by real newline to make it readable.

That's because you're on Windows and your editor can't handle Unix line endings. I recommend using Notepad++ or Geany.

[Klaus Halfmann (klaus-halfmann)]

Fixed that codecheck error and compiler warnings about thos table_recursive functions.

review: Needs Resubmitting

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3560. State: errored. Details: https://travis-ci.org/widelands/widelands/builds/385284758.
Appveyor build 3363. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3363.

[Klaus Halfmann (klaus-halfmann)]

Looks trike travis had a _lot_ of Problems with http://apt.llvm.org and other apt soruces.

Gun: can this still go in?

[GunChleoc (gunchleoc)]

We didn't get an debug builds on GCC out of this one, so best merge trunk to trigger Travis again.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 3566. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/386331581.
Appveyor build 3369. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_campaign_data-3369.

[Klaus Halfmann (klaus-halfmann)]

@bunnybot merge

finally....

review: Approve

=== modified file 'src/logic/filesystem_constants.h'
--- src/logic/filesystem_constants.h	2018-04-07 16:59:00 +0000
+++ src/logic/filesystem_constants.h	2018-05-31 18:15:43 +0000
@@ -54,6 +54,10 @@
 // Default autosave interval in minutes
 constexpr int kDefaultAutosaveInterval = 15;
 
+// Filesystem names for campaign data
+const std::string kCampaignDataDir = "campaigns";
+const std::string kCampaignDataExtension = ".wcd";
+
 /// Filesystem names for screenshots
 const std::string kScreenshotsDir = "screenshots";
 

=== modified file 'src/scripting/CMakeLists.txt'
--- src/scripting/CMakeLists.txt	2018-02-13 10:14:35 +0000
+++ src/scripting/CMakeLists.txt	2018-05-31 18:15:43 +0000
@@ -109,11 +109,13 @@
     logic
     logic_campaign_visibility
     logic_constants
+    logic_filesystem_constants
     logic_game_controller
     logic_game_settings
     logic_tribe_basic_info
     logic_widelands_geometry
     map_io
+    profile
     scripting_base
     scripting_coroutine
     scripting_errors

=== modified file 'src/scripting/lua_bases.cc'
--- src/scripting/lua_bases.cc	2018-04-07 16:59:00 +0000
+++ src/scripting/lua_bases.cc	2018-05-31 18:15:43 +0000
@@ -19,15 +19,19 @@
 
 #include "scripting/lua_bases.h"
 
+#include <boost/algorithm/string.hpp>
 #include <boost/format.hpp>
 
 #include "economy/economy.h"
+#include "io/filesystem/layered_filesystem.h"
+#include "logic/filesystem_constants.h"
 #include "logic/map_objects/checkstep.h"
 #include "logic/map_objects/tribes/tribe_descr.h"
 #include "logic/map_objects/tribes/tribes.h"
 #include "logic/map_objects/tribes/ware_descr.h"
 #include "logic/map_objects/world/world.h"
 #include "logic/player.h"
+#include "profile/profile.h"
 #include "scripting/factory.h"
 #include "scripting/globals.h"
 #include "scripting/lua_map.h"
@@ -83,6 +87,8 @@
    METHOD(LuaEditorGameBase, get_worker_description),
    METHOD(LuaEditorGameBase, get_resource_description),
    METHOD(LuaEditorGameBase, get_terrain_description),
+   METHOD(LuaEditorGameBase, save_campaign_data),
+   METHOD(LuaEditorGameBase, read_campaign_data),
    {nullptr, nullptr},
 };
 const PropertyType<LuaEditorGameBase> LuaEditorGameBase::Properties[] = {
@@ -319,6 +325,212 @@
 	return to_lua<LuaMaps::LuaTerrainDescription>(L, new LuaMaps::LuaTerrainDescription(descr));
 }
 
+/* Helper function for save_campaign_data()
+
+   This function reads the lua table from the stack and saves information about its
+   keys, values and data types and its size to the provided maps.
+   This function is recursive so subtables to any depth can be saved.
+   Each value in the table (including all subtables) is uniquely identified by a key_key.
+   The key_key is used as key in all the map.
+   For the topmost table of size x, the key_keys are called '_0' through '_x-1'.
+   For a subtable of size z at key_key '_y', the subtable's key_keys are called '_y_0' through '_y_z-1'.
+   If a table is an array, the map 'keys' will contain no mappings for the array's key_keys.
+*/
+static void save_table_recursively(lua_State* L, std::string depth, std::map<std::string, const char*> *data,
+std::map<std::string, const char*> *keys, std::map<std::string, const char*> *type, std::map<std::string, uint32_t> *size) {
+	lua_pushnil(L);
+	uint32_t i = 0;
+	while (lua_next(L, -2) != 0) {
+		std::string key_key = depth + "_" + std::to_string(i);
+
+		// check the value's type
+		const char* type_name = lua_typename(L, lua_type(L, -1));
+		std::string t = std::string(type_name);
+
+		(*type)[key_key] = type_name;
+
+		if (t == "number" || t == "string") {
+			// numbers may be treated like strings here
+			(*data)[key_key] = luaL_checkstring(L, -1);
+		} else if (t == "boolean") {
+			(*data)[key_key] = luaL_checkboolean(L, -1) ? "true" : "false";
+		} else if (t == "table") {
+			save_table_recursively(L, depth + "_" + std::to_string(i), data, keys, type, size);
+		} else {
+			report_error(L, "A campaign data value may be a string, integer, boolean, or table; but not a %s!", type_name);
+		}
+
+		++i;
+
+		// put the key on the stack top
+		lua_pop(L, 1);
+		if (lua_type(L, -1) == LUA_TSTRING) {
+			// this is a table
+			(*keys)[key_key] = luaL_checkstring(L, -1);
+		} else if (lua_type(L, -1) == LUA_TNUMBER) {
+			// this is an array
+			if (i != luaL_checkuint32(L, -1)) {
+				// If we get here, the scripter must have set some array values to nil.
+				// This is forbidden because it causes problems when trying to read the data later.
+				report_error(L, "A campaign data array entry must not be nil!");
+			}
+			// otherwise, this is a normal array, so all is well
+		} else {
+			report_error(L, "A campaign data key may be a string or integer; but not a %s!",
+					lua_typename(L, lua_type(L, -1)));
+		}
+	}
+	(*size)[depth] = i;
+}
+
+/* RST
+   .. function:: save_campaign_data(campaign_name, scenario_name, data)
+
+      :arg campaign_name: the name of the current campaign, e.g. "empiretut" or "frisians"
+      :arg scenario_name: the name of the current scenario, e.g. "emp04" or "fri03"
+      :arg data: a table of key-value pairs to save
+
+      Saves information that can be read by other scenarios.
+
+      If an array is used, the data will be saved in the correct order. Arrays may not contain nil values.
+      If the table is not an array, all keys have to be strings.
+      Tables may contain subtables of any depth. Cyclic dependencies will cause Widelands to crash.
+      Only tables/arrays, strings, integer numbers and booleans may be used as values.
+*/
+int LuaEditorGameBase::save_campaign_data(lua_State* L) {
+
+	const std::string campaign_name = luaL_checkstring(L, 2);
+	const std::string scenario_name = luaL_checkstring(L, 3);
+	luaL_checktype(L, 4, LUA_TTABLE);
+
+	std::string dir = kCampaignDataDir + g_fs->file_separator() + campaign_name;
+	boost::trim(dir);
+	g_fs->ensure_directory_exists(dir);
+
+	std::string complete_filename = dir + g_fs->file_separator() + scenario_name + kCampaignDataExtension;
+	boost::trim(complete_filename);
+
+	std::map<std::string, const char*> data;
+	std::map<std::string, const char*> keys;
+	std::map<std::string, const char*> type;
+	std::map<std::string, uint32_t> size;
+
+	save_table_recursively(L, "", &data, &keys, &type, &size);
+
+	Profile profile;
+	Section& data_section = profile.create_section("data");
+	for (const auto &p : data) {
+		data_section.set_string(p.first.c_str(), p.second);
+	}
+	Section& keys_section = profile.create_section("keys");
+	for (const auto &p : keys) {
+		keys_section.set_string(p.first.c_str(), p.second);
+	}
+	Section& type_section = profile.create_section("type");
+	for (const auto &p : type) {
+		type_section.set_string(p.first.c_str(), p.second);
+	}
+	Section& size_section = profile.create_section("size");
+	for (const auto &p : size) {
+		size_section.set_natural(p.first.c_str(), p.second);
+	}
+
+	profile.write(complete_filename.c_str(), false);
+
+	return 0;
+}
+
+/* Helper function for read_campaign_data()
+
+   This function reads the campaign data file and re-creates the table the data was created from.
+   This function is recursive so subtables to any depth can be created.
+   For information on section structure and key_keys, see the comment for save_table_recursively().
+   This function first newly creates the table to write data to, and the number of items in the table is read.
+   For each item, the unique key_key is created. If the 'keys' section doesn't contain an entry for that key_key,
+   it must be because this table is supposed to be an array. Then the data type is checked
+   and the key-value pair is written to the table as the correct type.
+*/
+static void push_table_recursively(lua_State* L, std::string depth, Section* data_section, Section* keys_section,
+Section* type_section, Section* size_section) {
+	const uint32_t size = size_section->get_natural(depth.c_str());
+	lua_newtable(L);
+	for (uint32_t i = 0; i < size; i++) {
+		std::string key_key_str(depth + '_' + std::to_string(i));
+		const char* key_key = key_key_str.c_str();
+
+		log("Checking whether a key for '%s' (data type is %s) exists ... ",
+				key_key, type_section->get_string(key_key)); // NOCOM remove this log
+		if (keys_section->has_val(key_key)) {
+			// this is a table
+			log("YES, the key is called '%s'.\n", keys_section->get_string(key_key)); // NOCOM remove this log
+			lua_pushstring(L, keys_section->get_string(key_key));
+		}
+		else {
+			// this must be an array
+			log("NO, [%i] will be used as key.\n", i + 1); // NOCOM remove this log
+			lua_pushinteger(L, i + 1);
+		}
+
+		// check the data type and push the value
+		const std::string type = type_section->get_string(key_key);
+
+		if (type == "boolean") {
+			lua_pushboolean(L, data_section->get_bool(key_key));
+		} else if (type == "number") {
+			lua_pushinteger(L, data_section->get_int(key_key));
+		} else if (type == "string") {
+			lua_pushstring(L, data_section->get_string(key_key));
+		} else if (type == "table") {
+			// creates a new (sub-)table at the stacktop, populated with its own key-value-pairs
+			push_table_recursively(L, depth + "_" + std::to_string(i),
+					data_section, keys_section, type_section, size_section);
+		} else {
+			// this code should not be reached unless the user manually edited the .wcd file
+			log("Illegal data type %s in campaign data file, setting key %s to nil\n",
+					type.c_str(), luaL_checkstring(L, -1));
+			lua_pushnil(L);
+		}
+		lua_settable(L, -3);
+	}
+}
+
+/* RST
+   .. function:: read_campaign_data(campaign_name, scenario_name)
+
+      :arg campaign_name: the name of the campaign, e.g. "empiretut" or "frisians"
+      :arg scenario_name: the name of the scenario that saved the data, e.g. "emp04" or "fri03"
+
+      Reads information that was saved by another scenario.
+      The data is returned as a table of key-value pairs.
+      The table is not guaranteed to be in any particular order, unless it is an array,
+      in which case it will be returned in the same order as it was saved.
+      This function returns :const:`nil` if the file cannot be opened for reading.
+*/
+int LuaEditorGameBase::read_campaign_data(lua_State* L) {
+	const std::string campaign_name = luaL_checkstring(L, 2);
+	const std::string scenario_name = luaL_checkstring(L, 3);
+
+	std::string complete_filename = kCampaignDataDir + g_fs->file_separator() + campaign_name +
+			g_fs->file_separator() + scenario_name + kCampaignDataExtension;
+	boost::trim(complete_filename);
+
+	Profile profile;
+	profile.read(complete_filename.c_str());
+	Section* data_section = profile.get_section("data");
+	Section* keys_section = profile.get_section("keys");
+	Section* type_section = profile.get_section("type");
+	Section* size_section = profile.get_section("size");
+	if (data_section == nullptr || keys_section == nullptr || type_section == nullptr || size_section == nullptr) {
+		log("Unable to read campaign data file, returning nil\n");
+		lua_pushnil(L);
+	}
+	else {
+		push_table_recursively(L, "", data_section, keys_section, type_section, size_section);
+	}
+
+	return 1;
+}
+
 /*
  ==========================================================
  C METHODS

=== modified file 'src/scripting/lua_bases.h'
--- src/scripting/lua_bases.h	2018-04-07 16:59:00 +0000
+++ src/scripting/lua_bases.h	2018-05-31 18:15:43 +0000
@@ -68,6 +68,8 @@
 	int get_worker_description(lua_State* L);
 	int get_resource_description(lua_State* L);
 	int get_terrain_description(lua_State* L);
+	int save_campaign_data(lua_State* L);
+	int read_campaign_data(lua_State* L);
 
 	/*
 	 * C methods

=== added file 'test/maps/plain.wmf/scripting/test_campaign_data.lua'
--- test/maps/plain.wmf/scripting/test_campaign_data.lua	1970-01-01 00:00:00 +0000
+++ test/maps/plain.wmf/scripting/test_campaign_data.lua	2018-05-31 18:15:43 +0000
@@ -0,0 +1,77 @@
+-- Test writing and reading of campaign data
+
+run(function()
+   sleep(5000)
+
+   local game = wl.Game()
+
+   print("Writing campaign data...")
+   game:save_campaign_data("test_suite_campaign_data", "test_suite_campaign_data", {
+      abc = "Hello",
+      d = 8,
+      efg = "World",
+      h = -999,
+      i = true,
+      jklmno = 0,
+      pq = "Hello",
+      rst = {
+         false,
+         "abc",
+         -1,
+         20,
+         { hello = "World" }
+      },
+      uv = 999,
+      wxyz = false
+   })
+   print("Done.")
+
+   print("Reading campaign data...")
+   local result = game:read_campaign_data("test_suite_campaign_data", "test_suite_campaign_data")
+   assert_equal("string", type(result.abc))
+   assert_equal("Hello", result.abc)
+
+   assert_equal("number", type(result.d))
+   assert_equal(8, result.d)
+
+   assert_equal("string", type(result.efg))
+   assert_equal("World", result.efg)
+
+   assert_equal("number", type(result.h))
+   assert_equal(-999, result.h)
+
+   assert_equal("boolean", type(result.i))
+   assert_equal(true, result.i)
+
+   assert_equal("number", type(result.jklmno))
+   assert_equal(0, result.jklmno)
+
+   assert_equal("string", type(result.pq))
+   assert_equal("Hello", result.pq)
+
+   assert_equal("number", type(result.uv))
+   assert_equal(999, result.uv)
+
+   assert_equal("boolean", type(result.wxyz))
+   assert_equal(false, result.wxyz)
+
+   assert_equal("boolean", type(result.rst[1]))
+   assert_equal(false, result.rst[1])
+
+   assert_equal("string", type(result.rst[2]))
+   assert_equal("abc", result.rst[2])
+
+   assert_equal("number", type(result.rst[3]))
+   assert_equal(-1, result.rst[3])
+
+   assert_equal("number", type(result.rst[4]))
+   assert_equal(20, result.rst[4])
+
+   assert_equal("string", type(result.rst[5].hello))
+   assert_equal("World", result.rst[5].hello)
+
+   print("Done.")
+
+   print("# All Tests passed.")
+   wl.ui.MapView():close()
+end)