Merge lp:~widelands-dev/widelands/appveyor_linking_memory into lp:widelands

[Tino]

With the last update https://www.appveyor.com/updates Appveyor is now using a different virtualization with less memory.
So at the moment our 64bit-debug builds fail due to memory usage on linking.

This branch introduces the "-no-keep-memory" option in cmake for the linking process and activates it on Appveyor.

[SirVer (sirver)]

lgtm. Thanks for keeping an eye on the appveyor stuff!

review: Approve

[Tino (tino79)]

Not work, have to dig deeper...

review: Needs Fixing

[Klaus Halfmann (klaus-halfmann)]

I actually did the same (but less sophisticated) for the Raspi
https://wl.widelands.org/forum/topic/2031/
but with some different flags, mmh.

What can I do to test this? (other then waiting for appveyor)

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[GunChleoc (gunchleoc)]

I am also in favour of making this an option for compile.sh - you can "steal" some code from the --no-webpage option - best rename the short option for that to -w then.

[Tino (tino79)]

@GunChleoc: Will do when it is working.

@Klaus:
I started with your code ;). Was looking for your branch, but it seems deleted, so I took it from the forum.

The flag is the same "-no-keep-memory", the difference is how it is given from the compiler to the linker (-Xlinker vs -Wl), thats platform dependant i think.

But at least on appveyor with MinGW-GCC it is not working, the 64bit-debug-builds still fail:
https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100/job/by40g7rtw2e40d48

For a time (last week) it was working again because I contacted Appveyor and they switched the widelands build back to GCE, but now it seems back to RackSpace again.

I am not sure if this long linking times are perhaps caused by those internal circular dependencies of the widelands code base / sub libs. Even on my system linking a debug widelands.exe takes 5 minutes alone.

[SirVer (sirver)]

> I am not sure if this long linking times are perhaps caused by those internal circular dependencies of the widelands code base / sub libs. Even on my system linking a debug widelands.exe takes 5 minutes alone.

Yes, that is the main reason. Basically when you have a cyclic dependency, each library has to appear multiple times on the linking command line and the linker has to keep all symbols of everything in memory. That brings up linking time by a lot. Linking is much quicker though if you build a release binary (less symbols) - one of the reasons I rarely build in debug mode.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

The read operation timed out

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

The read operation timed out

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

''

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

The read operation timed out

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

Running 'bzr pull --overwrite' failed. Output:

ssh_exchange_identification: Connection closed by remote host
ConnectionReset reading response for 'BzrDir.open_2.1', retrying
ssh_exchange_identification: Connection closed by remote host
bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~widelands-dev/widelands/appveyor_linking_memory/

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

Running 'bzr pull --overwrite' failed. Output:

ssh_exchange_identification: Connection closed by remote host
ConnectionReset reading response for 'BzrDir.open_2.1', retrying
ssh_exchange_identification: Connection closed by remote host
bzr: ERROR: Connection closed: Unexpected end of message. Please check connectivity and permissions, and report a bug if problems persist.
Using saved parent location: bzr+ssh://bazaar.launchpad.net/~widelands-dev/widelands/appveyor_linking_memory/

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

The read operation timed out

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

The read operation timed out

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

The read operation timed out

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[bunnybot (widelandsofficial)]

Bunnybot encountered an error while working on this merge proposal:

The read operation timed out

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 1258. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/150651096.
Appveyor build 1100. State: failed. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_appveyor_linking_memory-1100.

[GunChleoc (gunchleoc)]

I have started working on a set of branches to reduce cyclic dependency. The goal is not to have logic and economy depend on any UI stuff. I hope that that will be sufficient savings on linking time to solve the problem.

[GunChleoc (gunchleoc)]

Setting this to WiP to give Bunnybot a break.

=== modified file 'CMakeLists.txt'
--- CMakeLists.txt	2016-08-05 12:32:08 +0000
+++ CMakeLists.txt	2016-08-08 14:28:10 +0000
@@ -7,6 +7,7 @@
 option(OPTION_USE_GLBINDING "Use glbinding instead of GLEW" OFF)
 option(OPTION_GLEW_STATIC "Use static GLEW Library" OFF)
 option(OPTION_BUILD_WEBSITE_TOOLS "Build website-related tools" ON)
+option(OPTION_LINKING_REDUCE_MEMORY "Reduce memory consumption on linking" OFF)
 
 if (CMAKE_SOURCE_DIR STREQUAL CMAKE_BINARY_DIR)
   message(FATAL_ERROR "Build directory and source directory must not be the same.")
@@ -184,13 +185,20 @@
 IF (WIN32)
   add_definitions(-DMINGW_HAS_SECURE_API)
   if (CMAKE_SIZEOF_VOID_P EQUAL 4)
-    set (CMAKE_EXE_LINKER_FLAGS "-Wl,--large-address-aware" CACHE STRING "Set by widelands CMakeLists.txt" FORCE)
+    set (CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,--large-address-aware" CACHE STRING "Set by widelands CMakeLists.txt" FORCE)
     message (STATUS "Enabled large address awareness on mingw32")
   else (CMAKE_SIZEOF_VOID_P EQUAL 4)
     message (STATUS "Detected mingw32-w64")
   endif (CMAKE_SIZEOF_VOID_P EQUAL 4)
 endif (WIN32)
 
+if (OPTION_LINKING_REDUCE_MEMORY)
+    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -Wl,-no-keep-memory")
+    #set(CMAKE_STATIC_LINKER_FLAGS "${CMAKE_STATIC_LINKER_FLAGS} -Wl,-no-keep-memory")
+    set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -Wl,-no-keep-memory")
+    message (STATUS "Enabled -no-keep-memory for linker")
+endif()
+
 # on BSD this must be explicitly linked
 if (CMAKE_SYSTEM_NAME MATCHES "FreeBSD" OR CMAKE_SYSTEM_NAME MATCHES "OpenBSD")
   # Not needed on Debian GNU/kFreeBSD..

=== modified file 'appveyor.yml'
--- appveyor.yml	2016-05-04 17:23:16 +0000
+++ appveyor.yml	2016-08-08 14:28:10 +0000
@@ -32,7 +32,7 @@
   - cmd: md build
   - cmd: cd build
   - cmd: echo %APPVEYOR_BUILD_VERSION%_%CONFIGURATION%_%PLATFORM% > %APPVEYOR_BUILD_FOLDER%\WL_RELEASE
-  - cmd: "cmake -G \"Ninja\" -DCMAKE_BUILD_TYPE=%CONFIGURATION% -DOPTION_USE_GLBINDING=ON %APPVEYOR_BUILD_FOLDER%"
+  - cmd: "cmake -G \"Ninja\" -DCMAKE_BUILD_TYPE=%CONFIGURATION% -DOPTION_USE_GLBINDING=ON -DOPTION_LINKING_REDUCE_MEMORY=ON %APPVEYOR_BUILD_FOLDER%"
   - cmd: ninja
 
 on_success: