Merge lp:~widelands-dev/widelands/travis-osx into lp:widelands

[Hans Joachim Desserud]

Let travis build each commit on macos in addition to our linux builds.

Note that the regression test suite is only run for linux (couldn't get it to work on macos), though everything else builds and works as expected.

Ready for feedback, probably not ready for merge.

As mentioned I came across the travis documentation for running on osx [1][2][3] on travis.

So far, I've set added two simple build to run a debug and release build on osx [4]. This is mostly working, except for the regression test suite [5], which is where I need some help.

I stronly suspect it needs an equivalent of xvfb which we use to present the graphic on Linux, though I don't know what that would be in the osx world. It could be xvfb installed through brew or it could be something completely different. Would appreciate input here. :) If it turns out to be really tricky, we could keep the regression tests for linux only and figure out how to deal with them later. Presumably, verifying that things are buildable would be better than no checks at all.

The second thing is compilers, where I don't know what the situation is like in osx-land. We have a wide array of compilers tested for linux where we ensure that WL is buildable by both latest and newest and oldest supported versions of GCC and Clang. For osx, I've gone the default compiler. It seemed like brew could install alternative ones, but I don't know how many would be needed/useful. (Also want to avoid too many builds in total). I added one debug and release build initially, which seemed like a good place to start.

Somewhat related, when starting out, I tried to follow the osx-section fot he build page[6]. This turned out to be outdated and after "converting" things to brew as I went along, I finally discovered the separate osx build page [7]. I am not sure why this is completely separate from the rest of the build instructions, nor why there are two set of instructions.
Since the former seems rather outdated, I wonder if the best would be to
move the standalone page in as part of the overall build page and keep it up to date there.

This is bound to have conflicts with the other travis MP I have going. I knew that from the get-go, but they do different things, so I didn't want to mix them too much. I can sort out the conflicts when one of them is landed. :) I suspect the other one might be first.

Just a quick headsup: I might not be able to respond to comments here the next couple of days.

[1] https://docs.travis-ci.com/user/osx-ci-environment/
[2] https://docs.travis-ci.com/user/multi-os/
[3] And yes, I know it's called macOS these days :p
[4] https://travis-ci.org/widelands/widelands/builds/254399849
[5] https://travis-ci.org/widelands/widelands/jobs/254399850
[6] https://wl.widelands.org/wiki/BuildingWidelands/#building-under-mac-os-x
[7] https://wl.widelands.org/wiki/BuildingWidelandsMac/

-- Original description
Strictly WIP at the moment, but I believe I need a merge proposal for bunnybot to pick it up and run the builds? (Also is there a way to re-trigger builds manually? Might come in handy for this)

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 2457. State: failed. Details: https://travis-ci.org/widelands/widelands/builds/254255034.
Appveyor build 2282. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_travis_osx-2282.

[Hans Joachim Desserud (hjd)]

Ready for feedback (probably not merge yet)

[GunChleoc (gunchleoc)]

Thanks for starting this!

regarding the Wikipages, the separate page is for homebrew only. I have no idea how current the information for the other 2 methods is. I think that we should update the description to homebrew, or add the information to the main page and delete the separate page.

[Hans Joachim Desserud (hjd)]

> I have no idea how current the information for the other 2 methods is.

The mac-ports one talks about installing sdl 1 dependencies, so that's somewhat outdated. That is possibly fairly easy to update with newer package names.

The other one which talks about getting all dependencies from a bzr branch seems to have been added in 2010 (https://code.launchpad.net/~wsk/widelands/OSX-libs) and not updated since. It seems to mostly concern building via Xcode, and I am not sure if one can convince Xcode to simply use the dependencies installed by brew or ports instead...

It doesn't seem like either has been used for quite some time though :/

[GunChleoc (gunchleoc)]

Where did you find that link?

I have updated https://wl.widelands.org/wiki/Building%20Widelands/#building-under-mac-os-x as well as I can.

[Hans Joachim Desserud (hjd)]

> Where did you find that link?

"Translated" the command bzr `branch lp:~wsk/widelands/OSX-libs` to the corresponding branch.

As a side note, it doesn't seem to be shown on https://code.launchpad.net/widelands, which I thought all non-merged branches would be regardless of age. It has develompent status too, so it shouldn't be that filtering it out.

[Hans Joachim Desserud (hjd)]

>travis-retry sudo apt-get install -qq g++-$GCC_VERSION;

>etc, so that we can at least ensure that we can get the compilers?

That's a good idea :) I've added travis_retry for the compiler installation and package installation on macos. That should give us better control over the parts we install directly ourselves at least.

I found a way which (hopefully) starts xvfb on macos inspired by https://github.com/travis-ci/travis-ci/issues/7313. One of the last comments mentions that while the reporter got things running, it still didn't seem to have opengl support. Let's see if it is a step in the right direction.

[GunChleoc (gunchleoc)]

Sgrìobh Hans Joachim Desserud na leanas 23/07/2017 aig 11:55:
>> Where did you find that link?
>
> "Translated" the command bzr `branch lp:~wsk/widelands/OSX-libs` to the corresponding branch.
>
> As a side note, it doesn't seem to be shown on https://code.launchpad.net/widelands, which I thought all non-merged branches would be regardless of age. It has develompent status too, so it shouldn't be that filtering it out.

I overlooked that.

This branch doesn't appear on the widelands page, because it is not a
branch of the Widelands code, and also not owned by widelands-dev.

I have added a note to the wiki that the libs are outdated.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 2474. State: errored. Details: https://travis-ci.org/widelands/widelands/builds/257196795.
Appveyor build 2299. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_travis_osx-2299.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 2480. State: failed. Details: https://travis-ci.org/widelands/widelands/builds/257496284.
Appveyor build 2305. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_travis_osx-2305.

[Hans Joachim Desserud (hjd)]

Still can't get the regression tests to run on macos :(

Most of them fail with:
Your OpenGL installation must be __very__ broken. Missing GL version

libc++abi.dylib: terminating with uncaught exception of type WException: [/Users/travis/build/widelands/widelands/src/graphic/gl/initialize.cc:90] glewInit returns 1: Broken OpenGL installation.

or similar error messages. See https://travis-ci.org/widelands/widelands/jobs/258520338 for full log.

I've tried to follow instructions on https://github.com/travis-ci/travis-ci/issues/7313 to enable xvfb, which seems to be working (though I'm not 100% sure how to properly verify). It looks like this has some problems with (or possibly not) opengl support.

I'm kind of in the dark now, since I don't have real hardware to test on and I'm not familiar enough with the system to know exactly what to expect.

I'll add a comment on the bug report as well on the current situation. I'm pondering cleaning up this MP (remove some of the debugging) and only run the regression tests on linux for the time being. While I would like to include the regression tests as well, I'm not quite sure how to proceed, and being able to see that each commit is buildable on macos should bring us a step in the right direction.

[GunChleoc (gunchleoc)]

Agreed - let's get this in and fix the regression test later.

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 2491. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/259451618.
Appveyor build 2315. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_travis_osx-2315.

[Hans Joachim Desserud (hjd)]

Cleaned up and ready to go (even re-fixed indentation which had been bothering me)

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 2497. State: errored. Details: https://travis-ci.org/widelands/widelands/builds/259848338.
Appveyor build 2321. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_travis_osx-2321.

[SirVer (sirver)]

I think building on Mac OS is sufficient, the logic should be unchanged. And I also do not think that we require a ton of compilers there - one build should be sufficient. We test all the compilers on Linux and they should be compatible across OSes.

The test suite would be nice, but is run by me on OS X for each daily I build, so it is not super critical either. I think this can go as is.

Thanks for your work, as usual :).

However, the .travis.yml file seems to be broken right now (see last bunnybot comment), so this needs to get fixed first.

review: Needs Fixing

[Hans Joachim Desserud (hjd)]

>I think this can go as is.

:)

>However, the .travis.yml file seems to be broken right now (see last bunnybot comment), so this needs to get fixed first.

Oh, I missed that. Found the misaligned indentation, so the next build should be happier.

[GunChleoc (gunchleoc)]

Travis is green now, so this can go in.

@bunnybot merge

review: Approve

[bunnybot (widelandsofficial)]

Continuous integration builds have changed state:

Travis build 2521. State: passed. Details: https://travis-ci.org/widelands/widelands/builds/263553747.
Appveyor build 2345. State: success. Details: https://ci.appveyor.com/project/widelands-dev/widelands/build/_widelands_dev_widelands_travis_osx-2345.

=== modified file '.travis.sh'
--- .travis.sh	2017-07-17 11:05:42 +0000
+++ .travis.sh	2017-08-11 15:21:11 +0000
@@ -1,12 +1,25 @@
 set -ex
 
-if [ "$CXX" = "g++" ]; then
-  sudo apt-get install -qq g++-$GCC_VERSION;
-  export CXX="g++-$GCC_VERSION" CC="gcc-$GCC_VERSION";
+if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
+  #Install requested compiler version for linux
+
+  if [ "$CXX" = "g++" ]; then
+    sudo apt-get install -qq g++-$GCC_VERSION;
+    export CXX="g++-$GCC_VERSION" CC="gcc-$GCC_VERSION";
+  fi
+  if [ "$CXX" = "clang++" ]; then
+    sudo apt-get install -qq clang-$CLANG_VERSION;
+    export CXX="clang++-$CLANG_VERSION" CC="clang-$CLANG_VERSION";
+  fi
 fi
-if [ "$CXX" = "clang++" ]; then
-  sudo apt-get install -qq clang-$CLANG_VERSION;
-  export CXX="clang++-$CLANG_VERSION" CC="clang-$CLANG_VERSION";
+
+if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
+  # Install osx dependencies
+  # boost and cmake are preinstalled :)
+  brew install gettext glew icu4c sdl2 sdl2_image sdl2_mixer sdl2_ttf zlib
+  # brew doesn't add a link by default
+  brew link --force gettext
+  brew link --force icu4c
 fi
 
 # Configure the build
@@ -41,6 +54,8 @@
 # Do the actual build.
 make -k -j3
 
-# Run the regression suite.
-cd ..
-./regression_test.py -b build/src/widelands
+if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then
+  # Run the regression suite. Haven't gotten it working on osx, due to problems with xvfb and/or opengl support.
+  cd ..
+  ./regression_test.py -b build/src/widelands
+fi

=== modified file '.travis.yml'
--- .travis.yml	2017-07-16 21:14:17 +0000
+++ .travis.yml	2017-08-11 15:21:11 +0000
@@ -5,7 +5,8 @@
 
 before_script:
   - export DISPLAY=:99.0
-  - sh -e /etc/init.d/xvfb start
+  - if [[ "$TRAVIS_OS_NAME" == "linux" ]]; then sh -e /etc/init.d/xvfb start ; fi
+  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then ( sudo Xvfb :99 -ac -screen 0 1024x768x8; echo ok )& fi
   - sleep 3 # give xvfb some time to start
 
 # ignore this branch because it is only a mirror of master
@@ -44,35 +45,58 @@
 
 matrix:
   include:
-    - compiler: clang
+    ### MAC OSX BUILDS
+    - os: osx
+      env: BUILD_TYPE="Debug"
+    - os: osx
+      env: BUILD_TYPE="Release"
+    ### LINUX + CLANG BUILDS
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="3.4" BUILD_TYPE="Debug"
-    - compiler: clang
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="3.5" BUILD_TYPE="Debug"
-    - compiler: clang
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="3.9" BUILD_TYPE="Debug"
-    - compiler: clang
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="4.0" BUILD_TYPE="Debug"
-    - compiler: clang
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="3.4" BUILD_TYPE="Release"
-    - compiler: clang
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="3.5" BUILD_TYPE="Release"
-    - compiler: clang
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="3.9" BUILD_TYPE="Release"
-    - compiler: clang
+    - os: linux
+      compiler: clang
       env: CLANG_VERSION="4.0" BUILD_TYPE="Release"
-    - compiler: gcc
+    ### LINUX + GCC BUILDS
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="4.8" BUILD_TYPE="Debug"
-    - compiler: gcc
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="4.9" BUILD_TYPE="Debug"
-    - compiler: gcc
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="6" BUILD_TYPE="Debug"
-    - compiler: gcc
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="7" BUILD_TYPE="Debug"
-    - compiler: gcc
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="4.8" BUILD_TYPE="Release"
-    - compiler: gcc
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="4.9" BUILD_TYPE="Release"
-    - compiler: gcc
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="6" BUILD_TYPE="Release"
-    - compiler: gcc
+    - os: linux
+      compiler: gcc
       env: GCC_VERSION="7" BUILD_TYPE="Release"