Merge ~dirk.zimoch/epics-base:raspberryPi into ~epics-core/epics-base/+git/epics-base:7.0

[Dirk Zimoch]

Configuration file for raspberryPi added.

[mdavidsaver (mdavidsaver)]

Do you have references to upstream bug reports?

I've never encountered problems like this with the hosted compiler.

[mdavidsaver (mdavidsaver)]

I hadn't come across https://github.com/raspberrypi/tools previously. Having seen it, I don't think I would ever use it!

Judging by the (3 year old) commit message, these binaries are from a crosstool-ng built, not the native raspbian. ABI mis-matches with the raspian toolchain are to be expected.

fyi. seeing linking issues involving libtinfo.so is a sign of mixing debian and non-debian builds of readline.

If you use this toolchain then imo. full static linking (including libc) is a must!

If interop with raspbian libs is desirable, then I'd suggest looking at QEMU user mode emulation to run the hosted toolchain.

https://wiki.debian.org/RaspberryPi/qemu-user-static

[Dirk Zimoch (dirk.zimoch)]

> Do you have references to upstream bug reports?
>
> I've never encountered problems like this with the hosted compiler.

What problem? This is no bugfix. I just wanted to share my setup to cross-compile for raspberryPi. Maybe it is useful for others wanting to use EPICS on a pi.

[Dirk Zimoch (dirk.zimoch)]

> I hadn't come across https://github.com/raspberrypi/tools previously. Having
> seen it, I don't think I would ever use it!
>
> Judging by the (3 year old) commit message, these binaries are from a
> crosstool-ng built, not the native raspbian. ABI mis-matches with the raspian
> toolchain are to be expected.

There is a comment about readline in the file.

>
> fyi. seeing linking issues involving libtinfo.so is a sign of mixing debian
> and non-debian builds of readline.
>
> If you use this toolchain then imo. full static linking (including libc) is a
> must!
>
> If interop with raspbian libs is desirable, then I'd suggest looking at QEMU
> user mode emulation to run the hosted toolchain.
>
> https://wiki.debian.org/RaspberryPi/qemu-user-static

This was the "official" cross tool chain, AFAIK.

[Andrew Johnson (anj)]

Is this configuration intended for cross-building? Have you used it for building native as well?

The EPICS build system was designed so that end-users should never have to modify any CONFIG.* files to build Base, they should only edit CONFIG_SITE.* files, so your SDK_TARGET value should get in a CONFIG_SITE.* file. If this configuration is used for cross-building only then it really belongs in a CONFIG_SITE.<host-arch>.raspbian-arm file, the CONFIG.Common.<target-arch> files are for things that really are specific to the target irrespective of where the code is being compiled.

I have asked Pete Jemian who has lots of experience with running EPICS on the Pi for his comments too.

[Andrew Johnson (anj)]

There seem to be a number of possibilities for Raspbian cross-building, the various answers to the stack-overflow question at https://stackoverflow.com/questions/19162072/how-to-install-the-raspberry-pi-cross-compiler-on-my-linux-host-machine show several options. The Github tool-chain that Dirk is using does seem to be the "official" one. A fairly comprehensive alternative set with more recent versions is available at https://github.com/abhiTronix/raspberry-pi-cross-compilers

There is also a post on Medium at https://medium.com/@au42/the-useful-raspberrypi-cross-compile-guide-ea56054de187 that covers resolving library issues using the official tool-chain, but it's configuring application builds using CMAKE and I haven't looked at it in detail.

[Dirk Zimoch (dirk.zimoch)]

> Is this configuration intended for cross-building? Have you used it for
> building native as well?

This is for cross build. Native builds do not need to care for SDK locations. But I like to cross build for all our archs.

> The EPICS build system was designed so that end-users should never have to
> modify any CONFIG.* files to build Base, they should only edit CONFIG_SITE.*
> files, so your SDK_TARGET value should get in a CONFIG_SITE.* file. If this
> configuration is used for cross-building only then it really belongs in a
> CONFIG_SITE.<host-arch>.raspbian-arm file, the CONFIG.Common.<target-arch>
> files are for things that really are specific to the target irrespective of
> where the code is being compiled.

This is simply how I got it working. I‘ll try to separate it into user config and generic settings.

>
> I have asked Pete Jemian who has lots of experience with running EPICS on the
> Pi for his comments too.

[Dirk Zimoch (dirk.zimoch)]

> There seem to be a number of possibilities for Raspbian cross-building, the
> various answers to the stack-overflow question at
> https://stackoverflow.com/questions/19162072/how-to-install-the-raspberry-pi-
> cross-compiler-on-my-linux-host-machine show several options. The Github tool-
> chain that Dirk is using does seem to be the "official" one. A fairly
> comprehensive alternative set with more recent versions is available at
> https://github.com/abhiTronix/raspberry-pi-cross-compilers
>
> There is also a post on Medium at https://medium.com/@au42/the-useful-
> raspberrypi-cross-compile-guide-ea56054de187 that covers resolving library
> issues using the official tool-chain, but it's configuring application builds
> using CMAKE and I haven't looked at it in detail.

I‘ll try to support them as well. I had stopped looking for alternatives once I had a working solution.

[Dirk Zimoch (dirk.zimoch)]

I tried cross-pi-gcc-8.3.0-1 from https://sourceforge.net/projects/raspberry-pi-cross-compilers/ (linked from https://github.com/abhiTronix/raspberry-pi-cross-compilers) but it has no readline.h.

[Dirk Zimoch (dirk.zimoch)]

I could build from cross-pi-gcc-8.3.0-1 with readline but had to manually copy the readline headers, readline library and tinfo library (plus links) to the cross toolchain.

56a4bb5... by Dirk Zimoch on 2019-05-13

moved user configuration to CONFIG_SITE file and support cross-pi-gcc tool chain

[mdavidsaver (mdavidsaver)]

> There seem to be a number of possibilities for Raspbian cross-building

Actually, there are few options for cross-compiling for Raspbian. I'm specifically referring to the Debian derived distro. Raspbian doesn't provide any "official" cross-compiler binaries. Variations on the qemu-user approach are the only ones I would expect to produce guaranteed ABI compatible binaries. There are docker/lxc approaches which might be more palatable than the manual chroot() recipe I linked.

If you aren't willing/able to use qemu-user, then I would advising using buildroot to replace the raspbian image entirely.

When evaluating a particular recipe. Look for mentions of a specific Raspbian release name/version, or copying from a live Pi. Anything else is likely to be a variation on crosstool-ng, and have the same ABI issues you've encountered.

[mdavidsaver (mdavidsaver)]

Maybe a wiki page or blog post would be a more appropriate place to document this build recipe?

review: Needs Fixing

[Andrew Johnson (anj)]

I heard back from Pete Jemian, who wrote:

----------------------- 8< -----------------------

Agree that Dirk references the official RPi tools for cross-compile. I have very little experience with getting that tool chain to work and only rely on native builds for EPICS base. (Those are easy and do not require any custom files.) The abhiTronix/raspberry-pi-cross-compilers repo is very new (August 2018) with one contributor. Haven't tried this yet. This tool chain might be much easier to assemble than the official one.

The proposed changes benefit those who use cross-compilers for the RPi. If merged, the changes should not affect native builds.

----------------------- 8< -----------------------

I like Michael's suggestion that a lot of this information might make more sense as a Wiki, blog or tech-talk post. Updating it then wouldn't have to synchronize with Base releases.

You might also consider splitting your CONFIG_SITE file into CONFIG_SITE.linux-x86.raspbian-arm and CONFIG_SITE.linux-x86_64.raspbian-arm where you would put all the parts specific to a 32-bit host into the first file and the 64-bit parts into the second.

[Dirk Zimoch (dirk.zimoch)]

The 32 bit tool chains work on 64 bit hosts as well (as long as 32 bit libs are installed).
Much here depends on the host. I have tried to hive as much information as possible that the user has a choice. I doubt that it is possible to split the configuration in way that makes it more easy to get the correct settings. It only makes information harder to find.

Any suggestions as to where the documentation may go? I found something here: http://prjemian.github.io/epicspi/ (linked from https://wiki-ext.aps.anl.gov/epics/index.php/HowTo_Documents). It says: "EPICS Base is very easy to build. The wheezy-raspbian distribution already has all the tools necessary to build EPICS Base. All that is necessary is to define the host architecture and then build it." But it does not say how. And it seems it refers to a native build not a cross build.

I can add an article to https://wiki-ext.aps.anl.gov/epics about how to cross compile for raspberry pi. (If you create a login for me). Then all this information including an the example config file can go there and we can leave this here out of EPICS base.

[Dirk Zimoch (dirk.zimoch)]

Seems I already had an account for the EPICS wiki.

[mdavidsaver (mdavidsaver)]

I still don't think that configure/ is the right place to document this recipe.

Also, I dusted off my pi3 and tried a buildroot image. It worked on the first go.

https://github.com/buildroot/buildroot/tree/master/board/raspberrypi

review: Disapprove

[Andrew Johnson (anj)]

Group 10/4: Disapprove this solution.

review: Disapprove

Unmerged commits

56a4bb5... by Dirk Zimoch on 2019-05-13

moved user configuration to CONFIG_SITE file and support cross-pi-gcc tool chain

9866f3e... by Dirk Zimoch on 2019-05-10

support for raspberryPi addedconfigure/os/CONFIG.Common.raspbian-arm

diff --git a/configure/os/CONFIG.Common.raspbian-arm b/configure/os/CONFIG.Common.raspbian-arm
 new file mode 100644
index 0000000..206d984
--- /dev/null
+++ b/configure/os/CONFIG.Common.raspbian-arm
@@ -0,0 +1,11 @@
+# CONFIG.Common.raspbian-arm
+#
+# Definitions for raspbian-arm target cross-builds
+# Override these settings in CONFIG_SITE.Common.raspbian-arm
+#-------------------------------------------------------
+
+# Include definitions common to all Linux ARM targets
+include $(CONFIG)/os/CONFIG.Common.linux-arm
+
+GNU_DIR = $(if $(filter %/arm-bcm2708,$(SDK_DIR)),$(SDK_DIR)/$(SDK_TARGET),$(SDK_DIR))
+GNU_TARGET = $(if $(filter arm-bcm2708%,$(SDK_TARGET)),$(SDK_TARGET),arm-linux-gnueabihf)
diff --git a/configure/os/CONFIG_SITE.Common.raspbian-arm b/configure/os/CONFIG_SITE.Common.raspbian-arm
 new file mode 100644
index 0000000..e7fddc3
--- /dev/null
+++ b/configure/os/CONFIG_SITE.Common.raspbian-arm
@@ -0,0 +1,69 @@
+# CONFIG_SITE.Common.raspbian-arm
+#
+# Site-specific information for raspbian-arm target cross-builds
+#---------------------------------------------------------------
+
+# Tested on:
+# Raspberry Pi 3B+ Raspbian 9
+# Raspberry Pi 2   Raspbian 7
+
+# Two different cross tool chains are supported,
+# either from https://github.com/raspberrypi/tools
+# (the "arm-bcm2708" tool chain)
+# or from  https://sourceforge.net/projects/raspberry-pi-cross-compilers.
+# (the "cross-pi-gcc" tool chain).
+
+# Using the arm-bcm2708 tool chain.
+#
+# This tool chains offers 5 different pre-compiled targets. Which one
+# can be used depends on the glibc version installed on the host computer.
+#
+# Can be installed anywhere, e.g:
+SDK_DIR = /opt/raspberrypi/tools-master/arm-bcm2708
+#
+# Available SDK_TARGETs for arm-bcm2708 SDK:
+#
+# gcc 4.8.3 for 32 bit hosts with GLIBC 2.3 or higher
+SDK_TARGET = gcc-linaro-arm-linux-gnueabihf-raspbian
+#
+# gcc 4.7.1 for 64 bit hosts with GLIBC 2.14 or higher
+# SDK_TARGET = arm-linux-gnueabihf
+#
+# gcc 4.8.3 for 64 bit hosts with GLIBC 2.14 or higher
+# SDK_TARGET = gcc-linaro-arm-linux-gnueabihf-raspbian-x64
+#
+# gcc 4.7.1 for 32 bit hosts with GLIBC 2.4 or higher
+# SDK_TARGET = arm-bcm2708hardfp-linux-gnueabi
+# SDK_TARGET = arm-bcm2708-linux-gnueabi
+
+
+# Using the cross-pi-gcc tool chain.
+#
+# gcc 8.3.0 for 64 bit hosts with GLIBC 2.14 or higher.
+# Must be installed at the following location:
+# SDK_DIR = /opt/cross-pi-gcc-8.3.0-1
+
+
+# Using readline:
+#
+# Due to missing/messed up libs and missing headers in both toolchains,
+# readline needs copies of libtinfo.so.5.9 and libreadline.so.6.2 from
+# a Raspbian 7 rootfs /lib/arm-linux-gnueabihf/ to the toolchain, e.g.
+# $(SDK_DIR)/gcc-linaro-arm-linux-gnueabihf-raspbian/arm-linux-gnueabihf/libc/lib/arm-linux-gnueabihf/
+# or /opt/cross-pi-gcc-8.3.0-1/arm-linux-gnueabihf/lib/, depending on
+# the used toolchain, and manually created links libtinfo.so.5 and libreadline.so.
+#
+# For SDK_TARGET gcc-linaro-arm-linux-gnueabihf-raspbian, an existing incompatible
+# libtinfo.so.5 is in the way. Remove it.
+# (Built with glibc 2.16 like installed on Raspbian 9 but toolchain uses glibc 2.13.)
+#
+# The cross-pi-gcc toolchain is also missing libtinfo.so* as well. Copy the file and links
+# from /lib/arm-linux-gnueabihf/ to /opt/cross-pi-gcc-8.3.0-1/arm-linux-gnueabihf/lib/.
+#
+# Also copy the /usr/include/readline/ directory from some readline 6 installation
+# to $(SDK_DIR)/gcc-linaro-arm-linux-gnueabihf-raspbian/arm-linux-gnueabihf/libc/usr/include/
+# or /opt/cross-pi-gcc-8.3.0-1/arm-linux-gnueabihf/include/, depending on the
+# used toolchain.
+#
+# Once the above fixes are applied, uncomment the next line.
+# COMMANDLINE_LIBRARY = READLINE