Cross Compiling for Raspberry Pi – Part II

Angry-Face-II

In a previous post I described how to use a Ubuntu instance to compile a simple “hello world” program that runs on the Raspberry Pi.

In this post I look at how to cross compile the OpenCV test code [which I created and covered in another post] using cmake and the prebuilt Raspberry Pi toolchain on GitHub for both the Jessie and Wheezy versions of Raspbian.

I’m also going to assume you’ve already read part I – about cross compiling the hello world program – so I will continue straight on from where that article left off.

On the Ubuntu instance go into the pidev folder, clone the PiCamCVTest project and call it camtest:

solderspot@ubuntu:~/raspidev$ git clone https://github.com/solderspot/PiCamCVTest.git camtest

Handling Dependencies

This project already compiles on my Raspberry Pi as is, however it depends on various system libraries on the Pi, including some that I had to explicitly install; notably the PiCamera support libraries and OpenCV.

So to compile this code on the Ubuntu instance we will need to have those libraries also present. One way to do this is to simply copy them over.

The copied files we’ll put into a folder called piroot inside pidev. This represents the root folder of our remote Pi system and any files we copy over will be stored here with their relative paths preserved.

For this project I’m copying everything in the /usr, /lib /etc and /opt folders using the rsync command:

solderspot@ubuntu:~/pidev $ mkdir piroot
solderspot@ubuntu:~/pidev $ rsync -rl --delete-after --safe-links pi@<ip address  of the pi>:/{lib,usr,opt,etc} piroot

The rsync command is going take a significant amount of time the first time it runs as a lot of data has to be copied. However, future rsync’s will only copy files that have changed.

You may a see some permissions errors for some folders under /etc but they can be ignored.

Keep in mind that you need to re-sync the files anytime you make updates to the originals.

Compiling the Project

Now to get the compiler to cross compile our project we need to tell it where it should look for any required libraries. In this case we want it to only search relative to the piroot folder, i.e. the native Pi libraries, and not ones belonging to our Ubuntu instance.

And interestingly enough this is a very simple thing to do if you are using cmake as it has very good cross compiler support.

A quick look at this wiki page tells us all we need to know to get cmake using an alternate toolchain.

Simply create a file called pi-toolchain.cmake in the pidev directory with the following content:

SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)

SET(DEVROOT $ENV{HOME}/pidev)
SET(PIROOT ${DEVROOT}/piroot)
SET(PITOOLS ${DEVROOT}/pitools)

SET(TOOLROOT ${PITOOLS}/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64)

# specify the cross compiler
SET(CMAKE_C_COMPILER   ${TOOLROOT}/bin/arm-linux-gnueabihf-gcc)
SET(CMAKE_CXX_COMPILER ${TOOLROOT}/bin/arm-linux-gnueabihf-g++)

SET(CMAKE_SYSROOT ${PIROOT})
SET(CMAKE_FIND_ROOT_PATH ${PIROOT})


# search for programs in the build host directories
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
# for libraries and headers in the target directories
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

Note: If you are using a 32bit Ubuntu instance then TOOLROOT would be the gcc-linaro-arm-linux-gnueabihf-raspbian folder, i.e. no -x64 at the end.

The above file also assumes that pidev is in your home folder. If you have it somewhere else you will need to modify DEVROOT accordingly.

If you don’t already have cmake installed then do it now:


solderspot@ubuntu:~$ sudo apt-get install cmake

To build the project create a build folder in camtest and run cmake using the -DCMAKE_TOOLCHAIN_FILE option to passing the path to pi-toolchain.cmake:

solderspot@ubuntu:~/pidev/camtest$ mkdir build
solderspot@ubuntu:~/pidev/camtest$ cd build
solderspot@ubuntu:~/pidev/camtest/build$ cmake -DCMAKE_TOOLCHAIN_FILE=../../pi-toolchain.cmake ..

Then run make:

solderspot@ubuntu:~/pidev/camtest/build$ make
[ 20%] Building CXX object CMakeFiles/PiCamCVTest.dir/main.cpp.o
[ 40%] Building CXX object CMakeFiles/PiCamCVTest.dir/camera.cpp.o
[ 60%] Building CXX object CMakeFiles/PiCamCVTest.dir/cameracontrol.cpp.o
[ 80%] Building CXX object CMakeFiles/PiCamCVTest.dir/graphics.cpp.o
[100%] Linking CXX executable PiCamCVTest
/home/solderspot/pidev/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/../lib/gcc/arm-linux-gnueabihf/4.8.3/../../../../arm-linux-gnueabihf/bin/ld: warning: libvchiq_arm.so, needed by /home/solderspot/pidev/piroot/opt/vc/lib/libmmal_vc_client.so, not found (try using -rpath or -rpath-link)
/home/solderspot/pidev/pitools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/../lib/gcc/arm-linux-gnueabihf/4.8.3/../../../../arm-linux-gnueabihf/bin/ld: warning: libvcsm.so, needed by /home/solderspot/pidev/piroot/opt/vc/lib/libmmal_vc_client.so, not found (try using -rpath or -rpath-link)
/home/solderspot/pidev/piroot/opt/vc/lib/libmmal_vc_client.so: undefined reference to `vcsm_lock'
:
:
/home/solderspot/pidev/piroot/opt/vc/lib/libbcm_host.so: undefined reference to `vchi_mphi_message_driver_func_table'
/home/solderspot/pidev/piroot/opt/vc/lib/libmmal_vc_client.so: undefined reference to `vchiq_open_service'
collect2: error: ld returned 1 exit status
make[2]: *** [PiCamCVTest] Error 1
make[1]: *** [CMakeFiles/PiCamCVTest.dir/all] Error 2
make: *** [all] Error 2

The files compiled without any problems but there were linker errors.

However, first I need to point out that I’ve cheated somewhat in the CMakeLists.txt for this project. If you look at that file you will notice that I’ve added the variable PIROOT to all the paths:

cmake_minimum_required(VERSION 2.8)
project( PiCamCVTest )
SET(COMPILE_DEFINITIONS -Werror)

include_directories(SYSTEM ${PIROOT}/opt/vc/include ${PIROOT}/opt/vc/include/interface/vcos/pthreads ${PIROOT}/opt/vc/include/interface/vmcs_host/linux )
link_directories( ${PIROOT}/opt/vc/lib )
add_executable(PiCamCVTest main.cpp camera.cpp cameracontrol.cpp graphics.cpp)

target_link_libraries(PiCamCVTest libmmal_core.so libmmal_util.so libmmal_vc_client.so libvcos.so librt.so libbcm_host.so GLESv2 EGL libopencv_core.so libopencv_imgproc.so)

Technically I don’t feel I should have to do this hack. Setting CMAKE_SYSROOT in pi-toolcahin.cmake to PIROOT should have had the same effect… but it doesn’t. Very annoying. Nevertheless, with the ${PIROOT} hack the code compiles fine.

But there are a lot of linker errors still.

This has to do with the dynamic linker config file /etc/ld.so.conf (in piroot) not being properly parsed by the toolchain, as explained in this article. We could work around the problem by modifying the config file – as suggested in the article – but we want to keep things as transparent as possible so instead of modifying the file I explicitly add the required linker paths to the compiler flags in pi-toolchain.cmake like this:

SET(FLAGS "-Wl,-rpath-link,${PIROOT}/opt/vc/lib -Wl,-rpath-link,${PIROOT}/lib/arm-linux-gnueabihf -Wl,-rpath-link,${PIROOT}/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,${PIROOT}/usr/local/lib")

UNSET(CMAKE_C_FLAGS CACHE)
UNSET(CMAKE_CXX_FLAGS CACHE)

SET(CMAKE_CXX_FLAGS ${FLAGS} CACHE STRING "" FORCE)
SET(CMAKE_C_FLAGS ${FLAGS} CACHE STRING "" FORCE)

[The UNSET…CACHE STRING thing has to do with the way cmake works internally]

So now with this change, and building for the Wheezy release of Raspbian, we get a successful outcome:

solderspot@ubuntu:~/pidev/camtest/build$ cmake -DCMAKE_TOOLCHAIN_FILE=../../pi-toolchain.cmake ..
-- Configuring done
-- Generating done
-- Build files have been written to: /home/solderspot/pidev/camtest/build
solderspot@ubuntu:~/pidev/camtest/build$ make
[ 20%] Building CXX object CMakeFiles/PiCamCVTest.dir/main.cpp.o
[ 40%] Building CXX object CMakeFiles/PiCamCVTest.dir/camera.cpp.o
[ 60%] Building CXX object CMakeFiles/PiCamCVTest.dir/cameracontrol.cpp.o
[ 80%] Building CXX object CMakeFiles/PiCamCVTest.dir/graphics.cpp.o
[100%] Linking CXX executable PiCamCVTest
[100%] Built target PiCamCVTest

However, if you do this same process with the Jessie release of Raspbian we hit a problem:

solderspot@ubuntu:~/pidev/camtest/build$ make
[ 20%] Building CXX object CMakeFiles/PiCamCVTest.dir/main.cpp.o
[ 40%] Building CXX object CMakeFiles/PiCamCVTest.dir/camera.cpp.o
[ 60%] Building CXX object CMakeFiles/PiCamCVTest.dir/cameracontrol.cpp.o
[ 80%] Building CXX object CMakeFiles/PiCamCVTest.dir/graphics.cpp.o
[100%] Linking CXX executable PiCamCVTest
/home/solderspot/pidev/piroot/usr/lib/arm-linux-gnueabihf/libopencv_core.so: undefined reference to `std::__throw_out_of_range_fmt(char const*, ...)@GLIBCXX_3.4.20'
collect2: error: ld returned 1 exit status
make[2]: *** [PiCamCVTest] Error 1
make[1]: *** [CMakeFiles/PiCamCVTest.dir/all] Error 2
make: *** [all] Error 2

This is unfortunately due to the OpenCV core library (for Jessie) being built with a newer version of gcc/g++ and making use of a feature that is not present in our (older) version of the toolchain. To get around this we need to upgrade the toolchain to gcc 4.9.

For now though I’m happy enough to simply use the Wheezy release of Raspbian.

UPDATE: A 4.9 version of the toolchain has been added to the repo. You can find it in arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf. Unfortunately it does not work for this project. For some reason the --sysroot option seems to be broken.

Next step is to look into how to cross-compile ROS (Robotic Operating System) applications for the Pi… though, to be honest, there might not be a need for that as ROS has a very robust dev environment that is platform agnostic…

Advertisements

16 comments

  1. It’s really kind for you to share this tutorial…thanks a lot
    btw, any suggestion to upgrade the toolchain to gcc4.9?

    1. A gcc4.9 toolchain has been added to the tool repo I referred to in the article. It is in the folder: arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf but I’ve not been able to use it successfully yet.

      I’ll update the article once I figure it out.

  2. Awesome guide, i’m looking for a long time for a guide like that !

    I will give it a try since i tried for some weeks to achieve something like that and it really becomes a pain in the …

    1. I got some troubles here.
      I am trying to compile camtest for jessie, so i look for the

      “arm-bcm2708/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf”

      that you mentioned.

      I don’t find it in the repo i just cloned, i got instead

      “arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf”

      which is 2 days ago in the repo.

      I maybe misunderstood something, but a few tips would be appreciated :).

    2. Okay, actually i fgured it out browsing the repository…

      They addes the linaro 4.9.3 support 14/02/16, and removed it today, replacing it with the new folder.
      In fact, the (…)/bin/arm-linux-gnueabihf-gcc-4.9.3 appears to be broken for CMake

      I’ll give it a try with the removed version

      1. Thanks, Morz. I’ll look into this shortly and update the article accordingly.

  3. I did used the older version of the pi tools to compile the project for raspbian jessie.

    I worked, and here are some easy steps to do it :

    First, you need to clone an older version of the tools, so execute

    git clone https://github.com/raspberrypi/tools.git pitools

    instead of

    git clone https://github.com/raspberrypi/tools.git --depth=1 pitools

    in the first part of this guide.

    THEN, Run

    git checkout b81697c

    to get the working version of the linaro compiler.

    After that, you have in your ~/pidev/pitools/arm-bcm2708/ the
    gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf folder.

    You just have to use the .cmake of the part 2 guide, with 2 modifications ::

    SET(CMAKE_SYSTEM_NAME Linux)
    SET(CMAKE_SYSTEM_VERSION 1)
    
    SET(DEVROOT $ENV{HOME}/pidev)
    SET(PIROOT ${DEVROOT}/piroot)
    SET(PITOOLS ${DEVROOT}/pitools)
    
    SET(TOOLROOT ${PITOOLS}/arm-bcm2708/gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf)
    
    #specify the cross compiler
    
    SET(CMAKE_C_COMPILER   ${TOOLROOT}/bin/arm-linux-gnueabihf-gcc-4.9.3)
    SET(CMAKE_CXX_COMPILER ${TOOLROOT}/bin/arm-linux-gnueabihf-g++)
    
    SET(CMAKE_SYSROOT ${PIROOT})
    SET(CMAKE_FIND_ROOT_PATH ${PIROOT})
    
    #search for programs in the build host directories
    
    SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
    
    #for libraries and headers in the target directories
    
    SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
    SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
    
    SET(FLAGS "-Wl,-rpath-link,${PIROOT}/opt/vc/lib -Wl,-rpath-link,${PIROOT}/lib/arm-linux-gnueabihf -Wl,-rpath-link,${PIROOT}/usr/lib/arm-linux-gnueabihf -Wl,-rpath-link,${PIROOT}/usr/local/lib")
    
    UNSET(CMAKE_C_FLAGS CACHE)
    UNSET(CMAKE_CXX_FLAGS CACHE)
    
    SET(CMAKE_CXX_FLAGS ${FLAGS} CACHE STRING "" FORCE)
    SET(CMAKE_C_FLAGS ${FLAGS} CACHE STRING "" FORCE)
    

    Here you are, now the project should compile without problems for raspbian jessie.

    I have done that with a rpi 2 running raspbian jessie, with the last version of opencv atm
    (pkg-config –modversion opencv returning 3.1.0)

    Thanks again for the guide, it really helped me 🙂

    1. Thanks, Morz, for the update.

      There is some work being done on fixing the problems with the 4.9.3 toolchain in the repo (https://github.com/raspberrypi/tools/issues/50) but it might be a while.

      This is a good work-around for now if Jessie support is a requirement.

  4. One more thing to say,

    I tried to fix the linking issue with the solution proposed on the article you mentionned
    (https://sysprogs.com/w/fixing-rpath-link-issues-with-cross-compilers/)

    This solution seems more elegant, but does not seems to work for me (libraries still missing)

    I had to add more opencv libraries for the work i have to do, so i sticked to your “easier” solution and add these in the CMakeList.txt

    1. Yes, agreed. Best to localized changes to the toolchain file in the long term, even if they seem hacky.

      I’m still annoyed I have to add ${PIROOT} to my project’s CMakeList.txt. Hopefully that is something that can also be fixed in due course.

  5. I really enjoyed reading part II of your tutorial Cross Compiling for Raspberry PI.
    In order to search relative to the piroot folder, i.e the prebuilt native Raspberry libraries
    and not ones belonging to your Ubuntu instance , I noticed that you explicity added
    the required linker paths to the compiler flags in pi-toolchain.cmake like this:

    SET(FLAGS “-Wl,-rpath-link,${PIROOT}/opt/vc/lib
    -Wl,-rpath-link,${PIROOT}/lib/arm-linux-gnueabihf
    -Wl,-rpath-link,${PIROOT}/usr/lib/arm-linux-gnueabihf
    -Wl,-rpath-link,${PIROOT}/usr/local/lib”)

    UNSET(CMAKE_C_FLAGS CACHE)
    

    UNSET(CMAKE_CXX_FLAGS CACHE)

    SET(CMAKE_CXX_FLAGS ${FLAGS} CACHE STRING “” FORCE)
    SET(CMAKE_C_FLAGS ${FLAGS} CACHE STRING “” FORCE)

    I was wondering if this approach could be used to solve the problem where I
    am trying to build an ArmV6 executable in our Ubuntu 16.04 instance containing both
    a recently downloaded ARM v6 arm-linux-gnueabi-g++-4.6.3 compiler and
    preinstalled ARM v7 arm-linux-gnueabi-g++-5.3.1 compiler which does not
    SIGSEGV when it is pscp-ed and executed in the /home/pi folder of an Raspian
    ARMv6 Wheezy emulator mounted on an SD card. Here are the bash command line
    statememts I used to build the problematic executable: 1. arm-linux-gnueabi-g++-4.6.3 -o Program.o
    -g -DLINUX -fPIC -c Program.cpp 2. arm-linux-gnueabi-ld-4.6.3 -o Program.exe
    -L /home/frank/tools/usr/lib -lgcc_s -L /home/frank/tools/usr/lib -lgcc.
    When I run strace on command line statement #2, I noticed that I am
    picking up stray Ubuntu 16.04 instance Armv7 libraries and object files
    in the /usr/lib/arm-linux-gnueabi folder. Could that possibly cause a
    SIGSEGV? May I ask the best way to build an ARMv6 executable in my
    heterogenous ARMv6-g++ and ARMv7-g++ compiler situation? Thank you.

  6. Tony_T · · Reply

    Hello and thank you for the amazing tutorial. I have a missing file issue right on the final ‘make’ commands.
    fatal error: opencv2/core/core.hpp: No such file or directory
    Though I have 2 versions of this file? Could it be something to do with the version of OpenCV I have? I have both 3.0.0 and 3.0.1 installed due to some errors I was hitting with 3.0.1.

    1. Tony_T · · Reply

      I’ve now tried removing the 2nd version of opencv (OpenCV-3.1.0) and removed the release file from OpenCV-3.0.0 and recreated and re-built (cmake/make) and yet I still get the same error when trying to make this CamCVTest?
      For some reason the Gcc is not finding the core.hpp and therefore any CV files from it’s root folder in /home?

      Anybody got suggestions?

    2. Hi Tony,

      Can you tell me the full path for the core.hpp file? I presume that you have OpenCV installed on the remote system and are copying it over to your Ubuntu host using the rsync command?

  7. Hi Morz and Solderspot – thanks for both the tutorial and your comments. I have tried to compile a Hello World programme with gcc-linaro-4.9-2015.02-3-x86_64_arm-linux-gnueabihf. It compiled and linked with no issues.However, the executable was 3.7MB.

    Ignoring my suspicions I tried to run it on my Pi Zero (running Jessie) after transferring using scp. On trying to run it I am hit with a Segmentation fault message.

    On compiling the same hello world using gcc-linaro-arm-linux-gnueabihf-raspbian-x64 I have no troubles running it and the executable is a more appropriate 7KB.

    I am not well versed in GDB at all. So I was hoping if you guys could shed some light and help me out..

    Thanks tons in advance.
    JD

    1. Another quick update –>

      I also tried to compile hello world with gcc-linaro-4.9.4-2017.01-x86_64_arm-linux-gnueabihf. Picked this toolchaing from LInaro directly.

      The executable was a lot smaller (11.7KB) but on copying to the Pi I have the same Segmentation fault issue.

      So yes – it seems that the size of the executable did not matter in this case. There’s definitely something in the toolchain which is different to Linaro’s other toolchain.

      Any help would be appreciated.

      Thanks again
      JD

Comments welcome

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s