Compiling Tensorflow from sources

Compiling Tensorflow from sources

First of all, we need a clean Ubuntu VM. I installed an Ubuntu 18.04LTS x86_64 with 80GB HD, 4 CPUs and 4GB of RAM.

After installing and setting the network, update:

apt-get update
apt-get upgrade

Then install basic tools:

apt-get install python3-numpy python3-dev python3-pip python3-wheel
apt-get install python

Then, download and install Bazel:

wget https://github.com/bazelbuild/bazel/releases/download/0.15.2/bazel_0.15.2-linux-x86_64.deb
apt-get install ./bazel-0.15.2-linux-x86_64.deb

Then, download and extract Tensorflow source code:

wget https://github.com/tensorflow/tensorflow/archive/v1.9.0.tar.gz
tar xvfz v1.9.0.tar.gz
cd tensorflow-1.9.0

and configure it:

./configure

During the configuration, specify the python location (in this case, /usr/bin/python3) and just accept the defaults.

After that, it's time to compile:

bazel build --config=opt //tensorflow/tools/pip_package:build_pip_package

during the compilation process, it may show many warnings.

The entire process will take about 2-3 hours.

To create a pip package:

./bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/tensorflow_pkg

Under /tmp/tensorflow_pkg there will be the pip package: tensorflow-1.0-cp36-linux_x86_64.whl

It can be moved to the target machine and installed:

pip install /tmp/tensorflow_pkg/tensorflow-1.0-cp36-linux_x86_64.whl

Cross Compiling for Nanopi Neo (arm)

First of all, let's give the machine at least 8GB of RAM (otherwise you will run into a subtle 'Error 4' when compiling).

Download the toolchain from linaro:

https://releases.linaro.org/components/toolchain/binaries/latest/arm-linux-gnueabihf/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf.tar.xz
tar xvf gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf.tar.xz

In tensorflow directory:

File BUILD:

package(default_visibility = ["//visibility:public"])

cc_toolchain_suite(
    name = "toolchain",
    toolchains = {
        "armv7l|compiler": ":cc-compiler-armv7l",
    },
)

 filegroup(
     name = "empty",
     srcs = [],
)

filegroup(
    name = "arm_linux_all_files",
    srcs = [
        "@toolchain_target_armv7l//:compiler_pieces",
    ],
)

cc_toolchain(
    name = "cc-compiler-armv7l",
    all_files = ":arm_linux_all_files",
    compiler_files = ":arm_linux_all_files",
    cpu = "armv7l",
    dwp_files = ":empty",
    dynamic_runtime_libs = [":empty"],
    linker_files = ":arm_linux_all_files",
    objcopy_files = "arm_linux_all_files",
    static_runtime_libs = [":empty"],
    strip_files = "arm_linux_all_files",
    supports_param_files = 1,
)

Add to the end of the file WORKSPACE:

new_local_repository(
   name = 'toolchain_target_armv7l',
   path = '/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf',
   build_file = 'arm_compiler.BUILD'
)

File CROSSTOOL:

major_version: "local"
minor_version: ""
default_target_cpu: "armv7l"

default_toolchain {
  cpu: "armv7l"
  toolchain_identifier: "armv7l"
}

toolchain {
  abi_version: "armv7l"
  abi_libc_version: "armv7l"
  builtin_sysroot: ""
  compiler: "compiler"
  host_system_name: "armv7l"
  needsPic: true
  supports_gold_linker: false
  supports_incremental_linker: false
  supports_fission: false
  supports_interface_shared_objects: false
  supports_normalizing_ar: true
  supports_start_end_lib: false
  supports_thin_archives: true
  target_libc: "armv7l"
  target_cpu: "armv7l"
  target_system_name: "armv7l"
  toolchain_identifier: "armv7l"

  tool_path { name: "ar" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-ar" }
  tool_path { name: "compat-ld" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-ld" }
  tool_path { name: "cpp" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-cpp" }
  tool_path { name: "gcc" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcc" }
  tool_path { name: "gcov" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-gcov" }
  tool_path { name: "ld" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-ld" }
  tool_path { name: "nm" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-nm" }
  tool_path { name: "objcopy" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-objcopy" }
  objcopy_embed_flag: "-I"
  objcopy_embed_flag: "binary"
  tool_path { name: "objdump" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-objdump" }
  tool_path { name: "strip" path: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/bin/arm-linux-gnueabihf-strip" }

  cxx_builtin_include_directory: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/include"
  cxx_builtin_include_directory: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/lib/gcc/arm-linux-gnueabihf/7.3.1/include"
  cxx_builtin_include_directory: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/include"
  cxx_builtin_include_directory: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/lib/gcc/arm-linux-gnueabihf/7.3.1/include-fixed"
  cxx_builtin_include_directory: "/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/libc/usr/include"
  cxx_builtin_include_directory: "/usr/include"
  cxx_builtin_include_directory: "/usr/lib/python2.7/dist-packages/numpy/core/include/numpy/"

  linker_flag: "-L/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/lib"
  linker_flag: "-L/home/fabio/gcc-linaro-7.3.1-2018.05-x86_64_arm-linux-gnueabihf/arm-linux-gnueabihf/lib"

  # Anticipated future default.
  # This makes GCC and Clang do what we want when called through symlinks.
  unfiltered_cxx_flag: "-no-canonical-prefixes"
  linker_flag: "-no-canonical-prefixes"

  # Make C++ compilation deterministic. Use linkstamping instead of these
  # compiler symbols.
  unfiltered_cxx_flag: "-Wno-builtin-macro-redefined"
  unfiltered_cxx_flag: "-D__DATE__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIMESTAMP__=\"redacted\""
  unfiltered_cxx_flag: "-D__TIME__=\"redacted\""

  # Security hardening on by default.
  # Conservative choice; -D_FORTIFY_SOURCE=2 may be unsafe in some cases.
  # We need to undef it before redefining it as some distributions now have
  # it enabled by default.
  compiler_flag: "-U_FORTIFY_SOURCE"
  compiler_flag: "-fstack-protector"
  compiler_flag: "-fPIE"
  compiler_flag: "-std=gnu99"
  linker_flag: "-pie"
  linker_flag: "-Wl,-z,relro,-z,now"

  # Enable coloring even if there's no attached terminal. Bazel removes the
  # escape sequences if --nocolor is specified.
  # compiler_flag: "-fdiagnostics-color=always"

  # All warnings are enabled. Maybe enable -Werror as well?
  compiler_flag: "-Wall"
  compiler_flag: "-std=c++0x"
  # Enable a few more warnings that aren't part of -Wall.
  compiler_flag: "-Wunused-but-set-parameter"
  # But disable some that are problematic.
  compiler_flag: "-Wno-free-nonheap-object" # has false positives

  # Keep stack frames for debugging, even in opt mode.
  compiler_flag: "-fno-omit-frame-pointer"

  # Stamp the binary with a unique identifier.
  linker_flag: "-Wl,--build-id=md5"
  linker_flag: "-Wl,--hash-style=gnu"

  compilation_mode_flags {
    mode: DBG
    # Enable debug symbols.
    compiler_flag: "-g"
  }
  compilation_mode_flags {
    mode: OPT

    # No debug symbols.
    # Maybe we should enable https://gcc.gnu.org/wiki/DebugFission for opt or
    # even generally? However, that can't happen here, as it requires special
    # handling in Bazel.
    compiler_flag: "-g0"

    # Conservative choice for -O
    # -O3 can increase binary size and even slow down the resulting binaries.
    # Profile first and / or use FDO if you need better performance than this.
    compiler_flag: "-O2"

    # Disable assertions
    compiler_flag: "-DNDEBUG"

    # Removal of unused code and data at link time (can this increase binary size in some cases?).
    compiler_flag: "-ffunction-sections"
    compiler_flag: "-fdata-sections"
    linker_flag: "-Wl,--gc-sections"
  }
}

Launch ./configure, the only parameter to set is -march=armv7-a . Then,

bazel build --cpu=armv7l --crosstool_top=//:toolchain --host_crosstool_top=@bazel_tools//tools/cpp:toolchain --config=opt //tensorflow/tools/pip_package:build_pip_package --verbose_failures

If you get the error:

/home/fabio/.cache/bazel/_bazel_root/fd12a6b6cbea8cd3e3322a5a0a22d119/external/nsync/BUILD:466:13: Configurable attribute "copts" doesn't match this configuration (would a default condition help?).

then, you have to edit the file /home/fabio/.cache/bazel/_bazel_root/fd12a6b6cbea8cd3e3322a5a0a22d119/external/nsync/BUILD and add the line:

...
NSYNC_OPTS_GENERIC = select({
   ...
   ...
   "//conditions:default": [],
}) + [
...

Then, relaunch the compile process.

If you get an error about pyconfig.h not found, then you have to copy it from the directory:

/usr/include/arm-linux-gnueabihf/python2.7/pyconfig.h

of a native arm installation in the same directory of your x86_64 machine. It's part of the python-dev package, so you have to install it via

apt-get install python-dev

in the arm machine first. Then, you have to modify the /usr/include/python2.7/pyconfig.h file consequently:

#  if defined(__ARMEL__)
//#  include <arm-linux-gnueabihf/python2.7/pyconfig.h>
# include "/usr/include/arm-linux-gnueabihf/python2.7/pyconfig.h"
#  else
# include "/usr/include/arm-linux-gnueabihf/python2.7/pyconfig.h"
//#  include <armeb-linux-gnueabihf/python2.7/pyconfig.h>
#  endif
# elif defined(__ARM_EABI__) && !defined(__ARM_PCS_VFP)
#  if defined(__ARMEL__)
//#  include <arm-linux-gnueabi/python2.7/pyconfig.h>
# include "/usr/include/arm-linux-gnueabihf/python2.7/pyconfig.h"

Only one of these lines is necessary, but I modified all ARM lines in order to be sure that the file will be imported.

To make the compile process successfull, you have to install the mock and the enum34 package.

apt-get install python-pip
pip install mock

Note: the enum34 package will be automatically installed with the python-pip. Don't install (or uninstall if already installed) the python enum package, as it's a totally different package and it will make the compile process fail.

Differences with python 3

For compiling for with python 3 support, of course you need to change the links in the CROSSTOOL file, and install the corresponding packets.

Notice that, since python 3 has different subversions, you need to choose a linux distribution that installs the same packets as the target machine. For example, if you need to install to an Ubuntu 16, you need to create an Ubuntu 16 virtual machine.