LXC 1.0: Some more advanced container usage [4/10]

This is post 4 out of 10 in the LXC 1.0 blog post series.

Running foreign architectures

By default LXC will only let you run containers of one of the architectures supported by the host. That makes sense since after all, your CPU doesn’t know what to do with anything else.

Except that we have this convenient package called “qemu-user-static” which contains a whole bunch of emulators for quite a few interesting architectures. The most common and useful of those is qemu-arm-static which will let you run most armv7 binaries directly on x86.

The “ubuntu” template knows how to make use of qemu-user-static, so you can simply check that you have the “qemu-user-static” package installed, then run:

sudo lxc-create -t ubuntu -n p3 -- -a armhf

After a rather long bootstrap, you’ll get a new p3 container which will be mostly running Ubuntu armhf. I’m saying mostly because the qemu emulation comes with a few limitations, the biggest of which is that any piece of software using the ptrace() syscall will fail and so will anything using netlink. As a result, LXC will install the host architecture version of upstart and a few of the networking tools so that the containers can boot properly.

stgraber@castiana:~$ file /bin/ls
/bin/ls: ELF 64-bit LSB  executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.24, """BuildID[sha1]""" =e50e0a5dadb8a7f4eaa2fd715cacb9842e157dc7, stripped
stgraber@castiana:~$ sudo lxc-start -n p3 -d
stgraber@castiana:~$ sudo lxc-attach -n p3
root@p3:/# file /bin/ls
/bin/ls: ELF 32-bit LSB  executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, """BuildID[sha1]""" =88ff013a8fd9389747fb1fea1c898547fb0f650a, stripped
root@p3:/# exit
stgraber@castiana:~$ sudo lxc-stop -n p3
stgraber@castiana:~$

Hooks

As we know people like to script their containers and that our configuration can’t always accommodate every single use case, we’ve introduced a set of hooks which you may use.

Those hooks are simple paths to an executable file which LXC will run at some specific time in the lifetime of the container. Those executables will also be passed a set of useful environment variables so they can easily know what container invoked them and what to do.

The currently available hooks are (details in lxc.conf(5)):

  • lxc.hook.pre-start (called before any initialization is done)
  • lxc.hook.pre-mount (called after creating the mount namespace but before mounting anything)
  • lxc.hook.mount (called after the mounts but before pivot_root)
  • lxc.hook.autodev (identical to mount but only called if using autodev)
  • lxc.hook.start (called in the container right before /sbin/init)
  • lxc.hook.post-stop (run after the container has been shutdown)
  • lxc.hook.clone (called when cloning a container into a new one)

Additionally each network section may also define two additional hooks:

  • lxc.network.script.up (called in the network namespace after the interface was created)
  • lxc.network.script.down (called in the network namespace before destroying the interface)

All of those hooks may be specified as many times as you want in the configuration so you can use each hooking point multiple times.

As a simple example, let’s add the following to our “p1” container:

lxc.hook.pre-start = /var/lib/lxc/p1/pre-start.sh

And create the hook itself at /var/lib/lxc/p1/pre-start.sh:

#!/bin/sh
echo "arguments: $*" > /tmp/test
echo "environment:" >> /tmp/test
env | grep LXC >> /tmp/test

Make it executable (chmod 755) and then start the container.
Checking /tmp/test you should see:

arguments: p1 lxc pre-start
environment:
LXC_ROOTFS_MOUNT=/usr/lib/x86_64-linux-gnu/lxc
LXC_CONFIG_FILE=/var/lib/lxc/p1/config
LXC_ROOTFS_PATH=/var/lib/lxc/p1/rootfs
LXC_NAME=p1

Android containers

I’ve often been asked whether it was possible to run Android in an LXC container. Well, the short answer is yes. However it’s not very simple and it really depends on what you want to do with it.

The first thing you’ll need if you want to do this is get your machine to run an Android kernel, you’ll need to have any modules needed by Android built and loaded before you can start the container.

Once you have that, you’ll need to create a new container by hand.
Let’s put it in “/var/lib/lxc/android/”, in there, you need a configuration file similar to this one:

lxc.rootfs = /var/lib/lxc/android/rootfs
lxc.utsname = armhf

lxc.network.type = none

lxc.devttydir = lxc
lxc.tty = 4
lxc.pts = 1024
lxc.arch = armhf
lxc.cap.drop = mac_admin mac_override
lxc.pivotdir = lxc_putold

lxc.hook.pre-start = /var/lib/lxc/android/pre-start.sh

lxc.aa_profile = unconfined

/var/lib/lxc/android/pre-start.sh is where the interesting bits happen. It needs to be an executable shell script, containing something along the lines of:

#!/bin/sh
mkdir -p $LXC_ROOTFS_PATH
mount -n -t tmpfs tmpfs $LXC_ROOTFS_PATH

cd $LXC_ROOTFS_PATH
cat /var/lib/lxc/android/initrd.gz | gzip -d | cpio -i

# Create /dev/pts if missing
mkdir -p $LXC_ROOTFS_PATH/dev/pts

Then get the initrd for your device and place it in /var/lib/lxc/android/initrd.gz.

At that point, when starting the LXC container, the Android initrd will be unpacked on a tmpfs (similar to Android’s ramfs) and Android’s init will be started which in turn should mount any partition that Android requires and then start all of the usual services.

Because there are no apparmor, cgroup or even network configuration applied to it, the container will have a lot of rights and will typically completely crash the machine. You unfortunately have to be familiar with the way Android works and not be afraid to modify its init scripts if not even its init process to only start the bits you actually want.

I can’t provide a generic recipe there as it completely depends on what you’re interested on, what version of Android and what device you’re using. But it’s clearly possible to do and you may want to look at Ubuntu Touch to see how we’re doing it by default there.

One last note, Android’s init script isn’t in /sbin/init, so you need to tell LXC where to load it with:

lxc-start -n android -- /init

LXC on Android devices

So now that we’ve seen how to run Android in LXC, let’s talk about running Ubuntu on Android in LXC.

LXC has been ported to bionic (Android’s C library) and while not feature-equivalent with its glibc build, it’s still good enough to be used.

Unfortunately due to the kind of low level access LXC requires and the fact that our primary focus isn’t Android, installation could be easier…You won’t be finding LXC on the Google PlayStore and we won’t provide you with a .apk that you can install.

Instead every time something changes in the upstream git branch, we produce a new tarball which can be downloaded here: https://jenkins.linuxcontainers.org/view/LXC/view/LXC%20builds/job/lxc-build-android/lastSuccessfulBuild/artifact/lxc-android.tar.gz

This build is known to work with Android >= 4.2 but will quite likely work on older versions too.

For this to work, you’ll need to grab your device’s kernel configuration and run lxc-checkconfig against it to see whether it’s compatible with LXC or not. Unfortunately it’s very likely that it won’t be… In that case, you’ll need to go hunt for the kernel source for your device, add the missing feature flags, rebuild it and update your device to boot your updated kernel.

As scary as this may sound, it’s usually not that difficult as long as your device is unlocked and you’re already using an alternate ROM like Cyanogen which usually make their kernel git tree easily available.

Once your device has a working kernel, all you need to do is unpack our tarball as root in your device’s / directory, copy an arm container to /data/lxc/containers/<container name>, get into /data/lxc and run “./run-lxc lxc-start -n <container name>”.
A few seconds later you’ll be greeted by a login prompt.

About Stéphane Graber

Project leader of Linux Containers, Linux hacker, Ubuntu core developer, conference organizer and speaker.
This entry was posted in Canonical voices, LXC, Planet Ubuntu, Ubuntu Touch and tagged . Bookmark the permalink.

32 Responses to LXC 1.0: Some more advanced container usage [4/10]

  1. many group says:

    I checked the configuration of my android emulator using lxc-checkconfig and it is showing namespaces, cgroups, etc required. How to enable those kernel features for running lxc in android?

    1. You need to grab the source code for your current kernel, then edit .config either by hand or using something like “make menuconfig” to find and enable the missing options.

  2. Philippe says:

    about the hooks…

    i’m trying to set up an OS container running archlinux. Here I encounter 2 distinct road bumps:
    – start the container’s systemd (lower priority for me).
    – get pacman running to update and install new stuff.

    systemd-nspawn also has been described as “chroot on steroids” and uses cgroup to “partition” the resources. this may be an interoperability with lxc a breeze or ugly nightmare. I Guess time will tell if more distro use it as their init system (or not).

    to get pacman properly running in a chroot one needs to mount/create /proc (for mtab link actually), /dev/random, /dev/urandom and /dev/pts (although my personal experience is that it is not REQUIRED). Now in all the containers I’ve tried pacman fails on a GPGME: ioctl not appropriate error even if the resources are present and accounted for.

    None of the (many) variations I’ve tried so far on the container configuration file did improve that point.

    I’ve been testing this with wheezy (kernel 3.2) and arch (kernel 3.12) hosts.

    After that longish preamble my question: can you suggest some magic stanza to put in say lxc.hook.autodev and or lxc.hook.start ?

  3. ziomario ziomario says:

    Hello.

    I’m on the samsung arm chromebook. I’ve installed ubuntu 12.04 LTS with the kernel 3.13 already configured for lxc and android. Now I would like to know how to configure the android container to run android as guest os and ubuntu as host os….

    1. I can’t really help you much there, though things that you need to know are that ChromeOS isn’t Android, it actually doesn’t share any code with Android that I’m aware of.

      Running Android in LXC requires an Android kernel which if you’re using a ChromeOS kernel, you don’t have.

      So you’d first need to find an actual Android build for your Chromebook, including its kernel, then make your Chromebook run using that kernel (or rebuild it so that it does) and after that, you should be able to run that Android build inside LXC.

      1. why did I type this? says:

        I’m way late to the party here but, he said (>3 years ago) he was running Ubuntu 12.04 LTS on the device rather than ChromeOS – & that his kernel was already configured for LXC and Android (presumably a modded Android kernel to support LXC).

        That said, I’m not sure what the question was. Seems like he just needed to read up on LXC commands and fire up a container (I only know Docker, so I’m no help here – not sure why I typed this tbh!)

  4. Andrew says:

    Your hook example runs the script in the host. Is there a way to run the script inside the container?

  5. Bruce says:

    could you please shed me the light on how to build the lxc running on Android platform? I saw the script of build-android in the lxc-ci folder in github but I can’t make it workable. thx…

  6. tsg says:

    Hey Stéphane, first of all, great blog series. Very helpful. Thanks for putting this together!

    In the Android-in-a-container context: I was wondering if you had any pointers on borrowing the lxc generic-x86 recipe from ubuntu-touch to run on real, generic x86 hardware, like a simple Intel Atom-based SoC that has PowerVR graphics. Do you have any plans on writing on the subject?

  7. Hi Stephane,

    I just rooted a spare Galaxy S3 of mine and I was thinking about experimenting Docker on it…Is that possible today?

    thanks!

    1. RayGo says:

      I’m just trying to do the same thing!
      I’m trying to translate the go lang to java using gobind then run docker on a android device!
      But I really doubt if it works, for it is true that docker may require few low access.If you have more information about this,could you please share it with me?really appreciate that!thanks!

  8. feryllt says:

    Hi Stephane,
    I was trying to use the Android binaries as in paragraph “LXC on Android devices”.

    Unfortunatly it does not work on 4.4.4. It keeps saying:

    lxc-start: No such file or directory – failed to create symlink for kmsg

    As a matter of fact lxc-checkconfig says that everything is setup correctly.

    Do you may have any clue?

    PS: is it possible to have access to the pathches/build configuration to be able to recompile LXC for bionic?

    Thanks in advance.

    Do you may have any clue on this?

    1. mahasamoot says:

      This is likely due to the changes in permitions that happened in 4.4. They locked down SD cards without giving you a way to assign the permitions needed to use the files. They fixed this by adding new permitions to let programs access only the files they need on the card in 4.5… in the mean time lots of stuff is broken in 4.4. Hope this helps.

  9. Positron says:

    Hi Stephane; how would one go about sticking an Android container in AppArmor?

  10. abraham says:

    about lxc on android..

    I couldn’t find a “run-lxc” file in your pre-built-tarball.
    When I execute “lxc-create” or “sh lxc-create” immediately, an error message is occurred like as followings:
    # /system/bin/sh ./lxc-create
    ./lxc-create[1]: syntax error: ‘?4’ unexpected

    Running “lxc-checkconfig” was success through installing busybox, but running “lxc-create” was failed.
    please help me.

    1. Hi abraham,
      The code has changed somewhat since Stephane first wrote the blog post and he has not had time to update it. On https://code.launchpad.net/~ubuntu-lxc/lxc/android-build-scripts click “Browse the code” and you will see the “run-lxc” script.

      – yba

  11. Torsten says:

    Hey,

    may someone who had success in following the steps in this post can name the ROM/Kernel Version and device they used. I’m currently trying to use the Nexus 5 (hammerhead).

    Thanks!

    1. Torsten says:

      Hi, i have some problems starting a container under Android following this blog post.

      Here is my setup:

      Nexus 5 (Hammerhead)
      Cyanogenmod 11 (Kernel 3.4, Android 4.4.4) – I added the missing options and checked them with lxc-checkconfig, then recompiled the kernel.

      Created a container on my Ubuntu Machine using:
      sudo lxc-create -t ubuntu -n p3 — -a armhf

      Copied the container to the phone (/data/lxc/containers/p3) and adapted the path information in the p3/config file.

      I downloaded the tarball containing the prebuilt lxc-utils and installed them as mentioned above (and then added the missing run-lxc and prepare.sh from the link Jonathan provided on the replies). I also downloaded the lxc-utils source and compiled them but the results i get is the same.

      When i execute ./run-lxc lxc-start -n p3 -F i get the following error:
      1|root@hammerhead:/data/lxc # ./run-lxc lxc-start -n p3 -F
      lxc-start: cgfsng.c: all_controllers_found: 431 no systemd controller mountpoint found
      lxc-start: start.c: lxc_spawn: 1036 failed initializing cgroup support
      lxc-start: start.c: __lxc_start: 1286 failed to spawn ‘p3’
      lxc-start: lxc_start.c: main: 344 The container failed to start.

      Unfortunately, i was not able to track down the source of this error.

      Best,
      Torsten

      1. jeje38 says:

        Try to folow more details instructions here (in French)
        http://www.enten.fr/steven/2014/12/30/lxc-et-android-5-arm64/

        seem in your case that cgroup are not mounted…

  12. Hi says:

    Do you know what kernels are compatible with LXC(i. e. CyanogenMod)?

  13. GizmoChicken says:

    First please excuse me if my question doesn’t make sense. I’m very new to the LXC.

    Regarding running Android in an LXC container, this post, which is admittedly quite old, states:

    “The first thing you’ll need if you want to do this is get your machine to run an Android kernel, you’ll need to have any modules needed by Android built and loaded before you can start the container.”

    I’ve been trying to learn more about how Google manages to run Android apps on their Chromebooks. I’m under the impression (or possibly mis-impression) that, to do so, Google now relies on containerization to run a stripped down version of Android as a container on top of Chrome OS. But the Chromebooks that can run Android apps do so using a kernel that, so I’m told, doesn’t differ much from a mainline Linux kernel. See https://groups.google.com/a/chromium.org/forum/#!topic/chromium-os-discuss/OfBln-hl7ug

    Would you please share a few brief thoughts (or even speculation) on how Google is running an Android container on a machine that runs a nearly mainline Linux kernel?

    Thanks in advance for any information that you can provide!

    Best regards,
    GizmoChicken

  14. Sanjeev Sharma says:

    Hello All,

    We are trying an Android container on dra7xx ARM platform but every-time when starting container, it leads to segfault.

    wait4(2745, NULL, 0, NULL) = 2745
    — SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2745, si_uid=0, si_status=0, si_utime=0, si_stime=0} —
    socket(PF_LOCAL, SOCK_STREAM, 0) = 4
    connect(4, {sa_family=AF_LOCAL, sun_path=@”lxc/ad055575fe28ddd5//var/lib/lxc”}, 36) = -1 ECONNREFUSED (Connection refused)
    gettimeofday({1473837550, 356010}, NULL) = 0
    — SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x8} —
    +++ killed by SIGSEGV +++
    Segmentation fault

    Please provide your pointer, what could be the potential reason ?

  15. Roxas says:

    Hi,

    Thank you very much for your blog.

    I have a some Question.

    I can find “initrd.gz” file.

    How can I find “initrd.gz” file?

  16. JuniorJPDJ says:

    Hi, I’ve trouble to make lxc run on android 4.4 (cyanogenmod11).
    My kernel is compatible, I untared file, downloaded scripts from http://bazaar.launchpad.net/~ubuntu-lxc/lxc/android-build-scripts/files and when I try to do anything (at example lxc-ls) I get much errors:

    “`
    127|root@kis3:/data/lxc # sh ./run-lxc lxc-ls
    chroot: can’t execute ‘lxc-ls’: No such file or directory
    “`

    “`
    127|root@kis3:/data/lxc # sh ./prepare.sh
    link failed No such file or directory
    “`

    1. Sunil Ramanarayanan says:

      This looks like running 32 bit bins on 64 bit architecture, I am also trying to compile it for 64 bit ARM so that this doesnt happen

  17. Dickzhuang says:

    Lxc -build-android build lxc src .Can lxc-android.tar.gz run on android x86 ?
    This lxc need to use kernel>3.8?I find /proc/PID/ns/mnt can not work on kernel <3.8
    Can you give me a demo or link about runing lxc on android x86? because android x86 runing on PC with x86 much more fast!

  18. Sunil Ramanarayanan says:

    Could you possibly also host a job providing ARM64 configuration for https://jenkins.linuxcontainers.org/view/LXC/job/lxc-build-android/

    I wanted to try it on 64 bit ARM processor

  19. Patel Krunal says:

    Hi,

    Thank you very much for your blog.

    I have a some Question.

    I can find “initrd.gz” file.

    How can I find “initrd.gz” file?

Leave a Reply to ziomario ziomario Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.