LXC 1.0: Unprivileged containers [7/10]

This is post 7 out of 10 in the LXC 1.0 blog post series.

Introduction to unprivileged containers

The support of unprivileged containers is in my opinion one of the most important new features of LXC 1.0.

You may remember from previous posts that I mentioned that LXC should be considered unsafe because while running in a separate namespace, uid 0 in your container is still equal to uid 0 outside of the container, meaning that if you somehow get access to any host resource through proc, sys or some random syscalls, you can potentially escape the container and then you’ll be root on the host.

That’s what user namespaces were designed for and implemented. It was a multi-year effort to think them through and slowly push the hundreds of patches required into the upstream kernel, but finally with 3.12 we got to a point where we can start a full system container entirely as a user.

So how do those user namespaces work? Well, simply put, each user that’s allowed to use them on the system gets assigned a range of unused uids and gids, ideally a whole 65536 of them. You can then use those uids and gids with two standard tools called newuidmap and newgidmap which will let you map any of those uids and gids to virtual uids and gids in a user namespace.

That means you can create a container with the following configuration:

lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536

The above means that I have one uid map and one gid map defined for my container which will map uids and gids 0 through 65536 in the container to uids and gids 100000 through 165536 on the host.

For this to be allowed, I need to have those ranges assigned to my user at the system level with:

stgraber@castiana:~$ grep stgraber /etc/sub* 2>/dev/null
/etc/subgid:stgraber:100000:65536
/etc/subuid:stgraber:100000:65536

LXC has now been updated so that all the tools are aware of those unprivileged containers. The standard paths also have their unprivileged equivalents:

  • /etc/lxc/lxc.conf => ~/.config/lxc/lxc.conf
  • /etc/lxc/default.conf => ~/.config/lxc/default.conf
  • /var/lib/lxc => ~/.local/share/lxc
  • /var/lib/lxcsnaps => ~/.local/share/lxcsnaps
  • /var/cache/lxc => ~/.cache/lxc

Your user, while it can create new user namespaces in which it’ll be uid 0 and will have some of root’s privileges against resources tied to that namespace will obviously not be granted any extra privilege on the host.

One such thing is creating new network devices on the host or changing bridge configuration. To workaround that, we wrote a tool called “lxc-user-nic” which is the only SETUID binary part of LXC 1.0 and which performs one simple task.
It parses a configuration file and based on its content will create network devices for the user and bridge them. To prevent abuse, you can restrict the number of devices a user can request and to what bridge they may be added.

An example is my own /etc/lxc/lxc-usernet file:

stgraber veth lxcbr0 10

This declares that the user “stgraber” is allowed up to 10 veth type devices to be created and added to the bridge called lxcbr0.

Between what’s offered by the user namespace in the kernel and that setuid tool, we’ve got all that’s needed to run most distributions unprivileged.

Pre-requirements

All examples and instructions I’ll be giving below are expecting that you are running a perfectly up to date version of Ubuntu 14.04 (codename trusty). That’s a pre-release of Ubuntu so you may want to run it in a VM or on a spare machine rather than upgrading your production computer.

The reason to want something that recent is because the rough requirements for well working unprivileged containers are:

  • Kernel: 3.13 + a couple of staging patches (which Ubuntu has in its kernel)
  • User namespaces enabled in the kernel
  • A very recent version of shadow that supports subuid/subgid
  • Per-user cgroups on all controllers (which I turned on a couple of weeks ago)
  • LXC 1.0 beta2 or higher (released two days ago)
  • A version of PAM with a loginuid patch that’s yet to be in any released version

Those requirements happen to all be true of the current development release of Ubuntu as of two days ago.

LXC pre-built containers

User namespaces come with quite a few obvious limitations. For example in a user namespace you won’t be allowed to use mknod to create a block or character device as being allowed to do so would let you access anything on the host. Same thing goes with some filesystems, you won’t for example be allowed to do loop mounts or mount an ext partition, even if you can access the block device.

Those limitations while not necessarily world ending in day to day use are a big problem during the initial bootstrap of a container as tools like debootstrap, yum, … usually try to do some of those restricted actions and will fail pretty badly.

Some templates may be tweaked to work and workaround such as a modified fakeroot could be used to bypass some of those limitations but the goal of the LXC project isn’t to require all of our users to be distro engineers, so we came up with a much simpler solution.

I wrote a new template called “download” which instead of assembling the rootfs and configuration locally will instead contact a server which contains daily pre-built rootfs and configuration for most common templates.

Those images are built from our Jenkins server using a few machines I have on my home network (a set of powerful x86 builders and a quadcore ARM board). The actual build process is pretty straightforward, a basic chroot is assembled, then the current git master is downloaded, built and the standard templates are run with the right release and architecture, the resulting rootfs is compressed, a basic config and metadata (expiry, files to template, …) is saved, the result is pulled by our main server, signed with a dedicated GPG key and published on the public web server.

The client side is a simple template which contacts the server over https (the domain is also DNSSEC enabled and available over IPv6), grabs signed indexes of all the available images, checks if the requested combination of distribution, release and architecture is supported and if it is, grabs the rootfs and metadata tarballs, validates their signature and stores them in a local cache. Any container creation after that point is done using that cache until the time the cache entries expires at which point it’ll grab a new copy from the server.

The current list of images is (as can be requested by passing –list):

---
DIST      RELEASE   ARCH    VARIANT    BUILD
---
debian    wheezy    amd64   default    20140116_22:43
debian    wheezy    armel   default    20140116_22:43
debian    wheezy    armhf   default    20140116_22:43
debian    wheezy    i386    default    20140116_22:43
debian    jessie    amd64   default    20140116_22:43
debian    jessie    armel   default    20140116_22:43
debian    jessie    armhf   default    20140116_22:43
debian    jessie    i386    default    20140116_22:43
debian    sid       amd64   default    20140116_22:43
debian    sid       armel   default    20140116_22:43
debian    sid       armhf   default    20140116_22:43
debian    sid       i386    default    20140116_22:43
oracle    6.5       amd64   default    20140117_11:41
oracle    6.5       i386    default    20140117_11:41
plamo     5.x       amd64   default    20140116_21:37
plamo     5.x       i386    default    20140116_21:37
ubuntu    lucid     amd64   default    20140117_03:50
ubuntu    lucid     i386    default    20140117_03:50
ubuntu    precise   amd64   default    20140117_03:50
ubuntu    precise   armel   default    20140117_03:50
ubuntu    precise   armhf   default    20140117_03:50
ubuntu    precise   i386    default    20140117_03:50
ubuntu    quantal   amd64   default    20140117_03:50
ubuntu    quantal   armel   default    20140117_03:50
ubuntu    quantal   armhf   default    20140117_03:50
ubuntu    quantal   i386    default    20140117_03:50
ubuntu    raring    amd64   default    20140117_03:50
ubuntu    raring    armhf   default    20140117_03:50
ubuntu    raring    i386    default    20140117_03:50
ubuntu    saucy     amd64   default    20140117_03:50
ubuntu    saucy     armhf   default    20140117_03:50
ubuntu    saucy     i386    default    20140117_03:50
ubuntu    trusty    amd64   default    20140117_03:50
ubuntu    trusty    armhf   default    20140117_03:50
ubuntu    trusty    i386    default    20140117_03:50

The template has been carefully written to work on any system that has a POSIX compliant shell with wget. gpg is recommended but can be disabled if your host doesn’t have it (at your own risks).

The same template can be used against your own server, which I hope will be very useful for enterprise deployments to build templates in a central location and have them pulled by all the hosts automatically using our expiry mechanism to keep them fresh.

While the template was designed to workaround limitations of unprivileged containers, it works just as well with system containers, so even on a system that doesn’t support unprivileged containers you can do:

lxc-create -t download -n p1 -- -d ubuntu -r trusty -a amd64

And you’ll get a new container running the latest build of Ubuntu 14.04 amd64.

Using unprivileged LXC

Right, so let’s get you started, as I already mentioned, all the instructions below have only been tested on a very recent Ubuntu 14.04 (trusty) installation.
You may want to grab a daily build and run it in a VM.

Install the required packages:

  • sudo apt-get update
  • sudo apt-get dist-upgrade
  • sudo apt-get install lxc systemd-services uidmap

Then, assign yourself a set of uids and gids with:

  • sudo usermod --add-subuids 100000-165536 $USER
  • sudo usermod --add-subgids 100000-165536 $USER
  • sudo chmod +x $HOME

That last one is required because LXC needs it to access ~/.local/share/lxc/ after it switched to the mapped UIDs. If you’re using ACLs, you may instead use “u:100000:x” as a more specific ACL.

Now create ~/.config/lxc/default.conf with the following content:

lxc.network.type = veth
lxc.network.link = lxcbr0
lxc.network.flags = up
lxc.network.hwaddr = 00:16:3e:xx:xx:xx
lxc.id_map = u 0 100000 65536
lxc.id_map = g 0 100000 65536

And /etc/lxc/lxc-usernet with:

<your username> veth lxcbr0 10

And that’s all you need. Now let’s create our first unprivileged container with:

lxc-create -t download -n p1 -- -d ubuntu -r trusty -a amd64

You should see the following output from the download template:

Setting up the GPG keyring
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created an Ubuntu container (release=trusty, arch=amd64).
The default username/password is: ubuntu / ubuntu
To gain root privileges, please use sudo.

So looks like your first container was created successfully, now let’s see if it starts:

ubuntu@trusty-daily:~$ lxc-start -n p1 -d
ubuntu@trusty-daily:~$ lxc-ls --fancy
NAME  STATE    IPV4     IPV6     AUTOSTART  
------------------------------------------
p1    RUNNING  UNKNOWN  UNKNOWN  NO

It’s running! At this point, you can get a console using lxc-console or can SSH to it by looking for its IP in the ARP table (arp -n).

One thing you probably noticed above is that the IP addresses for the container aren’t listed, that’s because unfortunately LXC currently can’t attach to an unprivileged container’s namespaces. That also means that some fields of lxc-info will be empty and that you can’t use lxc-attach. However we’re looking into ways to get that sorted in the near future.

There are also a few problems with job control in the kernel and with PAM, so doing a non-detached lxc-start will probably result in a rather weird console where things like sudo will most likely fail. SSH may also fail on some distros. A patch has been sent upstream for this, but I just noticed that it doesn’t actually cover all cases and even if it did, it’s not in any released version yet.

Quite a few more improvements to unprivileged containers are to come until the final 1.0 release next month and while we certainly don’t expect all workloads to be possible with unprivileged containers, it’s still a huge improvement on what we had before and a very good building block for a lot more interesting use cases.

About Stéphane Graber

Project leader of Linux Containers, Linux hacker, Ubuntu core developer, conference organizer and speaker.
This entry was posted in Canonical voices, LXC, Planet Ubuntu and tagged . Bookmark the permalink.

111 Responses to LXC 1.0: Unprivileged containers [7/10]

  1. kevin wilson says:

    Hi,
    What do you mean by :
    >This declares that the user “stgraber” is allowed up to 10 veth type devices to be created and added to the bridge called lxcbr0.

    Is it that user stgraber can create 10 containers, and in each container there will be a single veth interface, and if he will try to start an eleventh container which is configured with a single veth this will fail ?

    Regards
    Kevin

    1. It means you can have up to 10 interfaces on the host that are bridged in lxcbr0.

      Whether you use all 10 for a single container or you use a single one per container doesn’t matter.

      Since that setting is per-bridge and there’s little point in having two interfaces in the same bridge for a given container, that typically means 10 containers.

      Containers then can do whatever they want with their network inside them and no restriction is applied to that (other than any limit the kernel may have).

  2. Jeremiah says:

    Hello Stéphane,

    Thanks for the great instructional posts.

    Can you add Centos 5 to your image repository?

    Thanks!

  3. kevin wilson says:

    Hi
    Yet I have another question about another point in your blog.
    You mention “you somehow get access to any host resource”:
    By saying “somehow” this sounds like you are talking about some security breach which is not known or hard to find. Do I get you right ?

    I made the following test – maybe i misunderstand something:
    – created a fedora container as root
    – start the conrainer, not as a daemon.
    Run from within the conrainer –
    mknod /dev/sda5 b majorNumber minorNumber

    mount /dev/sda5 /mnt/sda5

    And then I can access the host filesystem.

    Are there any protections against this ?
    Or are you talking about the case when the container was crreated by a non root and then the behavior will be (maybe) different ?

    Regards
    Kevin

    1. Andrew says:

      This is no longer allowed. I’m on 1.0.1

  4. vasilisc says:

    > sudo usermod –add-subuids 100000-165536 $USER
    sudo usermod --add-sub-uids ?

    1. stgraber@castiana:~$ usermod --help | grep sub
        -v, --add-subuids FIRST-LAST  add range of subordinate uids
        -V, --del-subuids FIRST-LAST  remvoe range of subordinate uids
        -w, --add-subgids FIRST-LAST  add range of subordinate gids
        -W, --del-subgids FIRST-LAST  remvoe range of subordinate gids
      
      1. vasilisc says:

        very strange
        man usermod|col -bx|grep sub
        -v, –add-sub-uids FIRST-LAST
        Add a range of subordinate uids to the users account.
        -V, –del-sub-uids FIRST-LAST
        Remove a range of subordinate uids from the users account.
        –del-sub-uids and –add-sub-uids are specified remove of all subordinate uid ranges happens before any
        subordinate uid ranges are added.
        -w, –add-sub-gids FIRST-LAST
        Add a range of subordinate gids to the users account.
        -W, –del-sub-gids FIRST-LAST
        Remove a range of subordinate gids from the users account.
        –del-sub-gids and –add-sub-gids are specified remove of all subordinate gid ranges happens before any
        subordinate gid ranges are added.

        usermod --help | grep sub
        -v, –add-subuids FIRST-LAST add range of subordinate uids
        -V, –del-subuids FIRST-LAST remvoe range of subordinate uids
        -w, –add-subgids FIRST-LAST add range of subordinate gids
        -W, –del-subgids FIRST-LAST remvoe range of subordinate gids

  5. Paul Thomson says:

    Hey, great work. Can you provide how you build the downloadable ubuntu images? Would be great to build my own release/image server. Thanks a lot.

    1. You can get the actual list of actions from the build logs at https://jenkins.linuxcontainers.org

      Technically, it’s really just a run of the standard template with a fixed container name put into rootfs.tar.xz, then a few plain text metadata files put into the meta.tar.xz tarball.

      1. Paul Thomson says:

        Okay, thanks for the hint.

  6. Very good article! How can we migrate existing privileged container to unprivileged?
    Thanks answer!

    1. There’s no easy way to do that unfortunately, you’d need to update your container config to match that from an unprivileged container, move the container’s directory over to the unprivileged user you want it to run as, then use Serge’s uidshift program to change the ownership of all files.

        1. Yeah, that’s the one.

  7. Alan Pater says:

    A couple of questions. I am running up-to-date Ubuntu 14.04:

    1) I get an error when running lxc-start: lxc_container: Executing ‘/sbin/init’ with no configuration file may crash the host

    2) On priviledged containers, I was able to use --bindhome $LOGNAME to give a logon id the same as my current one rather then ubuntu:ubuntu. Is something like that possible with unpriviledged containers?

    1. Alan Pater says:

      Ok, the lxc-start was my fault, I had a typo in the setup.

      Still wondering how to make the container use my userid rather then ubuntu:ubuntu.

      1. Noah F. SanTsorbutz says:

        I get the same error, but have no typo, AFAICT. Please describe what you found, and how you fixed it.

  8. Andy Johnson says:

    >1) I get an error when running lxc-start: lxc_container: Executing ‘/sbin/init’ >with no configuration file may crash the host

    I tried today with the latest daily Ubuntu release of 14.04, exactly as in the post, and I got the same error about “Executing ‘/sbin/init’ “.

    Any ideas ?

    Andy

  9. KeeperB5 says:

    Hi Stéphane!

    I followed your guide with the intention of using user account called “lxc” that does not have sudo permissions to create and run containers. After I change from my regular user account to the lxc user account to run lxc-create, I get permission denied issues.

    lxc@ns3095882:~$ lxc-create -t download -n nsd — -d ubuntu -r trusty -a amd64
    WARN: could not reopen tty: Permission denied
    lxc_container: Error opening /tmp/1001/lxc//srv/lxc/unprivileged/.local/share/lxc/nsd
    lxc_container: failed to save starting configuration for nsd
    lxc_container: Error creating container nsdlxc@ns3095882:~$: command not found
    lxc@ns3095882:~$ lxc-create: Permission denied – failed to create directory ‘/run/user/1000/lock/’

    So I wonder, if I followed your guide, is sudo still needed to run lxc-create?

    1. Robin Harvey says:

      Hi,

      I had the same issue and solved it by SSH’ing in to the host machine as the lxc user directly. It seems that using sudo (either as sudo -i -u lxc or sudo -i and then su lxc) doesn’t work because there’s still some reference to your original user’s $UID hanging around, this gets picked up and confuses things.

      HTH.
      –Robin

      1. Chris says:

        Sure, logging in through ssh works but what if you want to auto-start these containers or otherwise start them during boot? Starting them from the user’s crontab (@reboot) has the exact same problem.

        1. Thomas says:

          Hey,

          Does someone has a fix for this? I also want to autostart some unprivileged containers on boot, but have the same problem.

          Thomas

          1. Yves says:

            For me unsetting the XDG_* environment variables did the trick:


            $ unset XDG_RUNTIME_DIR
            $ unset XDG_SESSION_ID

            The warnings about reopening the tty remained, but creation succeeded:


            $ lxc-create -t download -n p1 -- -d ubuntu -r trusty -a amd64
            WARN: could not reopen tty: Permission denied
            WARN: could not reopen tty: Permission denied
            WARN: could not reopen tty: Permission denied
            WARN: could not reopen tty: Permission denied
            Using image from local cache
            Unpacking the rootfs

            ---
            You just created an Ubuntu container (release=trusty, arch=amd64, variant=default)

            To enable sshd, run: apt-get install openssh-server

            For security reason, container images ship without user accounts
            and without a root password.

            Use lxc-attach or chroot directly into the rootfs to set a root password
            or create user accounts

            If you also want to get rid of those, you need to replace the /dev/pts/?? with something writable by the user. The script utility can be used for this (although it is a really crude hack):


            $ script /dev/null
            $ lxc-create -t download -n p2 -- -d ubuntu -r trusty -a amd64
            Using image from local cache
            Unpacking the rootfs

            Nevertheless this only helps me with creation. Starting still raises some issues.

          2. Julian Lam says:

            Thomas, I spent much of today wrestling with lxc trying to figure this out.

            Here are my results: https://gist.github.com/julianlam/4e2bd91d8dedee21ca6f

            In short, use cgm! 🙂

  10. Fabien C. says:

    Hi Stéphane,

    I can’t get user namespace isolation to work here.

    If I add this within the container configuration file:
    lxc.id_map = u 0 100000 65536
    lxc.id_map = g 0 100000 65536

    and then lxc-start the container as root (I’m not trying full unprivileged for now), I get the following error messages:

    lxc-start: No such file or directory – failed to mount ‘/sys/fs/fuse/connections’ on ‘/usr/lib/x86_64-linux-gnu/lxc/rootfs/sys/fs/fuse/connections’
    lxc-start: Permission denied – error unlinking /usr/lib/x86_64-linux-gnu/lxc/rootfs/dev/kmsg
    lxc-start: failed to setup kmsg for ‘m1’
    lxc-start: Permission denied – failed to create directory ‘/usr/lib/x86_64-linux-gnu/lxc/rootfs/lxc_putold’
    lxc-start: Permission denied – failed to create pivotdir ‘/usr/lib/x86_64-linux-gnu/lxc/rootfs/lxc_putold’
    lxc-start: failed to setup pivot root
    lxc-start: failed to set rootfs for ‘m1’
    lxc-start: failed to setup the container
    lxc-start: invalid sequence number 1. expected 2
    lxc-start: failed to spawn ‘m1’

    I don’t know what I’m missing: I think my system is capable of doing this, lxc-usernsexec is also working fine, and the container starts without the lxc.id_map configuration.

    I’m on Debian Wheezy using lxc 1.0.3 which I backported, based on the sid lxc 1.0.0 package (the latter not working either). I’m also using the Debian backported 3.13 kernel.

    Any clue?

    1. ragavaluvijay says:

      Hi Fabien & Stéphane ,
      If I understood correctly from Fabien comment , I am also trying similar method . i am not trying full unprivileged container now. instead i am just trying to change user name space config for the container & start accordingly . so i changed container config with below key values ,
      lxc.id_map = u 0 201000 10
      lxc.id_map = g 0 201000 10

      and created/started the container from root . but not able to start and facing below error
      ***************************************************************
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-1: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-2: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-3: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-4: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-5: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-6: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-7: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-8: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-9: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-10: No such file or directory
      lxc-start: cgfsng.c: cgfsng_create: 1072 No such file or directory – Failed to create /sys/fs/cgroup/systemd//lxc/testecho-11: No such file or directory
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: start.c: lxc_spawn: 1161 failed to set up id mapping
      lxc-start: start.c: __lxc_start: 1353 failed to spawn ‘testecho’
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/systemd//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/blkio//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/perf_event//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/freezer//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/cpu//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/hugetlb//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/devices//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/net_cls//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/memory//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/cpuset//lxc/testecho-12
      newuidmap: uid range [0-10) -> [200000-200010) not allowed
      lxc-start: conf.c: userns_exec_1: 4315 Error setting up child mappings
      lxc-start: cgfsng.c: recursive_destroy: 983 Error destroying /sys/fs/cgroup/pids//lxc/testecho-12
      lxc-start: lxc_start.c: main: 344 The container failed to start.
      lxc-start: lxc_start.c: main: 348 Additional information can be obtained by setting the –logfile and –logpriority options.
      test@test:~$

      ***************************************************************************

      I am using ubuntu 16.04(kernel 4.4.0.34) which is systemd init system and also i changed below key value to start systemd init. however the same issue is seen for init.d(systemV) init also
      lxc.init_cmd=/lib/systemd/systemd

      ********************************************************************
      i also requested support for the same in below forum, waiting for feed back
      http://unix.stackexchange.com/questions/305428/uid-gid-privileged-lxc-container-systemd-lxc-start-failed-on-ubuntu-16-04

      and

      https://ubuntuforums.org/showthread.php?t=2334953

      if you know any solution already , let me know

      1. ragavaluvijay says:

        i am using lxc 2.0.3 version
        and , lxc 2.0.3 Man page of lxc.container.conf says range is from 200000 to 220000 , but same man page example configured as below
        lxc.id_map = u 0 100000 10000
        lxc.id_map = g 0 100000 10000
        please let me know which is correct

  11. Alan Pater says:

    Undo?

    Is there a way to undo the uid configuration?

    I suspect that changing the uid’s is not compatible with the recent change to cgmanager (from cgroup-lite) in Ubuntu 14.04. At least on this system, the depreciation and automatic removal of cgroup-lite results in all kinds of weird behaviour.

  12. wonderwoman says:

    A quick question… when doing useradd, sub uid/gids are enumerated, and then if I do usermod some more, eg.:

    cat /etc/subuid
    test1:150000:50001
    test1:165537:65536
    test2:231073:65536
    test2:150000:50001

    SHould there not be constraints or is this as expected? I am just messing around to see if the logic is consistent and I am not sure. Now two users map same subuids and also each user has redundant (in this case) mappings….

  13. Malina says:

    Heya.. I notice that if one starts a container (and daemonising it, although I am not sure if that has an effect); and attaches to it, the user’s env is preserved , even if one is “within the container, as root”

    eg. if one does cp something ~, it sets home to /home/outsidecontaineruser, rather than root 😮

    So cd -> can’t change to directory, (as I don’t have the same user in the container)… this surely is a bug? Everything is swell, if one ssh’s into the box rather.

  14. Jennifer Bell says:

    Hi,

    Thanks for your great instructions. They were invaluable in getting unprivileged LXCs working with Ubuntu 14.04. I have one problem though that I haven’t been able to fix. I sometimes have two bridges that I put LXC containers on, and often one container will be on both bridges in order to route traffic between the subnets. With regular old LXC containers running as root this was never a problem, but when I try to specify two veth interfaces with unprivileged containers, I’m unable to start the container. It always shows the following message:

    $ lxc-start -d -n mycontainer
    lxc_container: command get_cgroup failed to receive response

    This is how I want the networking to be configured and it works fine in a regular privileged LXC:

    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br0
    lxc.network.hwaddr = 00:16:3e:c4:b4:ca
    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br1
    lxc.network.hwaddr = 00:16:3e:35:fe:6a

    If I comment out either one of the veth interfaces, then it works:

    #lxc.network.type = veth
    #lxc.network.flags = up
    #lxc.network.link = br0
    #lxc.network.hwaddr = 00:16:3e:c4:b4:ca
    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br1
    lxc.network.hwaddr = 00:16:3e:35:fe:6a

    or:

    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br0
    lxc.network.hwaddr = 00:16:3e:c4:b4:ca
    #lxc.network.type = veth
    #lxc.network.flags = up
    #lxc.network.link = br1
    #lxc.network.hwaddr = 00:16:3e:35:fe:6a

    my /etc/lxc/lxc-usernet file looks like this:

    $ cat /etc/lxc/lxc-usernet
    # USERNAME TYPE BRIDGE COUNT
    lxcuser veth br0 10
    lxcuser veth br1 10

    There are no other containers running at the time I try to start this one, and anyway, I don’t believe I am hitting the limit referenced in /etc/lxc/lxc-usernet since either interface can work alone.

    Are there any existing problems that you know of preventing the use of two veth interfaces in one unprivileged LXC container? Or, are there any additional configuration changes required to LXC or cgroups required to make it work? Or am I missing something else very basic?

    Thanks,
    Jennifer

    1. Jennifer Bell says:

      Yay! I found the solution. Apparently you are allowed to specify a name (like eth0, eth1) in the set of network config lines, and it didn’t used to be required for multiple interfaces but apparently is in this case. This works perfectly:

      lxc.network.type = veth
      lxc.network.flags = up
      lxc.network.link = br0
      lxc.network.hwaddr = 00:16:3e:c4:b4:ca
      lxc.network.name = eth0
      lxc.network.type = veth
      lxc.network.flags = up
      lxc.network.link = br1
      lxc.network.hwaddr = 00:16:3e:35:fe:6a
      lxc.network.name = eth1

  15. Arthur says:

    Hi,

    Thanks for this great post, however is it possible to use “phys” network type with unprivileged containers?

    I have created an unprivileged container with below config without any issue but the 2nd/phys interface isn’t there after the container is started –

    ~$ cat .config/lxc/eo1.conf
    lxc.network.type = veth
    lxc.network.flags = up
    lxc.network.link = br0
    lxc.network.hwaddr = 00:16:3e:xx:xx:xx
    lxc.network.name = eth0

    lxc.network.type = phys
    lxc.network.flags = up
    lxc.network.link = eth5
    lxc.network.hwaddr = 00:1b:21:00:00:01
    lxc.network.ipv4 = 192.168.56.1/24
    lxc.network.name = eth1

    lxc.id_map = u 0 100000 65536
    lxc.id_map = g 0 100000 65536

    Any advice is appreciated.

  16. Ivan Ogai says:

    In a private container, I have tried to enable FUSE, but when trying to mount with sshfs I get the error: fusermount: mount failed: Operation not permitted

    The fuse device is present in the container and has the proper permissions.

    I have this in its config file:

    lxc.cgroup.devices.allow = c 10:229 rwm
    lxc.mount.entry = /dev/fuse dev/fuse none bind,optional,create=file
    lxc.loglevel = 2
    lxc.logfile = /home/ivan/.local/share/lxc/wuala/lxc.log
    lxc.cap.keep = CAP_SYS_ADMIN

    In the host I have added following line to /etc/apparmor.d/lxc/lxc-default.

    mount fstype=fuse options=(rw, bind, ro, nosuid, nodev, user),

    Unfortunately nothing is logged in the lxc.log file (not anywhere else either), and the -d option in sshfs doesn’t output more than without.

  17. Richard says:

    hi,

    thanks for your work, thats awesome especialy for unprivilege containers.

    i want to build a server like images.linuxcontainers.org, i read the post and you wrote that all stuff in in the jenkins output, but it s not very clear.

    i understand we build a root.tar.xz, and meta.tar.xz from images folders , and also the meta folders with descritpions.

    can you explain or put the jenkins config somewhere? it will be easier with sources than with output

  18. Christoph Willing says:

    A patched PAM is mentioned as a prerequisite. Can we presume that that prerequisite could be ignored for non-PAM systems?

    chris

  19. Garrett says:

    I am having an issue when starting the container:

    gauthig@main:~$ lxc-start -n test
    chown: changing ownership of `/dev/pts/10′: Operation not permitted
    lxc_container: Failed to chown /dev/pts/10
    lxc_container: Failed to shift tty into container
    lxc_container: failed to initialize the container
    lxc_container: The container failed to start.

    I have walked through the pre-requisites several times and ensured all steps completed. Any ideals?
    Host is 14.04

    1. Garrett says:

      Found a solution from Chris at Linuxquestions.org:
      Something in kernel updates after 3.14.5 broke it, but lxc 1.0.5 resolved it. The base Trusty LXC was 1.0.4.

  20. Ashvin Goel says:

    Hi,

    I was wondering if there is any security benefit with dropping Linux capabilities with unprivileged lxc containers, and whether it is possible to do so.

    I tried dropping some capabilities, but the output of /proc/$$/status in a container shows the following, which seems to suggest that all the capabilities are enabled.

    CapInh: 0000000000000000
    CapPrm: 0000001fffffffff
    CapEff: 0000001fffffffff
    CapBnd: 0000001fffffffff

    Thanks
    Ashvin

  21. norwood sisson says:

    could you publish a template for an ephemeral, unprivileged container for a gui which would start when a library patron logged in

  22. Something is wrong…

    I made the Ubuntu lucid unprivileged container with Ubuntu 14.04.1 (LXC 1.0.5):

    $ lxc-create -t download -n ubuntu_10.04 — -d ubuntu -r lucid -a amd64
    Setting up the GPG keyring
    Downloading the image index
    Downloading the rootfs
    Downloading the metadata
    The image cache is now ready
    Unpacking the rootfs


    You just created an Ubuntu container (release=lucid, arch=amd64, variant=default)
    The default username/password is: ubuntu / ubuntu
    To gain root privileges, please use sudo.

    And container start…
    $ lxc-start -n ubuntu_10.04
    init: hostname main process (4) terminated with status 1
    init: hwclock main process (6) terminated with status 77
    init: ureadahead main process (8) terminated with status 5
    init: ureadahead-other main process (58) terminated with status 4
    mount: mount point /dev/shm is a symbolic link to nowhere
    mountall: mount /dev/shm [59] terminated with status 32
    mountall: Filesystem could not be mounted: /dev/shm

    Ubuntu 14.04 is fresh install.

    1. Workaround:

      – Delete /dev/shm inside container.
      – Create /run/shm inside container.
      – Symlink: /dev/shm -> /run/shm (inside container)

      1. Hm, sshd does not work 🙁

        $ lxc-start -n ubuntu_10.04
        init: hostname main process (4) terminated with status 1
        init: hwclock main process (6) terminated with status 77
        init: ureadahead main process (8) terminated with status 5
        init: ureadahead-other main process (58) terminated with status 4
        init: console-setup main process (60) terminated with status 1
        init: procps (virtual-filesystems) main process (61) terminated with status 255
        init: Failed to spawn ssh pre-start process: unable to set oom adjustment: Permission denied
        udev: starting version 151

        Ubuntu 10.04.4 LTS ubuntu_10.04 /dev/console

      2. Esa-Matti Suuronen says:

        I was able to get Lucid container to boot by just removing the /dev/shm symlink from the container.

  23. anonima says:

    In debian 7 doesn’t exist uidmap. Any solution for unprivileged container in D7? Thank you

  24. Ovanes says:

    I just wonder if there is a way to clone from a global/privileged LXC container into unprivileged one? I currently use Ubuntu 14.04.1 with LXC 1.0.5 on it.

  25. Adam says:

    Does sudo work on unprivileged lxc container?

    After adding a user with adduser adam sudo and logging into the container with ssh, when I tried to do “sudo su” I’ve got failure with a responce:
    adam@p1:~$ sudo su
    sudo: effective uid is not 0, is /usr/bin/sudo on a file system with the ‘nosuid’ option set or an NFS file system without root privileges?

    Is it by design, or is it a bug?
    Anyway, I filed a bug against ubuntu on https://bugs.launchpad.net/ubuntu/+source/lxc/+bug/1389305

  26. Badiane says:

    I use macvlan’s for my interfaces. It’s been a while since I’ve built any new containers or upgrade the ones I’m using. How will the user namespace affect macvlan based interfaces?

  27. EY says:

    Any pre-buildt image for Arch linux ?

  28. joda says:

    On Ubuntu 14.04 I encountered the error message(s):
    lxc_container: The container failed to start.
    And from the logs:
    lxc-start ERROR lxc_cgmanager – call to cgmanager_create_sync failed: invalid request
    lxc-start ERROR lxc_cgmanager – Failed to create hugetlb:u1
    lxc-start ERROR lxc_cgmanager – Error creating cgroup hugetlb:u1

    Root cause my cgroups are not setup if you’re logging in over ssh. If your cgroups look like this:
    lxc@xen-jd1:~$ cat /proc/self/cgroup
    11:hugetlb:/
    10:perf_event:/
    9:blkio:/
    8:freezer:/
    7:devices:/
    6:memory:/
    5:cpuacct:/
    4:cpu:/
    3:cpuset:/
    2:name=systemd:/

    Check if you’ve somehow disabled or uninstalled the PAM module (libpam-systemd) used to setup cgroups. My problem was I disabled PAM for SSH (/etc/ssh/sshd_config):
    “UsePAM No”

  29. anon says:

    Thanks for the article!
    On debian jessie amd64 (with systemd) as a host, I got the following errors:
    unshare: Operation not permitted
    read pipe: No such file or directory
    lxc-create: Failed to chown container dir
    lxc-create: Error creating container tor
    ———————
    lxc-create 1422657281.890 WARN lxc_log – lxc_log_init called with log already initialized
    lxc-create 1422657281.891 INFO lxc_confile – read uid map: type u nsid 0 hostid 100000 range 65536
    lxc-create 1422657281.891 INFO lxc_confile – read uid map: type g nsid 0 hostid 100000 range 65536
    lxc-create 1422657281.894 ERROR lxc_container – Failed to chown container dir
    lxc-create 1422657281.895 ERROR lxc_create_ui – Error creating container tor

    This was solved by:
    echo 1 > /sys/fs/cgroup/cpuset/cgroup.clone_children
    echo 1 > /proc/sys/kernel/unprivileged_userns_clone

    Also I needed to apt-get install lxc libpam-systemd cgroup-bin cgmanager
    All of this only worked when in an login shell, sudo or su failed somehow.

    1. anon says:

      :/ sorry for the false info, cgmanager was causing problems with systemd. Removed that.

      After all these steps cgroups were still giving errors:
      lxc-start: Permission denied - Could not create cgroup '/tor' in '/sys/fs/cgroup/perf_event'.
      lxc-start: Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/perf_event/
      lxc-start: Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/blkio/
      lxc-start: Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/net_cls,net_prio/
      lxc-start: Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/
      lxc-start: Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/user.slice
      lxc-start: Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu,cpuacct/
      lxc-start: Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/
      lxc-start: failed creating cgroups

      This was fixed by manually changing the directory structure with the following script:
      https://www.mail-archive.com/lxc-devel@lists.linuxcontainers.org/msg01660.html

      After this the container starts as well.
      PS: I think I also needed to do `service apparmor stop` while creating.

  30. ¯(°_o)/¯ says:

    Is it right that for unprivileged LXC I need to start above “lxc-create” and “lxc-start” commands under my user (not root)? The problem is that I don’t see such commands under user in Gentoo (only root sees them). Is it a bug or there’s a way to start unpriviledged LXC under root?

    1. ¯(°_o)/¯ says:

      It looks like I was missing /usr/sbin inside PATH. After adding it I created unprivileded lxc under user.

  31. Eugene says:

    I have followed the steps as described and i got the container up and running. Th eonly problem is that I can’t login via lxc-console and i can’t SSH to the container. I’m not sure what is the username/password that I should be using here?
    I tired the same pair as the user that starts the container but the console does’t accept it.
    The SSH doesn’t work at all –
    ~ $ lxc-start -n c1
    ~ $ lxc-ls –fancy
    NAME STATE IPV4 IPV6 GROUPS AUTOSTART
    ————————————————-
    c1 RUNNING 10.0.3.57 – – NO
    p1 STOPPED – – – NO
    ~ $ ssh 10.0.3.57
    ssh: connect to host 10.0.3.57 port 22: Connection refused

    1. Eugene says:

      I probably should add that i used the download template and selected the ubuntu utopic i386 architecture

  32. Manuel says:

    Hi Stéphane,

    it seems that the latest Debian/Jessie templates are not usable as unprivileged containers (at least they are not shown as candidates with –list, and cannot be installed). Might be worthwhile pointing out that systemd based distros (such as Debian/Jessie) don’t work with unprivileged containers.

    1. They actually will appear in the list if you’re running LXC 1.1 or higher.

      The current list on my system (running as an unprivileged user) is:


      ---
      DIST RELEASE ARCH VARIANT BUILD
      ---
      centos 6 amd64 default 20150422_02:16
      centos 6 i386 default 20150420_02:16
      debian jessie amd64 default 20150421_22:42
      debian jessie armel default 20150420_22:42
      debian jessie armhf default 20150421_22:42
      debian jessie i386 default 20150420_22:42
      debian sid amd64 default 20150421_22:42
      debian sid armel default 20150421_22:42
      debian sid armhf default 20150421_22:42
      debian sid i386 default 20150421_22:42
      debian wheezy amd64 default 20150419_22:42
      debian wheezy armel default 20150421_22:42
      debian wheezy armhf default 20150420_22:42
      debian wheezy i386 default 20150421_22:42
      gentoo current amd64 default 20150421_14:12
      gentoo current armhf default 20150422_14:12
      gentoo current i386 default 20150421_14:12
      oracle 6.5 amd64 default 20150421_11:40
      oracle 6.5 i386 default 20150419_11:40
      plamo 5.x amd64 default 20150421_21:36
      plamo 5.x i386 default 20150420_21:36
      ubuntu precise amd64 default 20150421_03:49
      ubuntu precise armel default 20150421_03:49
      ubuntu precise armhf default 20150422_03:49
      ubuntu precise i386 default 20150422_03:49
      ubuntu trusty amd64 default 20150420_03:49
      ubuntu trusty armhf default 20150420_03:49
      ubuntu trusty i386 default 20150422_03:49
      ubuntu trusty ppc64el default 20150422_03:49
      ubuntu utopic amd64 default 20150420_03:49
      ubuntu utopic armhf default 20150422_03:49
      ubuntu utopic i386 default 20150422_03:49
      ubuntu utopic ppc64el default 20150422_03:49
      ubuntu vivid amd64 default 20150422_03:49
      ubuntu vivid armhf default 20150422_03:49
      ubuntu vivid i386 default 20150421_03:49
      ubuntu vivid ppc64el default 20150422_03:49

      1. itsso says:

        Stéphane, would you be able to help or point me to a location where I can find information about the following issue. Thank you.

        I have been running for months multiple ubuntu trusty and utopic unprivileged containers created with the download template. Today I tried to create a vivid machine but the create command failed. It seems the issue is that now I am only able to access the trusty and precise images, even utopic is gone, when running lxc-create as unprivileged user. Root is able to see all ubuntu images. The host is ubuntu 14.04.3, fully patched.

        user@ubuntu:~$ lxc-create -n x-vivid -t download -- --flush-cache -d ubuntu
        Setting up the GPG keyring
        Downloading the image index
        ---
        DIST RELEASE ARCH VARIANT BUILD
        ---
        ubuntu precise amd64 default 20151019_03:49
        ubuntu precise armel default 20151018_03:49
        ubuntu precise armhf default 20151018_03:49
        ubuntu precise i386 default 20151019_03:49
        ubuntu trusty amd64 default 20151019_03:49
        ubuntu trusty arm64 default 20150604_03:49
        ubuntu trusty armhf default 20151019_03:49
        ubuntu trusty i386 default 20151019_03:49
        ubuntu trusty ppc64el default 20151019_03:49
        ---
        Release:

  33. grobbelaar says:

    Hi, a few questions:

    1) do I need to have a unique uid/gid range in each container?

    2) can you describe in details how to convert a basic (privileged) container to unprivileged one?

    thanks a lot!

  34. tlangner says:

    Hey Stephane,
    thanks a lot for your great guide. I managed to set everything up as you said but I have one problem left and that concerns the access rights of files created inside the container. Ideally, I would love to be able to create the files such that they are owned by the user whose uid got mapped into the container through the virtual IDs, call him $USER with $UID. However, by design, I do not have access to $UID from within the container. Moreover, from outside of the container, I need root access to chown the files as they are not owned by $UID but rather by some virtual $UID. Is there any way how I can accomplish my goal?

    Thanks in advance, best, Tobias

  35. COLABORATI says:

    How can I install a template for running alpine linux in an unprivileged container?

  36. COLABORATI says:

    How to mount directories from the host into an unprivileged container?
    I just added a line like this

    lxc.mount.entry = /home/user/sites home/ubuntu/sites none bind 0 0

    to the file

    /home/user/.local/share/lxc/ubuntu-14-dev/config

    it seems to work, however all files and directories belong to nobody/nogroup – this makes usage a little bit complicated 🙂

    How do I get the mapping right? I tried to figure, but got lost.
    I want the sites directory to be the same owner/group like on the host.

    Security is not of concern, this is a dev and test environment, however I would like to have a shared directory.

    And thank you very much for your nice guides!

  37. COLABORATI says:

    GOT IT! This works:

    lxc.id_map = u 0 100000 1000
    lxc.id_map = g 0 100000 1000
    lxc.id_map = u 1000 1000 1
    lxc.id_map = g 1000 1000 1
    lxc.id_map = u 1001 101001 64535
    lxc.id_map = g 1001 101001 64535

    lxc.mount.entry = /home/user/sites home/ubuntu/sites none bind 0 0

  38. Noah F. San Tsorbutz says:

    It would be welcome to see less cryptic examples. Since Linux/Unix allow colons in file names, the one at the end of this comment makes it seem as if there should be two files found in the /etc/ directory named:

    subgid:user_name:100000:65536
    subuid:user_name:100000:65536

    rather than two files named:

    subgid
    subuid

    with identical contents:

    user_name:100000:65536

    It took quite a bit of time to grok through that, and figure out what wasn’t configured correctly. A few of your readers may be Unix zen gurus, but as for “the rest of us”, well …

    However, thanks for the effort, any help getting LXC working is valued, since it doesn’t exactly run itself!

    stgraber@castiana:~$ grep stgraber /etc/sub* 2>/dev/null
    /etc/subgid:stgraber:100000:65536
    /etc/subuid:stgraber:100000:65536

  39. chris says:

    Cheers Stephane,

    Used this page as a reference a few times now.

    Chris

  40. francois bussery says:

    Strange, it seems that in unprivileged containers, cap.keep & cap.drop is not working. Seems impossible to enable sys_admin & net_admin.
    Is it a feature?

  41. Art says:

    What happened to the Lucid container, I’m trying to migrate and old system to trusty and need lucid as an interim container. If you no longer have a copy, how can I build the container myself.
    Thanks, Art

  42. Jason says:

    When I download a debian wheezy amd64 lxc container and log in from debian host to unprivileged container, when I run `cd` from the initial root directory, it says,

    bash: cd: /home/my_host_username: No such file or directory

    Why is it trying `cd` into my host username?

    What separation logic am I missing that isn’t getting added.

    1. Unless you pass –clear-env to lxc-attach, your environment is kept when attaching into the container, which in this case includes the HOME environment variable.

  43. Csaba Dobo says:

    Hi,

    I just started experimenting with lxc. I installed ganesha-nfs in there and would like non container-based hosts to mount a share, shared from a container.
    I am running into errors, like this:
    client side:
    http://pastebin.com/sUi5ZHLc

    root@virt0:/mnt/nfs# mount.nfs -vvv 172.19.8.25:/home/exports/company2 /mnt/nfs
    mount.nfs: timeout set for Thu Mar 31 15:22:54 2016
    mount.nfs: trying text-based options ‘vers=4,addr=172.19.8.25,clientaddr=172.19.8.131’
    mount.nfs: mount(2): Operation not permitted
    mount.nfs: trying text-based options ‘addr=172.19.8.25’
    mount.nfs: prog 100003, trying vers=3, prot=6
    mount.nfs: trying 172.19.8.25 prog 100003 vers 3 prot TCP port 2049
    mount.nfs: prog 100005, trying vers=3, prot=17
    mount.nfs: trying 172.19.8.25 prog 100005 vers 3 prot UDP port 33487
    mount.nfs: mount(2): Invalid argument
    mount.nfs: an incorrect mount option was specified

    tail -f /var/log/syslog


    Mar 31 15:22:49 virt0 kernel: [2504001.096982] NFS reply getattr: -22
    Mar 31 15:22:49 virt0 kernel: [2504001.096984] nfs_create_server: getattr error = 22

    server side:
    http://fpaste.org/348098/49704014/
    so severeal lines like this: vfs_open_by_handle :FSAL :DEBUG :vfs_fs = / root_fd = 5
    vfs_open_by_handle :FSAL :DEBUG :Failed with Operation not permitted openflags 0x00000000
    vfs_fsal_open_and_stat :FSAL :DEBUG :Failed with Operation not permitted open_flags 0x000
    [ ands with this: getattrs :FSAL :DEBUG :Failed with Operation not permitted, fsal_error other
    martinetd: so what does this tell you, what do I need to set?
    here is the complete log: http://fpaste.org/348098/49704014/

    Someone suggested to set some capabilities-wise, but I have no clue as to what should I set.

    ould you direct me to the right direction?
    thx
    Csaba Dobo

  44. jtlpa says:

    Debian Jessie 8.4 64b, LXC 1.0.6-6+deb8u2 unprivileged container issue

    Can create manage and use Privileged container just fine using this syntax:
    # lxc-create -t debian -n test

    Can create UnPrivileged container w/o any error using this syntax:
    # lxc-create -t download -n test2 — -d debian -r jessie -a amd64

    When logged in as ROOT can use test2 privileged container just fine.
    # lxc-start -n test2 -d

    However,when logged in as USER and
    $ lxc-start -n test2 -d
    lxc_container: Executing ‘/sbin/init’ with no configuration file may crash the host

    Have tried the latest debian backports version of LXC, which chokes all over the place so that is not an option.

    As far as I can tell this IS NOT a syntax issue, CONFIG file issue or setup issue. Have verified ALL required files are available in the container folder and $home folders.

    Have spent days Googling for a solution to no avail. Would be grateful for any debugging solutions.

    1. JTL says:

      After even more Googeling found these instruction will not work for Debian Jessie 8.3 64bit.

      See this article for more details:
      https://myles.sh/configuring-lxc-unprivileged-containers-in-debian-jessie/

  45. Ludwig says:

    Hi,

    I followed a manual to create unprivileged containers for a common user. For each container I use another user to own the container. So user0 uses 100000-165536 and user1 uses 200000-265536 and so forth. I do this to keep things as separate as possible. However this requires me to actually login as each of these users to control its container (su won’t do for some reason). Also it’s not possible to do a simple “lxc-ls” as root and actually see all containers, also autostart is a bit of a hack right now.
    Is it possible to do same as root? Can root use containers but put each container in another namespace? This would be much simpler than doing it the way I’m doing it now.

    regards

    P.S.
    I followed these instructions
    https://myles.sh/configuring-lxc-unprivileged-containers-in-debian-jessie/

    1. Ludwig says:

      so I just tried it an it works…
      I simply created another default.conf called default1.conf and passed to lxc-create using -f
      that was way to easy…
      now I have two containers, one residing in 100000 and one residing in 200000
      I will run some more tests but I think I can kick my old solution :3

  46. Artem says:

    Hi.

    Sorry for my English.

    I have strange situation on server (Ubuntu 16.04.1 LTS) with bunch of unprivileged containers (Ubuntu 16.04.1 LTS) running.

    Аfter a while all processes moved from lxc cgroup structure to system cgroup structure.

    For example:

    At start container pids locate in:
    /sys/fs/cgroup/memory/user.slice/user-0.slice/session-2.scope/hoppyowl/lxc/hoppyowl.localhost/tasks

    Аfter a while they appear in:
    /sys/fs/cgroup/memory/user.slice/user-0.slice/session-2.scope/tasks

    Any ideas?

  47. Reginald says:

    Hi,

    I was confused about the description, In the article, u use unprivileged user “stgraber” to create container, but if I use root and create such things, Does it still work? In the final task, as u said, lxc-attach can’t use any more, how about a new version of lxc?

  48. Reginald says:

    Hi,
    While I’m trying to start an unprivileged container, I meet the same problem like https://github.com/lxc/lxcfs/issues/106. And trying to following instruction above, and also I add the modify of commit c30b61c37695a88f07ed5641ad270e4785a3a918, but unfortunately, It doesn’t work either.
    Before I add commit c30b61c37695a88f07ed5641ad270e4785a3a918, log as follows:
    lxc-start 20170105182841.368 ERROR lxc_cgfsng – cgfsng.c:all_controllers_found:431 – no systemd controller mountpoint found
    lxc-start 20170105182841.380 ERROR lxc_start – start.c:lxc_spawn:1079 – failed initializing cgroup support
    lxc-start 20170105182841.380 ERROR lxc_start – start.c:__lxc_start:1329 – failed to spawn ‘u4’
    lxc-start 20170105182846.385 ERROR lxc_start_ui – lxc_start.c:main:344 – The container failed to start.
    lxc-start 20170105182846.396 ERROR lxc_start_ui – lxc_start.c:main:346 – To get more details, run the container in foreground mode.
    lxc-start 20170105182846.400 ERROR lxc_start_ui – lxc_start.c:main:348 – Additional information can be obtained by setting the –logfile and –logpriority options.
    After I add the commit, error as follows:
    lxc-start 20170105183141.716 ERROR lxc_cgfsng – cgfsng.c:all_controllers_found:437 – no freezer controller mountpoint found
    lxc-start 20170105183141.719 ERROR lxc_start – start.c:lxc_spawn:1079 – failed initializing cgroup support
    lxc-start 20170105183141.719 ERROR lxc_start – start.c:__lxc_start:1329 – failed to spawn ‘u4’
    lxc-start 20170105183146.722 ERROR lxc_start_ui – lxc_start.c:main:344 – The container failed to start.
    lxc-start 20170105183146.742 ERROR lxc_start_ui – lxc_start.c:main:346 – To get more details, run the container in foreground mode.
    lxc-start 20170105183146.744 ERROR lxc_start_ui – lxc_start.c:main:348 – Additional information can be obtained by setting the –logfile and –logpriority options.
    So, I’m trying to add some debug info on src/lxc/cgfsng.c, also I revert the modify of commit c30b61c37695a88f07ed5641ad270e4785a3a918, and I found that:
    In src/lxc/cfgsng.c
    static bool parse_hierarchies(struct cgfsng_handler_data* d){

    if (!test_writeable(mountpoint, base_cgroup)) {

    // The debug info I add;
    ERROR(“No access write ‘%s'”, mountpoint);
    free(mountpoint);

    }

    }
    When I use non-root user to start unprivileged container, It’ll print the ERROR msg, so In my opinion, I think this is the reason.
    Does anyone know what should I do to fix it.

  49. Florian Haas says:

    As of January 2017, this post is still the top Google hit for the search string “LXC unprivileged containers”, but as you were kind enough to point out in https://bugs.launchpad.net/ubuntu/+source/lxcfs/+bug/1655932/comments/1, a lot of its content is obsolete as per LXC 2.0.

    Would you mind adding a notice that what’s described here largely only applies to LXC 1.0? I’m referring, in particular, to the bits about the directory structure in $HOME, and the invocations of lxc-create, lxc-start etc. as a non-root user.

  50. Nicholas says:

    Great article!
    I’ve managed to create and run Debian and Ubuntu containers inside a Ubuntu VM, but I can’t do it inside Debian 8 Jessie.
    lxc-start says it can’t mount rootfs, why?

  51. ordenadores says:

    hi ! Heya.. I notice that if one starts a container (and daemonising it, although I am not sure if that has an effect); and attaches to it, the user’s env is preserved , even if one is “within the container, as root”

    eg. if one does cp something ~, it sets home to /home/outsidecontaineruser, rather than root ?

    So cd -> can’t change to directory, (as I don’t have the same user in the container)… this surely is a bug? Everything is swell, if one ssh’s into the box rather. bye!

  52. qwerty says:

    I am getting this error, can you please help
    lxc-start: myunprivileged: lxccontainer.c: wait_on_daemonized_start: 842 Received container state “ABORTING” instead of “RUNNING”
    lxc-start: myunprivileged: tools/lxc_start.c: main: 330 The container failed to start
    lxc-start: myunprivileged: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
    lxc-start: myunprivileged: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the –logfile and –logpriority options

  53. qwerty says:

    lxc-start: mycontainer: lxccontainer.c: wait_on_daemonized_start: 842 Received container state “ABORTING” instead of “RUNNING”
    lxc-start: mycontainer: tools/lxc_start.c: main: 330 The container failed to start
    lxc-start: mycontainer: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
    lxc-start: mycontainer: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the –logfile and –logpriority options

  54. In the end LXC overthrew VZ. As Proxmox did in moving from VZ to LXC, a big step, as these are more practical, secure and require fewer resources.
    Thank you.

  55. Andrew Odigie says:

    Hi Stephane,

    Thank you for your well detailed steps. it is really helpful for a beginner like myself. I tried the step on lab I was doing but I got the following error seen below.

    ERROR: Invalid signature for /tmp/tmp.dcf4XuwCAL/rootfs.tar.xz.asc
    lxc-create: unpriv-cont-user: lxccontainer.c: create_run_template: 1617 failed to create container from template
    lxc-create: unpriv-cont-user: tools/lxc_create.c: main: 327 Failed to create container unpriv-cont-user

    Do you know what that means and how I can fix this.

    Thank you
    Andrew

Leave a Reply to joda Cancel reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.