LXC 1.0: Container storage [5/10]

This is post 5 out of 10 in the LXC 1.0 blog post series.

Storage backingstores

LXC supports a variety of storage backends (also referred to as backingstore).
It defaults to “none” which simply stores the rootfs under
/var/lib/lxc/<container>/rootfs but you can specify something else to lxc-create or lxc-clone with the -B option.

Currently supported values are:

directory based storage (“none” and “dir)

This is the default backingstore, the container rootfs is stored under
/var/lib/lxc/<container>/rootfs

The --dir option (when using “dir”) can be used to override the path.

btrfs

With this backingstore LXC will setup a new subvolume for the container which makes snapshotting much easier.

lvm

This one will use a new logical volume for the container.
The LV can be set with --lvname (the default is the container name).
The VG can be set with --vgname (the default is “lxc”).
The filesystem can be set with --fstype (the default is “ext4”).
The size can be set with --fssize (the default is “1G”).
You can also use LVM thinpools with --thinpool

overlayfs

This one is mostly used when cloning containers to create a container based on another one and storing any changes in an overlay.

When used with lxc-create it’ll create a container where any change done after its initial creation will be stored in a “delta0” directory next to the container’s rootfs.

zfs

Very similar to btrfs, as I’ve not used either of those myself I can’t say much about them besides that it should also create some kind of subvolume for the container and make snapshots and clones faster and more space efficient.

Standard paths

One quick word with the way LXC usually works and where it’s storing its files:

  • /var/lib/lxc (default location for containers)
  • /var/lib/lxcsnap (default location for snapshots)
  • /var/cache/lxc (default location for the template cache)
  • $HOME/.local/share/lxc (default location for unprivileged containers)
  • $HOME/.local/share/lxcsnap (default location for unprivileged snapshots)
  • $HOME/.cache/lxc (default location for unprivileged template cache)

The default path, also called lxcpath can be overridden on the command line with the -P option or once and for all by setting “lxcpath = /new/path” in /etc/lxc/lxc.conf (or $HOME/.config/lxc/lxc.conf for unprivileged containers).

The snapshot directory is always “snap” appended to lxcpath so it’ll magically follow lxcpath. The template cache is unfortunately hardcoded and can’t easily be moved short of relying on bind-mounts or symlinks.

The default configuration used for all containers at creation time is taken from
/etc/lxc/default.conf (no unprivileged equivalent yet).
The templates themselves are stored in /usr/share/lxc/templates.

Cloning containers

All those backingstores only really shine once you start cloning containers.
For example, let’s take our good old “p1” Ubuntu container and let’s say you want to make a usable copy of it called “p4”, you can simply do:

sudo lxc-clone -o p1 -n p4

And there you go, you’ve got a working “p4” container that’ll be a simple copy of “p1” but with a new mac address and its hostname properly set.

Now let’s say you want to do a quick test against “p1” but don’t want to alter that container itself, yet you don’t want to wait the time needed for a full copy, you can simply do:

sudo lxc-clone -o p1 -n p1-test -B overlayfs -s

And there you go, you’ve got a new “p1-test” container which is entirely based on the “p1” rootfs and where any change will be stored in the “delta0” directory of “p1-test”.
The same “-s” option also works with lvm and btrfs (possibly zfs too) containers and tells lxc-clone to use a snapshot rather than copy the whole rootfs across.

Snapshotting

So cloning is nice and convenient, great for things like development environments where you want throw away containers. But in production, snapshots tend to be a whole lot more useful for things like backup or just before you do possibly risky changes.

In LXC we have a “lxc-snapshot” tool which will let you create, list, restore and destroy snapshots of your containers.
Before I show you how it works, please note that “lxc-snapshot” currently doesn’t appear to work with directory based containers. With those it produces an empty snapshot, this should be fixed by the time LXC 1.0 is actually released.

So, let’s say we want to backup our “p1-lvm” container before installing “apache2” into it, simply run:

echo "before installing apache2" > snap-comment
sudo lxc-snapshot -n p1-lvm -c snap-comment

At which point, you can confirm the snapshot was created with:

sudo lxc-snapshot -n p1-lvm -L -C

Now you can go ahead and install “apache2” in the container.

If you want to revert the container at a later point, simply use:

sudo lxc-snapshot -n p1-lvm -r snap0

Or if you want to restore a snapshot as its own container, you can use:

sudo lxc-snapshot -n p1-lvm -r snap0 p1-lvm-snap0

And you’ll get a new “p1-lvm-snap0” container which will contain a working copy of “p1-lvm” as it was at “snap0”.

About Stéphane Graber

Project leader of Linux Containers, Linux hacker, Ubuntu core developer, conference organizer and speaker.
This entry was posted in Canonical voices, LXC, Planet Ubuntu and tagged . Bookmark the permalink.

23 Responses to LXC 1.0: Container storage [5/10]

  1. Simon Davy says:

    Good stuff.

    Some questions:

    – I tried using btrfs as a backing store ~6mo ago, and had a lot of problems with left-over subvolumes that had to be cleaned up by hand? Has this been addressed in any way in 1.0?

    – How reliable is overlayfs backing store? I recall a co-worker having some problems with things like unix sockets not working properly?

    1. Simon Davy says:

      Found the overlayfs bug, seems unfixed: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1214500

    2. bmullan says:

      Simon
      Using lxc and btrfs does start to create some obvious benefits with cloning or backups of containers and subsequent storage requirements. I’ll be watching for any answers to your question.
      brian

    3. Jeremiah says:

      Hi Simon,

      I’ve been using btrfs as my backing store for the past few months and I have not had any problems like you described.

      One thing I will point out however is a lot of documentation/tutorials out there say LXC automatically uses btrfs as the backing store if the /var/lib/lxc directory is formatted with btrfs. This is no longer true.

      Now you always have to specify btrfs when you lxc-create a container.

      It does however continue to be true that if a container was created with a btrfs backing store then lxc-clone -s will still automatically use btrfs when snapshot cloning.

      https://lists.linuxcontainers.org/pipermail/lxc-devel/2013-August/004928.html

  2. Martin Pitt says:

    Thanks for this series! I actually changed the default container root dir on my box recently as /var/ gets too crammed. It seems in newer versions that option is now called “lxc.lxcpath” instead of “lxcpath”, and neither falls back to the other.

    1. Hi Martin,

      So on the upstream side we technically never had a stable release with lxc.conf so we didn’t bother with migration code.

      For Ubuntu, my feeling was that as lxc.conf wasn’t documented until last week and that we weren’t shipping one by default, we could get away without migration code, however in light of you and someone else pinging me about this, I’ll reconsider.

  3. Muflon says:

    Hi,

    is there any possibility to backup running container without stopping it? If not how do you do a secure backups and what solution do you use on production evironment? openVZ for eg. has vzdump which allows backups of running containers.

    1. If you’re using a btrfs, zfs or lvm backed container, you can use lxc-snapshot for that. Otherwise, you can simply backup the container’s rootfs at /var/lib/lxc//rootfs with a regular backup tool.

      This will however not dump the container’s running state (memory, cpu, …), we need checkpoint/restart for that (CRIU) and hope to have it in LXC 1.1. Work is still needed both on the CRIU and LXC side to make this happen though…

      1. Muflon says:

        I’m using lvm backed container. And as far as I found lxc-snapshot is not an option here because it requires us to stop container and then run a snapshot. I was hoping for doing security snapshots w/out stopping container.
        So as I understand this is not possible now.
        Thanks.

        1. Ipeacocks says:

          JFYI openvz can create backup without stop of containers. But anyway with very short downtime. With vzdump/vzrestore tools.

  4. Blizzz says:

    Thank you for the blog series, really good and helpful to get into lxc!

    It seems that in the stable release snapshotting of directory-based containers was not fixed, am I right? Given the output I get.

    lxc_container: Snapshot of directory-backed container requested.
    lxc_container: Making a copy-clone. If you do want snapshots, then
    lxc_container: please create an aufs or overlayfs clone first, snapshot that
    lxc_container: and keep the original container pristine.
    lxc_container: Error creating a snapshot

    1. Tom Cao says:

      Hi there,

      I faced the same error. Did you run snap-comment>. You must input snapshot description into a file.

      Regards,
      Tom

  5. Konrad says:

    reverting snapshot delete ALL?

    – Fresh Ubuntu 14.04
    – New LVM-Container (Ubuntu 14.04 inside)
    lxc-create -n mail -t download -B lvm –vgname mygroup –fssize 10G

    After that…

    # lxc-snapshot -n mail
    Logical volume “snap0” created

    # lxc-snapshot -n mail -L -C
    snap0 (/containersnaps/mail) 2014:07:24 14:13:35

    # lvs
    LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
    mail solid-3-0 owi-a-s– 10,00g
    snap0 solid-3-0 swi-a-s– 10,00g mail 0,00

    # lxc-snapshot -n mail -r snap0
    File descriptor 3 (/run/lock/lxc/container/mail) leaked on lvremove invocation. Parent PID 29998: lxc-snapshot
    Logical volume “snap0” successfully removed
    Logical volume “mail” successfully removed
    lxc_container: failed to detect blockdev type for /dev/solid-3-0/snap0
    lxc_container: Error copying storage
    lxc_container: Error restoring snapshot snap0

    # lvs
    LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert

    ALL GONE!

  6. Konrad says:

    reverting snapshot delete ALL?

    – Fresh Ubuntu 14.04
    – New LVM-Container (Ubuntu 14.04 inside)
    lxc-create -n mail -t download -B lvm –vgname mygroup –fssize 10G

    After that…

    # lxc-snapshot -n mail
    Logical volume “snap0” created

    # lxc-snapshot -n mail -L -C
    snap0 (/containersnaps/mail) 2014:07:24 14:13:35

    # lvs
    LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
    mail solid-3-0 owi-a-s– 10,00g
    snap0 solid-3-0 swi-a-s– 10,00g mail 0,00

    # lxc-snapshot -n mail -r snap0
    File descriptor 3 (/run/lock/lxc/container/mail) leaked on lvremove invocation. Parent PID 29998: lxc-snapshot
    Logical volume “snap0” successfully removed
    Logical volume “mail” successfully removed
    lxc_container: failed to detect blockdev type for /dev/solid-3-0/snap0
    lxc_container: Error copying storage
    lxc_container: Error restoring snapshot snap0

    # lvs
    LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert

    ALL GONE!

  7. dhbolthausen says:

    I maybe incorrect, but I believe it the path for changing the default directory in /etc/lxc/lxc.conf or ~.config/lxc/lxc.conf should be “lxc.lxcpath= /new/path” ?
    If I use just “lxcpath” it does not seem to work. I am not sure if this is a change or not but I am using ubuntu 14.04.3.
    Great write ups by the way.

  8. Sai says:

    Hello,

    Is there a way to restore a snapshot created via ‘lxc-copy -s –name c1 -N c1_snap’ ? I want to restore c1_snap on to c1.

  9. Apurva says:

    How to change /home/user/.cache/lxc/downloads path permanently. (to centralize it for various users)

  10. Borys Kabakov says:

    > “lxc-snapshot” currently doesn’t appear to work with directory based containers.

    Is it true for current version? I tried for lxc-3.0.3 and failed.

  11. Alexey says:

    How can I change a root container path for LXC-containers ? Is it possible ?

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.