LXD 2.0: LXD and OpenStack [11/12]

This is the eleventh blog post in this series about LXD 2.0.

LXD logo

Introduction

First of all, sorry for the delay. It took quite a long time before I finally managed to get all of this going. My first attempts were using devstack which ran into a number of issues that had to be resolved. Yet even after all that, I still wasn’t be able to get networking going properly.

I finally gave up on devstack and tried “conjure-up” to deploy a full Ubuntu OpenStack using Juju in a pretty user friendly way. And it finally worked!

So below is how to run a full OpenStack, using LXD containers instead of VMs and running all of this inside a LXD container (nesting!).

Requirements

This post assumes you’ve got a working LXD setup, providing containers with network access and that you have a pretty beefy CPU, around 50GB of space for the container to use and at least 16GB of RAM.

Remember, we’re running a full OpenStack here, this thing isn’t exactly light!

Setting up the container

OpenStack is made of a lof of different components, doing a lot of different things. Some require some additional privileges so to make our live easier, we’ll use a privileged container.

We’ll configure that container to support nesting, pre-load all the required kernel modules and allow it access to /dev/mem (as is apparently needed).

Please note that this means that most of the security benefit of LXD containers are effectively disabled for that container. However the containers that will be spawned by OpenStack itself will be unprivileged and use all the normal LXD security features.

lxc launch ubuntu:16.04 openstack -c security.privileged=true -c security.nesting=true -c "linux.kernel_modules=iptable_nat, ip6table_nat, ebtables, openvswitch"
lxc config device add openstack mem unix-char path=/dev/mem

There is a small bug in LXD where it would attempt to load kernel modules that have already been loaded on the host. This has been fixed in LXD 2.5 and will be fixed in LXD 2.0.6 but until then, this can be worked around with:

lxc exec openstack -- ln -s /bin/true /usr/local/bin/modprobe

Then we need to add a couple of PPAs and install conjure-up, the deployment tool we’ll use to get OpenStack going.

lxc exec openstack -- apt-add-repository ppa:conjure-up/next -y
lxc exec openstack -- apt-add-repository ppa:juju/stable -y
lxc exec openstack -- apt update
lxc exec openstack -- apt dist-upgrade -y
lxc exec openstack -- apt install conjure-up -y

And the last setup step is to configure LXD networking inside the container.
Answer with the default for all questions, except for:

  • Use the “dir” storage backend (“zfs” doesn’t work in a nested container)
  • Do NOT configure IPv6 networking (conjure-up/juju don’t play well with it)
lxc exec openstack -- lxd init

And that’s it for the container configuration itself, now we can deploy OpenStack!

Deploying OpenStack with conjure-up

As mentioned earlier, we’ll be using conjure-up to deploy OpenStack.
This is a nice, user friendly, tool that interfaces with Juju to deploy complex services.

Start it with:

lxc exec openstack -- sudo -u ubuntu -i conjure-up
  • Select “OpenStack with NovaLXD”
  • Then select “localhost” as the deployment target (uses LXD)
  • And hit “Deploy all remaining applications”

This will now deploy OpenStack. The whole process can take well over an hour depending on what kind of machine you’re running this on. You’ll see all services getting a container allocated, then getting deployed and finally interconnected.

Conjure-Up deploying OpenStack

Once the deployment is done, a few post-install steps will appear. This will import some initial images, setup SSH authentication, configure networking and finally giving you the IP address of the dashboard.

Access the dashboard and spawn a container

The dashboard runs inside a container, so you can’t just hit it from your web browser.
The easiest way around this is to setup a NAT rule with:

lxc exec openstack -- iptables -t nat -A PREROUTING -p tcp --dport 80 -j DNAT --to <IP>

Where “<ip>” is the dashboard IP address conjure-up gave you at the end of the installation.

You can now grab the IP address of the “openstack” container (from “lxc info openstack”) and point your web browser to: http://<container ip>/horizon

This can take a few minutes to load the first time around. Once the login screen is loaded, enter the default login and password (admin/openstack) and you’ll be greeted by the OpenStack dashboard!

oslxd-dashboard

You can now head to the “Project” tab on the left and the “Instances” page. To start a new instance using nova-lxd, click on “Launch instance”, select what image you want, network, … and your instance will get spawned.

Once it’s running, you can assign it a floating IP which will let you reach your instance from within your “openstack” container.

Conclusion

OpenStack is a pretty complex piece of software, it’s also not something you really want to run at home or on a single server. But it’s certainly interesting to be able to do it anyway, keeping everything contained to a single container on your machine.

Conjure-Up is a great tool to deploy such complex software, using Juju behind the scene to drive the deployment, using LXD containers for every individual service and finally for the instances themselves.

It’s also one of the very few cases where multiple level of container nesting actually makes sense!

Extra information

The conjure-up website can be found at: http://conjure-up.io
The Juju website can be found at: http://www.ubuntu.com/cloud/juju

The main LXD website is at: https://linuxcontainers.org/lxd
Development happens on Github at: https://github.com/lxc/lxd
Mailing-list support happens on: https://lists.linuxcontainers.org
IRC support happens in: #lxcontainers on irc.freenode.net
Try LXD online: https://linuxcontainers.org/lxd/try-it

This entry was posted in Canonical voices, LXD, Planet Ubuntu and tagged . Bookmark the permalink.

11 Responses to LXD 2.0: LXD and OpenStack [11/12]

  1. Henry Lawson says:

    Thank you for this. I had attempted open stack on LXD a few months ago but got stuck! I didn’t know about conjure up then.

    I am looking forward to giving this a go!

  2. tom zhou says:

    seems this is a dev/test setup. Not for production .

    • Absolutely. Someone who would want to run a production setup will want to deploy OpenStack on a bunch of physical servers after doing some research on the exact options they want for storage, network, …

      The setup described here is a way to get OpenStack going so you can play around, see how things work and try nova-lxd.

  3. Satya Kada says:

    Hi Stéphane Graber,
    Thank you for the post. Had lot of information. Will try deploying the openstack with nova-lxd. I am new to LXD, I need help on the following
    1. Networking details to setup LXD.
    2. I want to test live-migration between remote hosts(here containers). So I want to add multiple nova-compute containers with nova-lxd. How can I add multiple compute nodes(here multiple compute node containers) using conjure-up.

  4. Wayne says:

    Is it possible to do this in a network without IPv6? I get lots of messages like this.

    ?: invalid HTTP proxy (http://[fe80::1%eth0]:13128): bad URI

  5. Harry says:

    Excellent guide.
    I am using devstack within LXD with only a few problems.
    I have to ask though.
    Why do you pass /dev/mem ?

    • I don’t actually know 🙂

      The reason why I’m passing it to the main container is because Juju through conjure-up attempts to pass it to the nested openstack containers. So not having it in the parent container was causing deployment to fail.

      I can’t think of a good reason for them to actually use /dev/mem, so hopefully someone can sort out Juju/charms so that they stop passing it through.

  6. Gaetan QUENTIN says:

    lxc exec openstack — ln -s /bin/true /usr/local/bin/modprobe
    error: flag provided but not defined: -s

    works better with the ” ‘ ” :
    lxc exec openstack — ‘ln -s /bin/true /usr/local/bin/modprobe’

    Tried your install on ubuntu 16.10

    i have a lot of:
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-tmpfiles-setup.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/proc-swaps.mount: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/conjure-up.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/systemd-update-utmp.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/cron.service: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/dev-hugepages.mount: Operation not permitted
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/sys-devices-virtual-net.mount: Operation not permitted
    Dec 8 19:35:13 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:35:13 openstack systemd[1]: Starting LSB: AppArmor initialization…
    Dec 8 19:35:13 openstack apparmor[75]: /etc/init.d/apparmor: 256: /etc/init.d/apparmor: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:35:13 openstack apparmor[75]: * Not starting AppArmor in container
    Dec 8 19:35:13 openstack apparmor[75]: …done.
    Dec 8 19:35:13 openstack systemd[1]: Started LSB: AppArmor initialization.
    Dec 8 19:35:13 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:38:35 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:39:56 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:43:05 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:43:07 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:47:19 openstack lxc-apparmor-load[3900]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:47:19 openstack lxc-apparmor-load[3900]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:47:19 openstack lxd[3905]: t=2016-12-08T19:47:19+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.”
    Dec 8 19:47:19 openstack lxd[3905]: t=2016-12-08T19:47:19+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.”
    Dec 8 19:50:10 openstack systemd[1]: Failed to reset devices.list on /system.slice/apparmor.service: Operation not permitted
    Dec 8 19:50:38 openstack lxc-apparmor-load[4242]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:50:38 openstack lxc-apparmor-load[4242]: /lib/apparmor/profile-load: 256: /lib/apparmor/profile-load: cannot open /sys/kernel/security/apparmor/.ns_stacked: Permission denied
    Dec 8 19:50:38 openstack lxd[4247]: t=2016-12-08T19:50:38+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because the mac_admin capability is missing.”
    Dec 8 19:50:38 openstack lxd[4247]: t=2016-12-08T19:50:38+0000 lvl=warn msg=”Per-container AppArmor profiles are disabled because LXD is already protected by AppArmor.”

    And:

    Dec 8 19:56:00 openstack conjure-up/openstack-novalxd: Running pre-deploy for OpenStack
    Dec 8 19:56:05 openstack conjure-up/openstack-novalxd: Processing lxd profile: conjure-up
    Dec 8 19:56:13 openstack conjure-up/openstack-novalxd: Running pre-deploy for OpenStack
    Dec 8 19:56:18 openstack conjure-up/openstack-novalxd: Processing lxd profile: conjure-up
    Dec 8 20:01:00 openstack conjure-up/openstack-novalxd: Running deploy-done for OpenStack installation.

    And in lxd.log:
    It looks like it is looping:
    t=2016-12-08T20:08:53+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:08:47+0000 ephemeral=false name=juju-5c77ff-3
    t=2016-12-08T20:09:05+0000 lvl=info msg=”Deleting container” name=juju-5c77ff-10 creation date=2016-12-08T20:08:59+0000 ephemeral=false
    t=2016-12-08T20:09:06+0000 lvl=info msg=”Deleted container” name=juju-5c77ff-10 creation date=2016-12-08T20:08:59+0000 ephemeral=false
    t=2016-12-08T20:09:27+0000 lvl=info msg=”Deleting container” creation date=2016-12-08T20:09:22+0000 ephemeral=false name=juju-5c77ff-10
    t=2016-12-08T20:09:28+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:09:22+0000 ephemeral=false name=juju-5c77ff-10
    t=2016-12-08T20:09:46+0000 lvl=info msg=”Deleting container” creation date=2016-12-08T20:09:42+0000 ephemeral=false name=juju-5c77ff-10
    t=2016-12-08T20:09:46+0000 lvl=info msg=”Deleted container” creation date=2016-12-08T20:09:42+0000 ephemeral=false name=juju-5c77ff-10

    lxc list:
    root@openstack:~# lxc list
    +—————-+———+——————–+——+————+———–+
    | NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
    +—————-+———+——————–+——+————+———–+
    | juju-5c77ff-11 | STOPPED | | | PERSISTENT | 0 |
    +—————-+———+——————–+——+————+———–+
    | juju-83da42-0 | RUNNING | 10.182.1.45 (eth0) | | PERSISTENT | 0 |
    +—————-+———+——————–+——+————+———–+

    In the conjure-up screen , all is at “waiting for machine state”

  7. Gaetan QUENTIN says:

    FYI:

    tested again today .
    This guide does not launch conjure-up anymore with updated 16.04 and conjure-up (0.1.2):

    – lxc exec openstack — sudo -u ubuntu -i conjure-up : need spell
    usage: conjure-up [-h] [-d] [-s] [–version] spell
    conjure-up: error: the following arguments are required: spell

    so trying: lxc exec openstack — sudo -u ubuntu -i conjure-up openstack
    (it shows tty error message)
    i choose nova-lxd and then in the “clouds” list: localhost

    -> oops there was a problem with your install
    Reason: flags provided but not defined: –upload-tools

    • Sounds like you forgot to add the two PPAs. Without them you’ll be getting that old conjure-up (0.1.2) instead of up to date (2.x) and same goes for the Juju version.

    • Gaetan QUENTIN says:

      I made a mistake: i have not added the profile with network when launching the container so the update did not work but it didn’t groaned. So it was a bad conjure-up version.

      Sorry for the bad post.

Leave a Reply

Your email address will not be published. Required fields are marked *