Jun 052019

On previous occasions (yes that does mean more than once) I have messed around with the network configuration of containers to get :-

  1. A consistent behaviour.
  2. A fixed IPv4 address with no DHCP configuration (this one is easy).
  3. A fixed IPv6 address with no autoconfigured global addresses (this one has been tricky)

This turns out to be relatively easy providing that you configure the addresses within the container rather than within the container configuration. At least it looks good to go so far (I’ve been mistaken in the past).

The container configuration is quite simple :-

lxc.net.0.type = veth
lxc.net.0.flags = down
lxc.net.0.link = br0

Note that the bridge interface (br0) may be different. Also note that there is no lxc.net.0.ipv4.address, lxc.net.0.ipv4.gateway, lxc.net.0.ipv6.address, or lxc.net.0.ipv6.gateway.

The configuration within the container is dependent on what userland you are running, but for Debian (and Ubuntu if you’re not using Netplan) :-

auto lo
iface lo inet loopback

auto eth0
iface eth0 inet static

iface eth0 inet6 static
    address 2001:0db8:cafe:dead:0000:0000:0000:3eb/64
    scope global
    gateway 2001:0db8:cafe:dead:0000:0000:0000:0001
    privext 0
    accept_ra 0
    autoconf 0

Not sure quite which options are required but having all of “privext 0”, “accept_ra 0” and “autoconf 0” does mean no additional autoconfigured IPv6 addresses.

(And no the part number of this post isn’t anything more than a joke)

Oct 032015

More up to date information can be found here.

One thing that has always puzzled me about Linux Containers was why it is necessary to configure the network address in two places – the container configuration, and the operating system configuration. The short answer is that it isn’t.

If you configure network addresses statically within the container configuration :-

» grep net /var/lib/lxc/mango/config 
# networking
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.ipv4 =
lxc.network.ipv4.gateway =
lxc.network.ipv6 =         2001:0db8:ca2c:dead:0000:0000:0000:000a/64
lxc.network.ipv6.gateway = 2001:0db8:ca2c:dead:0000:0000:0000:0001

Then the configuration within the container’s operating system can simply be :-

» cat /var/lib/lxc/mango/rootfs/etc/network/interfaces
auto lo
iface lo inet loopback

auto eth0
iface eth0 inet manual
iface eth0 inet6 manual

And that works fine.

Oct 032015

A newer post has more information (and more reliable information).

One of the things that has been mildly irritating me about my little collection of Linux containers has been that in addition to the statically defined IPv6 addresses, there is also an automatically defined IPv6 address :-

» lxc-ls --fancy
NAME      STATE    IPV4       IPV6                                                              AUTOSTART  
apricot   RUNNING  2001:db8:ca2c:dead:21e:a0ff:feb6:6a, 2001:db8:ca2c:dead::3eb      YES        
chagers   RUNNING  2001:db8:ca2c:dead:804a:bfff:fe83:f98d, 2001:db8:ca2c:dead::5e11  YES        
glanders  RUNNING  2001:db8:ca2c:dead:21e:a0ff:feb6:66, 2001:db8:ca2c:dead::ba11     YES        
lyme      RUNNING  2001:db8:ca2c:dead:21e:a0ff:feb6:65, 2001:db8:ca2c:dead::cafe     YES        
mango     RUNNING  2001:db8:ca2c:dead:6c42:24ff:fe7d:4e9, 2001:db8:ca2c:dead::a      YES        
peach     RUNNING  2001:db8:ca2c:dead:21e:a0ff:feb6:68, 2001:db8:ca2c:dead::3a11     YES        
rhubarb   RUNNING  2001:db8:ca2c:dead:21e:a0ff:feb6:69, 2001:db8:ca2c:dead::dead     YES

Now this is hardly the end of the world, but it is not tidy and it is the sort of thing that may lead to problems down the road if servers are communicating on an address that is not reverse DNS registered. Or indeed when someone contacts a server on an address such as 2001:db8:ca2c:dead::3eb and the reply comes from 2001:db8:ca2c:dead:21e:a0ff:feb6:6a.

After any number of false starts, the answer is quite simple – use sysctl to turn off autoconfigured address from within the container; which doesn’t make much sense logically – containers don’t have a kernel of their own, so the global kernel should be the one that is tuned. However :-

for container in $(lxc-ls)
  echo net.ipv6.conf.eth0.autoconf = 0 >> /var/lib/lxc/$container/rootfs/etc/sysctl.conf

Does the trick (after a reboot)  :-

» lxc-ls --fancy
NAME      STATE    IPV4       IPV6                                                              AUTOSTART  
apricot   RUNNING  2001:db8:ca2c:dead:21e:a0ff:feb6:6a, 2001:db8:ca2c:dead::3eb      YES        
chagers   RUNNING  2001:db8:ca2c:dead:18d9:99ff:fe28:3591, 2001:db8:ca2c:dead::5e11  YES        
glanders  RUNNING  2001:db8:ca2c:dead:21e:a0ff:feb6:66, 2001:db8:ca2c:dead::ba11     YES        
lyme      RUNNING  2001:db8:ca2c:dead::cafe                                          YES        
mango     RUNNING  2001:db8:ca2c:dead:2411:80ff:feb9:6600, 2001:db8:ca2c:dead::a     YES        
peach     RUNNING  2001:db8:ca2c:dead::3a11                                          YES        
rhubarb   RUNNING  2001:db8:ca2c:dead::dead                                          YES        

Except for the older containers 🙁

I’ve obviously missed something, but fixing nearly half of the containers is a good start.

After attending to pending upgrades (some of my old containers were still running wheezy), and setting the network configuration to manual, one of the recalictrant containers (glanders) lost it’s autoconfigured address.

Two more containers lost their unwanted extra addresses after “fixing” their configuration. I’m not sure what was wrong with the old configuration, but after copying and modifying a recently created container configuration, they rebooted with just one IPv6 address. The last one was mango, but after an extra reboot, it also was fixed :-

» lxc-ls --fancy
NAME      STATE    IPV4       IPV6                      AUTOSTART  
apricot   RUNNING  2001:db8:ca2c:dead::3eb   YES        
chagers   RUNNING  2001:db8:ca2c:dead::5e11  YES        
glanders  RUNNING  2001:db8:ca2c:dead::ba11  YES        
lyme      RUNNING  2001:db8:ca2c:dead::cafe  YES        
mango     RUNNING  2001:db8:ca2c:dead::a     YES        
peach     RUNNING  2001:db8:ca2c:dead::3a11  YES        
rhubarb   RUNNING  2001:db8:ca2c:dead::dead  YES        
May 022015

I have recently been upgrading my Linux containers from Debian wheezy to jessie, and each time have encountered a problem preventing the container from booting. Or rather as it turns out, preventing the equivalent of init from starting any daemons. Which is systemd of course.

Now this is not some addition to the Great Systemd Debate (although my contribution to that debate may well arrive someday), but a simple fix, or at this stage a workaround (to use the dreaded ITIL phrase).

The fix is to re-install the traditional SystemV init package replacing the new systemd package. This can be done during the upgrade by running the following at the end of the usual process :-

apt-get install sysvinit-core

Of course you will probably be reading this after you have encountered the problem. There are probably many ways of dealing with the situation after you have tried rebooting and encountered this issue, but my choice is to run the following commands from what I tend to call the "global container" :-

chroot ${container root filesystem}
apt-get install sysvinit-core

As mentioned before, this is not a fix. And indeed the problem may be my own fault – perhaps it doesn't help having the "global container" still running wheezy. Perhaps there are some instructions in the Debian upgrade manual that details some extra step you should run. And of course by switching back to System V init, we are missing out on all of the systemd fun.

Jan 212012

One of the things I miss from Solaris are the Solaris Containers – zones – which are extremely useful for isolating lightweight services in their own “virtual machines”. This blog entry is about LXC – Linux Containers. There are of course other methods of accomplishing much the same thing with Linux, but LXC has the advantage that the necessary kernel extensions are included by default.

Or in other words it isn’t necessary to compile a custom kernel. Which has advantages in certain environments.

There are of course some disadvantages too – LXC isn’t quite as mature as some of the alternatives, and there’s a few missing features. But it works well enough.


Operating system level virtualisation, or what I prefer to call lightweight virtualisation is a method by which you can run multiple virtual servers on a single physical (or virtual!) machine. Like normal virtualisation supplied by products such as ESX, Hyper-V, etc., light-weight virtualisation allows you to run multiple servers on a single instance of server hardware (actually you can do this on a virtual server too!).

Operating system level virtualisation is not quite the same as full virtualisation where you get a complete virtual machine with a VGA display, a keyboard, mouse, etc. Instead you trick the first program that starts on a normal Unix (or Linux) system – /sbin/init – into believing that it is running on a machine by itself when it is in fact running inside a specially created environment. That is if you want a full virtual operating system; it is also possible to setup an environment so that a container simply starts a single application.

In some ways this is similar to the ancient chroot mechanism which was often recommended for securely installing applications such as BIND which were prone to attack. But it has been improved with greater isolation from the operating system running on the hardware itself.

Note that I said /sbin/init – these containers do not run their own kernel. Merely their own user processes.


So why are these containers useful ? Especially in a world where virtualisation is ubiquitous and can even be free (VirtualBox, and various KVM solutions). Well of course they are, or they wouldn’t exist – the equivalent of BSD Jails has been introduced for every single remaining Unix-based system as well as Linux.

First of all, containers provide a perfectly viable virtualisation mechanism if all you require are numbers of Linux machines. Indeed it is possible to use this kind of virtualisation on already virtualised machines – for example on a Cloud-based virtual server (as you might get from Amazon) which could potentially save you money.

Secondly, containers provide lightweight virtualisation in that there is little to no overhead in using them. There is no virtualised CPU, no I/O virtualisation, etc. In most “heavyweight” virtualisation solutions the overhead of virtualisation is negligible except for I/O where there is often a considerable performance hit for disk-based applications.

Next if carefully setup, it is possible to reduce the incremental cost of each server installation if you use containers rather than full virtual machines. Every additional server you run has a cost associated with maintaining it in terms of money and time; a variety of different mechanisms can reduce this incremental cost, but there is still a cost there. Containers can be another means of reducing this incremental cost by making it easier to manage the individual servers.

As an example, it is possible to update which DNS servers each container uses by simply copying the /etc/resolv.conf file to each container :-

for container in $(lxc-ls | sort | uniq)
  cp /etc/resolv.conf /srv/lxc/${container}/rootfs/etc/resolv.conf

It’s also very handy for testing – create a container on a test server to mess around with some component or other to find out how it should be installed, and then throw away the container. This avoids “corrupting” a test server with the results of repeated experiments over time.

Of course there are also reasons why you should not use them.

First of all, it is another virtualisation technology to learn. Not a difficult one, but there is still more to learn.

In addition, it does not provide complete isolation – if you need to reboot the physical server, you will have to reboot all of the containers. So it is probably not a good technology for multiple servers that need to stay up forever – although the only real way of arranging that is to use a load balancer in front of multiple servers (even clustering sometimes requires an outage).

There is also the fact that this is not entirely a mature technology. That is not to say it isn’t stable, but more that the tools are not quite polished as yet

Finally there are hints that containers do not provide complete isolation – someone with root access on a container might be able to escape from the container. Thus it is probably not a good solution to provide isolation for security reasons.

How ?

The following instructions assume the use of Debian, although most modern distributions should be perfectly fine too – I’ve also done this with SLES, and seen instructions for ArchLinux. You can also mix and match distributions – SLES as the master operating system, and Debian for the distribution in the containers. That works perfectly fine.

To see if your currently running kernel supports the relevant extensions, see if the cgroups filesystem is available :-

# grep cgroup /proc/filesystems
nodev	cgroup

If the grep command doesn’t return the output as shown, you will need to upgrade and/or build your own kernel. Which is a step beyond where I’m going.

Initial Setup

Before installing your first container, you need to setup your server to support LXC. This is all pretty simple – the most complicated part is to setup your server to use a bridged adapter as it’s network interface. Which we will tackle first.

To start with, install the bridge-utils package :-

# apt-get install bridge-utils

Next edit your /etc/interfaces file. Leave the section dealing with the loopback interface (lo) alone and comment out anything relating to eth0. Once that’s done, add something to setup the bridge interface (br0) with :-

auto br0
iface br0 inet dhcp
   bridge_ports eth0
   bridge_fd 0

That sets up a bridged interface configured with an address supplied by a DHCP server – not perhaps the best idea for a production server, but perfectly fine for a test server. At least if you have a DHCP server on the network! If you need to configure a static address, copy the relevant parts from your commented out section – the address, netmask, and gateway keywords can be added below bridge_fd.

At this point it is worth rebooting the server to check that the new network interface is working fine. You could perform all of the steps in this initial setup section and do a reboot just at the end, but I prefer to reboot after each step when I’m trying something new. It’s easier to find the horrible mistake when you do it step by step.

Assuming the reboot fine, the next step is to automatically mount the cgroup filesystem. Add the following to the end of the /etc/fstab file :-

cgroup        /cgroup        cgroup        defaults    0    0

Make the /cgroup directory, and try to mount the filesystem :-

mkdir /cgroup
mount /cgroup

At this point you should be able to see a whole bunch of files in the /cgroup directory, but it won’t show up if you try a df. You should also reboot at this point to make sure that the filesystem automatically mounts when the system boots.

The final stage is to install the LXC runtime package :-

apt-get install lxc debootstrap

Note that the LXC package is available for many distributions, and the debootstrap package is a shell script runnable under most distributions given the presence of the right set of tools.

Now we are ready to start creating containers. Well, almost.

Creating Containers

When creating containers, it is usual to use a helper script to do the donkey work of setting this up. The LXC runtime package includes a number of such helper scripts. These scripts are very useful, but it is also worth indicating that they may require some local customisation – for instance the template script I used sets the root password to root; whilst it does suggest that this should be changed when the container is built, it is also very sensible to change this initial password to something at least half-reasonable.

And the longer and more extensively you use containers, the more local customisations you are likely to want. For instance, I tend to use a bind to ensure that the /site filesystem is mounted under each container, so I can be sure that my local scripts and configuration files are available on each and every container easily. So :-

  cd /usr/lib/lxc/templates
  cp lxc-debian lxc-local

When not using Debian, it is possible that the template scripts are installed in the main $PATH. In which case you may choose to remove them, or move them somewhere else to avoid their use in preference to your own versions.

It is also worth creating a template configuration file for the lxc-create script :-

cat > /etc/lxc/lxc.conf

To create your first container :-

  mkdir -p /var/lib/lxc/first
  lvcreate --name root-first --size=768M /dev/debian
  mkfs -t ext3 /dev/debian/root-first
  {Edit /etc/fstab to mount the new filesystem at /var/lib/lxc/first}
  mount /var/lib/lxc/root-first
  lxc-create -n first -t local -f /etc/lxc/lxc.conf

This goes through the process of “bootstrapping” Debian into a directory on your current system and setting up a configuration for your LXC container. Once complete, you are ready to start but first you will notice that I created a filesystem for the container’s root filesystem which should be self-evidently necessary if you want to avoid the possibility of a badly behaved container from bringing down other containers.

There are of course other things you can do at this stage before starting the container for the first time – for instance editing /srv/lxc/first/rootfs/etc/network/interfaces may be necessary to enter a static address which is particularly useful

Starting The Container

Once you have created a container, you will probably want to start it :-

lxc-start --daemon --name=first

You can start a container without the “daemon” option, but this means you are immediately connected to the console and it can be difficult to escape from. To connect to the system’s console try lsx-console, which should result in something like :-

# lxc-console --name=first

Type  to exit the console
--hit return here--
Debian GNU/Linux 5.0 first tty1

first login:

At this point you can login as root, and “fix” your container as you would do with an ordinary server. Install the software you need, make any changes you want, etc.

But there is one most noticeable oddity – rebooting a container does not seem to work properly. It seems to get stuck at the point where a normal server would reset the machine to boot again. Undoubtedly this will be fixed at some point … it’s possible that the fix is relatively simple anyway.

But for now the sequence :-

(in the container) shutdown -h now
(in the server) lxc-stop --name ${name-of-container}
(in the server) lxc-start --name ${name-of-container}

Will have to do.

WP2Social Auto Publish Powered By : XYZScripts.com