Dual InfiniBand Links At Home

 Computing, Working Notes  No Responses »
Jul 292013
 

… which is of course massive overkill. But fun. It should increase the raw bandwidth available between the two machines from 1Gbps to 20Gbps (with one link) and 40Gbps with both links bonded. It was a bit of a surprise to me when I looked around at prices of second-hand kit to realise that InfiniBand was so much cheaper to acquire than Fibre Channel; the kit I acquired cost less than £100 all in whereas FC kit would be in the region of £1,000, and InfiniBand is generally quicker. There is of course 16Gb FC and 10Gb InfiniBand, but that is hardly comparing like with like. So what is this overkill for? Networking of course. I’ve acquired two HP InfiniBand dual link cards which means I can connect my workstation to my server :- InfiniBand Network Using dual links is of course overkill on top of overkill, but given that these cards have dual links, why not use them? And it does give a couple of experiments to try later. To prepare in advance, the following network addresses will be used :-

Server Link Number IPv4 Address IPv6 Address
A 1 10.255.0.1 AAISP:d00d::1
A 2 10.255.1.1 AAISP:d00f::1
B 1 10.255.0.254 AAISP:d00d:2
B 1 10.255.1.254 AAISP:d00f:2

Yes I have cheated for the IPv6 addresses! The first step is to configure each “server” … one is running Debian Linux, and the other is running FreeBSD.

Configuring Linux

This was subject to much delay whilst I believed that I had a problem with the InfiniBand card, but putting the card into a new desktop machine caused it to spring back to life. Either some sort of incompatibility with my old desktop (which was quite old), or some sort of problem with the BIOS settings.

Inserting the card should load the core module (mlx4_core) automatically, and spit out messages similar to the following :-

[    3.678189] mlx4_core 0000:07:00.0: irq 108 for MSI/MSI-X
[    3.678195] mlx4_core 0000:07:00.0: irq 109 for MSI/MSI-X
[    3.678199] mlx4_core 0000:07:00.0: irq 110 for MSI/MSI-X
[    3.678204] mlx4_core 0000:07:00.0: irq 111 for MSI/MSI-X
[    3.678208] mlx4_core 0000:07:00.0: irq 112 for MSI/MSI-X
[    3.678212] mlx4_core 0000:07:00.0: irq 113 for MSI/MSI-X
[    3.678216] mlx4_core 0000:07:00.0: irq 114 for MSI/MSI-X
[    3.678220] mlx4_core 0000:07:00.0: irq 115 for MSI/MSI-X
[    3.678223] mlx4_core 0000:07:00.0: irq 116 for MSI/MSI-X
[    3.678228] mlx4_core 0000:07:00.0: irq 117 for MSI/MSI-X
[    3.678232] mlx4_core 0000:07:00.0: irq 118 for MSI/MSI-X
[    3.678236] mlx4_core 0000:07:00.0: irq 119 for MSI/MSI-X
[    3.678239] mlx4_core 0000:07:00.0: irq 120 for MSI/MSI-X
[    3.678243] mlx4_core 0000:07:00.0: irq 121 for MSI/MSI-X
[    3.678247] mlx4_core 0000:07:00.0: irq 122 for MSI/MSI-X
[    3.678250] mlx4_core 0000:07:00.0: irq 123 for MSI/MSI-X
[    3.678254] mlx4_core 0000:07:00.0: irq 124 for MSI/MSI-X
[    3.678259] mlx4_core 0000:07:00.0: irq 125 for MSI/MSI-X
[    3.678263] mlx4_core 0000:07:00.0: irq 126 for MSI/MSI-X
[    3.678267] mlx4_core 0000:07:00.0: irq 127 for MSI/MSI-X
[    3.678271] mlx4_core 0000:07:00.0: irq 128 for MSI/MSI-X
[    3.678275] mlx4_core 0000:07:00.0: irq 129 for MSI/MSI-X

This is just the core driver; at this point additional modules are needed to do anything useful. You can manually load the modules with modprobe but sooner or later it is better to make sure they’re loaded automatically by adding their names to /etc/modules. The modules you want to load are :-

  1. mlx4_ib
  2. ib_umad
  3. ib_uverbs
  4. ib_ipoib

This is a minimal set necessary for networking (“IP”) rather than additional features such as SCSI. It’s generally better to start with a minimal set of features initially. At this point, it is generally a good idea to reboot to verify that things are getting closer. After a reboot, you should have one or more new network interfaces listed by ifconfig :-

ib0       Link encap:UNSPEC  HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00  
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ib1       Link encap:UNSPEC  HWaddr 80-00-00-49-FE-80-00-00-00-00-00-00-00-00-00-00  
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Despite the appearance, we still have quite a way to go yet. The next step is to install some additional packages: ibutils, infiniband-diags, and opensm. The last package is for a subnet manager which is unnecessary if you have an InfiniBand switch (but I don’t). The first step is to get opensm up and running. Edit /etc/default/opensm and change the PORTS variable to “ALL” (unless you want to restrict the managed ports, and make things more complicated). And start opensm: /etc/init.d/opensm start; update-rc.d opensm defaults.

At this point, you can configure the network addresses by editing /etc/network/interfaces. If you need help doing this, then you’re in the tech pool beyond your depth! Without something at the other end, these interfaces won’t work (obviously), so it’s time to start work on the other end …

Configuring FreeBSD

See: https://wiki.freebsd.org/InfiniBand I hadn’t had cause to build a custom kernel before, so the very first task was to use subversion to checkout a copy of the FreeBSD source code :-

svn co svn://svn0.us-east.FreeBSD.org/base/stable/9 /usr/src

Updating will of course require just: cd /usr/src && svn update. Once installed, create a symlink from /sys to /usr/src/sys if the link does not already exist: ln -s /usr/src/sys /sys

Go to the kernel configuration directory (/usr/src/sys/amd64/conf), copy the GENERIC configuration file to a new file, and edit the new file to add in certain options :-

# Infiniband stuff (locally added)
options         OFED
options         IPOIB_CM
device          ipoib
device          mlx4ib

Again, this is a minimal set that will not offer full functionality … but should be enough to get IP networking up and running. The next step is to build and install the kernel :-

make buildkernel KERNCONF=${NAME-OF-YOUR-CONFIG}; make installkernel KERNCONF=${NAME-OF-YOUR-CONFIG}

The next step is to build the “world”  :-

  1. Edit /etc/src.conf and add “WITH_OFED=’yes'” to that file.
  2. Change to /usr/src and run: make buildworld
  3. Finalise with make installworld

As it happens I had to build the user-land first, as the kernel compilation needed a new user-land feature.

After a reboot, the new network interface(s) should show up as ib0 upwards. And these can be configured with an address in exactly the same as any other network interface.

Testing The Network

A tip for making sure the interfaces you think are connected together is to configure one of the machines, send a broadcast ping to the relevant network address of each interface in turn, and run tcpdump on the other machine to verify that the packets coming down the wire match what you expect.

Below the level of IP, it is possible to run an InfiniBand ping to verify connectivity. First you need a GUID on “the server”, which can be obtained by running ibstat and looking for the “Port GUID”, which will be something like “0x0002c90200273985”. Next run ibping -S on the server.

Now on the other machine (“the client”), run ibping :-

# ibping -G 0x0002c90200273985
Pong from polio.inside.zonky.org (Lid 3): time 0.242 ms
Pong from polio.inside.zonky.org (Lid 3): time 0.153 ms
Pong from polio.inside.zonky.org (Lid 3): time 0.160 ms

The next step is to run an IP ping to one of the hosts. If that works, it is time to start looking at something that will do a reasonable attempt at a speed test.

This can be done in a variety of different ways, but I chose to use nttcp which is widely available. On one of the hosts, run nttcp -i to act as the “partner” (or server). On the sending server, run nntcp -T ${ip-address-to-test} which will give output something like :-

# nttcp -T 10.0.0.26
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.70    0.01     95.7975   5592.4053    2048   2923.51  170666.7
1  8388608    0.71    0.04     94.0667   1878.6950    5444   7630.87  152403.3

According to the documentation, the second line should begin with ‘r’, but for a simple speed test we can simply average the numbers in the “Real-MBit/s” to get an approximate speed. Oddly my gigabit ethernet seems to have mysteriously degraded to 100Mbps! At least it makes the InfiniBand speed slightly more impressive :-

# nttcp -T 10.255.0.2
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.03    0.00   2521.9415  16777.2160    2048  76963.55  512000.0
1  8388608    0.03    0.03   2206.6574   2568.6620    4032 132579.25  154329.0

Before getting into a panic over what appears to be a pretty poor result, it is worth bearing in mind that IP over InfiniBand isn’t especially efficient, and InfiniBand seems to suffer from marketing exaggeration. From what I understand, DDR’s 20Gbps signalling rate becomes 16Gbps, which in turn becomes 8.5Gbps when looking at the output of ibstatus (not ibstat) – why the halving here is a bit of a mystery, but that may become apparent later.

There has also been a hint that FreeBSD is due for a significant improvement in InfiniBand performance sometime after the release of 9.2.

As a late addition, it would appear that running OpenSM (the subnet manager) on both hosts means that when one or other is rebooting, the other can take over the duties of the subnet manager. To enable on FreeBSD, simply add opensm_enable=”YES” to the file /etc/rc.conf and reboot.

Web Sites, Domains, and the DNS

 Computing  No Responses »
Jul 282013
 

Having recently assisted with getting my sister’s business web site online (the domain and DNS side of things), it occurred to me that many people assume that it is all the same thing. Which is most definitely not the case, and believing so leaves you open to being ripped off. It is not unknown for hosting companies to “make it easy” for you to perform the whole job of registering a domain, making the relevant DNS changes, and setting up your web site. Often by hiding as much of the detail as possible.

That would be fine if that were all they did, but it isn’t. Sometimes they go out of their way to imply that if you want to change hosting companies, then you have to get a new domain, and it can take quite a bit of digging to find out how to transfer the domain elsewhere. Now it is easy to think that this doesn’t matter too much, but the longer you use a domain, the more you want to stick with it. Especially if you have a blog site that is not entirely unpopular … an older domain with lots of content has value. And the more successful you are, the more likely you are to want to change hosting companies.

Other hosting companies may offer better value on web sites with lots of visitors, or perhaps you are blogging in a controversial and need some sort of added protection against hackers, or you just “grow out” of a simple web site editing tool and want to get down and dirty with the HTML.

So it may well be worth your time finding out a little bit about this stuff in advance, and registering a domain separately to the web hosting. Or get a friendly geek to do so.

The Web Site

At their very simplest, web servers are nothing more than simple file servers. A web browser asks for an object (“give me http://zonky.org/index.html”), and the web server responds with the object (“It’s an HTML object, and here it is.”). Even on the most sophisticated web sites, the overwhelming majority of objects that make up the web page that you see, are simple files. And when you graduate to advanced features such as server-side languages (PHP, Java, etc.), the conversation between the web browser is still relatively simple one consisting mostly of simple requests for objects.

The key part of asking a web server for an object is that you need some sort of identifier for that object. This is known as the URL … Uniform Resource Locator, which could be called a “web address” (although URL is also used for non-web things). The key part (as far as we’re concerned here) is the host part of the URL. This performs two functions :-

  1. If it is not already a network address (IP address), then the web browser uses the DNS to get a network address. This is used to determine what web server to talk to.
  2. It is also included in the request to the web server. This allows the web server to distinguish between different “virtual servers”.

When it is not a network address, the host part of the URL is also known as the domain. Which brings us to the next topic.

The Domain

Domains and the DNS are very tightly associated, but in theory you could register a domain without having a DNS service. In practice it is rarely done … and almost always when you are running your own DNS service, in which case you will probably not be reading this!

But domains are distinct from the DNS service. The process of registering a domain consists of picking a name, and a top-level domain in which to add that name. It is most common to use .com for an international business (or one that wants to become one), a local business domain (such as .co.uk) for a more local business, .org for a generic organisation, etc. You can be creative with your choice of top-level domain, where the top-level domain becomes part of the entire name – such as http://bit.ly/, but that makes the registration process trickier.

Some top-level domains restrict who can register domains – the .ac.uk domain for example is really just for Academic organisations, and JANET is quite restrictive about who can register a domain with them.

When you register a domain, some of the information that you provide (such as name, address, phone number, etc.) is made public by default! Whilst you can often hide this, you may want to consider whether that is wise … domains with hidden registration information are often used by those for nefarious purposes. That is a domain with public information has a higher reputation than those without.

The Domain Name System (DNS)

The DNS is a service that allows you lookup names. It is usually used to turn named (such as zonky.org) into network addresses (81.2.106.111 or 2001:8b0:ca2c:dead::d00d), but there are also other kinds of records. A hosting provider often hides all this extra detail from you, but not always. It’s easy to overlook that with one domain, it is perfectly possible to have as many names as you have the imagination for – for example, my own domain (zonky.org) has a web site that is very rarely visited, there is also a different web site at www.zonky.org, and a separate blog at really.zonky.org (plus a few others). This can be handy if you want different web sites for different purposes – a normal web site for a business, an additional web site for a blog (to publicise the business), a forum site for customer support, etc.

Each of which could be hosted with a different hosting company!

Most of these “other” DNS record types are not of interest if all you are interested in are web sites, but one – the CNAME – may be useful. It allows you to give an “alias” to another name – i.e. make www.zonky.org point to zonky.org. If you have a web site with multiple names – for example a web site that responds at your domain name (zonky.org), and your domain name with “www” added to the front (www.zonky.org) – then it may be better to use a CNAME for the “www” to point to your domain. This is simply so that you only have to enter the network address of your web site in one location, and only update it once.

However “aliases” can only exist as aliases … there can be no other additional records associated with that name. Your domain name (zonky.org) has at least one other DNS record associated with it, so you cannot use an alias here.

Cameron’s Confused Plans for a “Cleaner Internet”

 Computing, Politics, Security  No Responses »
Jul 232013
 

Sign me up for the perv’s list … I won’t trust a politician to come up with a sensible method of censorship, and neither should you.

Ignoring the civil liberties thing: That politicians with a censorship weapon will tend to over use it, to the eventual detriment of legitimate debate.

How is Cameron’s censorship thing supposed to work? It appears nobody has a clear idea. Probably not even Cameron himself.

It seems to be two separate measures :-

  1. Completely block “extreme” porn: child abuse images, and “rape porn”. Oddly enough, he also claimed that “50 Shades of Grey” would not be banned although there are those who categorise it as rape porn. Interestingly this is nothing new as child abuse images have been blocked for years ineffectively.
  2. An “optional” mechanism for blocking some other mysterious category of porn – the “family filter” mechanism.

Now it all sounds quite reasonable, but firstly let’s take a look at the first measure. Blocking child abuse images sounds like a great idea … and indeed it is something that is already done by the Internet Watch Foundation. Whilst their work is undoubtedly valuable – at the very least it prevents accidental exposure to child abuse images – it probably doesn’t stop anyone who is serious about obtaining access to such porn. There are just too many ways around even a country-wide block.

Onto the second measure.

This means that anyone with an Internet connection has to decide when signing up whether they want to be “family friendly” or if they want to be added to the government’s list of perverts … or possibly the ISP’s list of perverts. Of course, how quickly do you think that list will be extracted and leaked? I’m sure the gutter press is salivating at the thought of getting hold of those lists to see what famous people opt to get all the porn; the same gutter press that won’t be blocked despite publishing pictures that some might say meet the criteria for being classified as porn (see Page 3).

And who decides what gets onto the “naughty list” of stuff that you have to sign up as a perv to see? What is the betting that there will be lots of mistakes?

As we already block access by default to “adult sites” on mobile networks, I have already encountered this problem. Not as you might imagine, but whilst away on a course I used an “app” to locate hostelries around my location. On clicking on the link to take me to a local pub’s web site to see a few more details, I was blocked. The interesting thing here is that the app had no problems telling me where the pub was, but the pub’s web site was blocked. Two standards for some reason?

And there are plenty of other examples of misclassification such as Facebook’s long running problem with blocking access to breast feeding information, hospitals having to remove censorship products so that surgeons could get to breast cancer information sites, etc. I happen to work in a field where sales critters are desperate to sell censorship products, and I’m aware that many places that do install such products have the endless fun of re-classifying sites.

And finally, given this is all for the sake of the children, who thinks that children will come up with ways to get around the “family filter” anyway? It is almost impossible to completely censor Internet access without extreme measures such as pulling the entire country off the Internet – even China with it’s Great Firewall is unable to completely censor Internet activity. Solutions such as proxies, VPN access, and Tor all make censorship impossible to make totally effective. If you are thinking that this is all too technical for children, you are sorely mistaken … for a start it does not take many children able to figure this stuff out as they will distribute their knowledge.

This not to say that a censorship mechanism that you control is not a sensible idea. You can select what to censor – prevent the children getting access to information about the Flying Spaghetti Monster, but block access to other religious sites, etc. And such a product has to be network-wide, to prevent someone plugging in an uncensored device; such as using the OpenDNS FamilyShield (although I have never used it, I believe it to be a good product from independent reports). Of course even DNS blocking can be worked around, but it’s a reasonable effort.

Don’t Write Off British Manufacturing and Engineering …

 General  No Responses »
Jul 212013
 

It sometimes seems fashionable to put down British manufacturing and engineering such as when the well known idiot Jeremy Clarkson announced: “We don’t manufacture anything any more”.

Whilst it is true that Britain no longer makes more goods than the rest of the world combined, if you take the trouble to look you will find a surprisingly big industry. The trouble is that we all too often look backwards and compare today with the 19th century. Time to stop doing that, and actually look at today’s industry.

To quote the Wikipedia article (which has some other quite dated figures): “manufacturing output has increased in 35 of the 50 years between 1958 and 2007” and “output in 2007 was at record levels, approximately double that in 1958”. And: “In 2008, the UK was the sixth-largest manufacturer in the world measured by value of output.”

We may not make as much stuff as we used to, but what we do make is a lot more valuable.

A few points that illustrate just how well Britain is doing :-

  1. Of 11 constructors within Formula-1, 8 are based in the UK. Including teams such as Mercedes which you would quite reasonably assume were based in another country.
  2. Despite a series of governments that believe that spending money on space is a waste of time, the UK space industry is still worth £9 billion a year.
  3. BAE Systems is the third largest defence company in the world.
  4. GlaxoSmithKline is the fourth largest pharmaceutical company in the world.
  5. Of the 100 companies in the FTSE-100, around 33 can be regarded as manufacturing companies of one kind or another.
  6. A lesser known company (ARM) designs what is probably the most successful family of computer processors ever – ARM-based processors are found in 95% of all smartphones.

 

 Older Entries  Newer Entries