Computing – Page 39 – More Zonkyness

Dual InfiniBand Links At Home

Jul 292013

… which is of course massive overkill. But fun. It should increase the raw bandwidth available between the two machines from 1Gbps to 20Gbps (with one link) and 40Gbps with both links bonded. It was a bit of a surprise to me when I looked around at prices of second-hand kit to realise that InfiniBand was so much cheaper to acquire than Fibre Channel; the kit I acquired cost less than £100 all in whereas FC kit would be in the region of £1,000, and InfiniBand is generally quicker. There is of course 16Gb FC and 10Gb InfiniBand, but that is hardly comparing like with like. So what is this overkill for? Networking of course. I’ve acquired two HP InfiniBand dual link cards which means I can connect my workstation to my server :- Using dual links is of course overkill on top of overkill, but given that these cards have dual links, why not use them? And it does give a couple of experiments to try later. To prepare in advance, the following network addresses will be used :-

Server	Link Number	IPv4 Address	IPv6 Address
A	1	10.255.0.1	AAISP:d00d::1
A	2	10.255.1.1	AAISP:d00f::1
B	1	10.255.0.254	AAISP:d00d:2
B	1	10.255.1.254	AAISP:d00f:2

Yes I have cheated for the IPv6 addresses! The first step is to configure each “server” … one is running Debian Linux, and the other is running FreeBSD.

Configuring Linux

This was subject to much delay whilst I believed that I had a problem with the InfiniBand card, but putting the card into a new desktop machine caused it to spring back to life. Either some sort of incompatibility with my old desktop (which was quite old), or some sort of problem with the BIOS settings.

Inserting the card should load the core module (mlx4_core) automatically, and spit out messages similar to the following :-

[    3.678189] mlx4_core 0000:07:00.0: irq 108 for MSI/MSI-X
[    3.678195] mlx4_core 0000:07:00.0: irq 109 for MSI/MSI-X
[    3.678199] mlx4_core 0000:07:00.0: irq 110 for MSI/MSI-X
[    3.678204] mlx4_core 0000:07:00.0: irq 111 for MSI/MSI-X
[    3.678208] mlx4_core 0000:07:00.0: irq 112 for MSI/MSI-X
[    3.678212] mlx4_core 0000:07:00.0: irq 113 for MSI/MSI-X
[    3.678216] mlx4_core 0000:07:00.0: irq 114 for MSI/MSI-X
[    3.678220] mlx4_core 0000:07:00.0: irq 115 for MSI/MSI-X
[    3.678223] mlx4_core 0000:07:00.0: irq 116 for MSI/MSI-X
[    3.678228] mlx4_core 0000:07:00.0: irq 117 for MSI/MSI-X
[    3.678232] mlx4_core 0000:07:00.0: irq 118 for MSI/MSI-X
[    3.678236] mlx4_core 0000:07:00.0: irq 119 for MSI/MSI-X
[    3.678239] mlx4_core 0000:07:00.0: irq 120 for MSI/MSI-X
[    3.678243] mlx4_core 0000:07:00.0: irq 121 for MSI/MSI-X
[    3.678247] mlx4_core 0000:07:00.0: irq 122 for MSI/MSI-X
[    3.678250] mlx4_core 0000:07:00.0: irq 123 for MSI/MSI-X
[    3.678254] mlx4_core 0000:07:00.0: irq 124 for MSI/MSI-X
[    3.678259] mlx4_core 0000:07:00.0: irq 125 for MSI/MSI-X
[    3.678263] mlx4_core 0000:07:00.0: irq 126 for MSI/MSI-X
[    3.678267] mlx4_core 0000:07:00.0: irq 127 for MSI/MSI-X
[    3.678271] mlx4_core 0000:07:00.0: irq 128 for MSI/MSI-X
[    3.678275] mlx4_core 0000:07:00.0: irq 129 for MSI/MSI-X

This is just the core driver; at this point additional modules are needed to do anything useful. You can manually load the modules with modprobe but sooner or later it is better to make sure they’re loaded automatically by adding their names to /etc/modules. The modules you want to load are :-

mlx4_ib
ib_umad
ib_uverbs
ib_ipoib

This is a minimal set necessary for networking (“IP”) rather than additional features such as SCSI. It’s generally better to start with a minimal set of features initially. At this point, it is generally a good idea to reboot to verify that things are getting closer. After a reboot, you should have one or more new network interfaces listed by ifconfig :-

ib0       Link encap:UNSPEC  HWaddr 80-00-00-48-FE-80-00-00-00-00-00-00-00-00-00-00  
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

ib1       Link encap:UNSPEC  HWaddr 80-00-00-49-FE-80-00-00-00-00-00-00-00-00-00-00  
          UP BROADCAST RUNNING MULTICAST  MTU:2044  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:256 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Despite the appearance, we still have quite a way to go yet. The next step is to install some additional packages: ibutils, infiniband-diags, and opensm. The last package is for a subnet manager which is unnecessary if you have an InfiniBand switch (but I don’t). The first step is to get opensm up and running. Edit /etc/default/opensm and change the PORTS variable to “ALL” (unless you want to restrict the managed ports, and make things more complicated). And start opensm: /etc/init.d/opensm start; update-rc.d opensm defaults.

At this point, you can configure the network addresses by editing /etc/network/interfaces. If you need help doing this, then you’re in the tech pool beyond your depth! Without something at the other end, these interfaces won’t work (obviously), so it’s time to start work on the other end …

Configuring FreeBSD

See: https://wiki.freebsd.org/InfiniBand I hadn’t had cause to build a custom kernel before, so the very first task was to use subversion to checkout a copy of the FreeBSD source code :-

svn co svn://svn0.us-east.FreeBSD.org/base/stable/9 /usr/src

Updating will of course require just: cd /usr/src && svn update. Once installed, create a symlink from /sys to /usr/src/sys if the link does not already exist: ln -s /usr/src/sys /sys

Go to the kernel configuration directory (/usr/src/sys/amd64/conf), copy the GENERIC configuration file to a new file, and edit the new file to add in certain options :-

# Infiniband stuff (locally added)
options         OFED
options         IPOIB_CM
device          ipoib
device          mlx4ib

Again, this is a minimal set that will not offer full functionality … but should be enough to get IP networking up and running. The next step is to build and install the kernel :-

make buildkernel KERNCONF=${NAME-OF-YOUR-CONFIG}; make installkernel KERNCONF=${NAME-OF-YOUR-CONFIG}

The next step is to build the “world” :-

Edit /etc/src.conf and add “WITH_OFED=’yes'” to that file.
Change to /usr/src and run: make buildworld
Finalise with make installworld

As it happens I had to build the user-land first, as the kernel compilation needed a new user-land feature.

After a reboot, the new network interface(s) should show up as ib0 upwards. And these can be configured with an address in exactly the same as any other network interface.

Testing The Network

A tip for making sure the interfaces you think are connected together is to configure one of the machines, send a broadcast ping to the relevant network address of each interface in turn, and run tcpdump on the other machine to verify that the packets coming down the wire match what you expect.

Below the level of IP, it is possible to run an InfiniBand ping to verify connectivity. First you need a GUID on “the server”, which can be obtained by running ibstat and looking for the “Port GUID”, which will be something like “0x0002c90200273985”. Next run ibping -S on the server.

Now on the other machine (“the client”), run ibping :-

# ibping -G 0x0002c90200273985
Pong from polio.inside.zonky.org (Lid 3): time 0.242 ms
Pong from polio.inside.zonky.org (Lid 3): time 0.153 ms
Pong from polio.inside.zonky.org (Lid 3): time 0.160 ms

The next step is to run an IP ping to one of the hosts. If that works, it is time to start looking at something that will do a reasonable attempt at a speed test.

This can be done in a variety of different ways, but I chose to use nttcp which is widely available. On one of the hosts, run nttcp -i to act as the “partner” (or server). On the sending server, run nntcp -T ${ip-address-to-test} which will give output something like :-

# nttcp -T 10.0.0.26
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.70    0.01     95.7975   5592.4053    2048   2923.51  170666.7
1  8388608    0.71    0.04     94.0667   1878.6950    5444   7630.87  152403.3

According to the documentation, the second line should begin with ‘r’, but for a simple speed test we can simply average the numbers in the “Real-MBit/s” to get an approximate speed. Oddly my gigabit ethernet seems to have mysteriously degraded to 100Mbps! At least it makes the InfiniBand speed slightly more impressive :-

# nttcp -T 10.255.0.2
     Bytes  Real s   CPU s Real-MBit/s  CPU-MBit/s   Calls  Real-C/s   CPU-C/s
l  8388608    0.03    0.00   2521.9415  16777.2160    2048  76963.55  512000.0
1  8388608    0.03    0.03   2206.6574   2568.6620    4032 132579.25  154329.0

Before getting into a panic over what appears to be a pretty poor result, it is worth bearing in mind that IP over InfiniBand isn’t especially efficient, and InfiniBand seems to suffer from marketing exaggeration. From what I understand, DDR’s 20Gbps signalling rate becomes 16Gbps, which in turn becomes 8.5Gbps when looking at the output of ibstatus (not ibstat) – why the halving here is a bit of a mystery, but that may become apparent later.

There has also been a hint that FreeBSD is due for a significant improvement in InfiniBand performance sometime after the release of 9.2.

As a late addition, it would appear that running OpenSM (the subnet manager) on both hosts means that when one or other is rebooting, the other can take over the duties of the subnet manager. To enable on FreeBSD, simply add opensm_enable=”YES” to the file /etc/rc.conf and reboot.

Web Sites, Domains, and the DNS

Computing No Responses »

Jul 282013

Having recently assisted with getting my sister’s business web site online (the domain and DNS side of things), it occurred to me that many people assume that it is all the same thing. Which is most definitely not the case, and believing so leaves you open to being ripped off. It is not unknown for hosting companies to “make it easy” for you to perform the whole job of registering a domain, making the relevant DNS changes, and setting up your web site. Often by hiding as much of the detail as possible.

That would be fine if that were all they did, but it isn’t. Sometimes they go out of their way to imply that if you want to change hosting companies, then you have to get a new domain, and it can take quite a bit of digging to find out how to transfer the domain elsewhere. Now it is easy to think that this doesn’t matter too much, but the longer you use a domain, the more you want to stick with it. Especially if you have a blog site that is not entirely unpopular … an older domain with lots of content has value. And the more successful you are, the more likely you are to want to change hosting companies.

Other hosting companies may offer better value on web sites with lots of visitors, or perhaps you are blogging in a controversial and need some sort of added protection against hackers, or you just “grow out” of a simple web site editing tool and want to get down and dirty with the HTML.

So it may well be worth your time finding out a little bit about this stuff in advance, and registering a domain separately to the web hosting. Or get a friendly geek to do so.

The Web Site

At their very simplest, web servers are nothing more than simple file servers. A web browser asks for an object (“give me http://zonky.org/index.html”), and the web server responds with the object (“It’s an HTML object, and here it is.”). Even on the most sophisticated web sites, the overwhelming majority of objects that make up the web page that you see, are simple files. And when you graduate to advanced features such as server-side languages (PHP, Java, etc.), the conversation between the web browser is still relatively simple one consisting mostly of simple requests for objects.

The key part of asking a web server for an object is that you need some sort of identifier for that object. This is known as the URL … Uniform Resource Locator, which could be called a “web address” (although URL is also used for non-web things). The key part (as far as we’re concerned here) is the host part of the URL. This performs two functions :-

If it is not already a network address (IP address), then the web browser uses the DNS to get a network address. This is used to determine what web server to talk to.
It is also included in the request to the web server. This allows the web server to distinguish between different “virtual servers”.

When it is not a network address, the host part of the URL is also known as the domain. Which brings us to the next topic.

The Domain

Domains and the DNS are very tightly associated, but in theory you could register a domain without having a DNS service. In practice it is rarely done … and almost always when you are running your own DNS service, in which case you will probably not be reading this!

But domains are distinct from the DNS service. The process of registering a domain consists of picking a name, and a top-level domain in which to add that name. It is most common to use .com for an international business (or one that wants to become one), a local business domain (such as .co.uk) for a more local business, .org for a generic organisation, etc. You can be creative with your choice of top-level domain, where the top-level domain becomes part of the entire name – such as http://bit.ly/, but that makes the registration process trickier.

Some top-level domains restrict who can register domains – the .ac.uk domain for example is really just for Academic organisations, and JANET is quite restrictive about who can register a domain with them.

When you register a domain, some of the information that you provide (such as name, address, phone number, etc.) is made public by default! Whilst you can often hide this, you may want to consider whether that is wise … domains with hidden registration information are often used by those for nefarious purposes. That is a domain with public information has a higher reputation than those without.

The Domain Name System (DNS)

The DNS is a service that allows you lookup names. It is usually used to turn named (such as zonky.org) into network addresses (81.2.106.111 or 2001:8b0:ca2c:dead::d00d), but there are also other kinds of records. A hosting provider often hides all this extra detail from you, but not always. It’s easy to overlook that with one domain, it is perfectly possible to have as many names as you have the imagination for – for example, my own domain (zonky.org) has a web site that is very rarely visited, there is also a different web site at www.zonky.org, and a separate blog at really.zonky.org (plus a few others). This can be handy if you want different web sites for different purposes – a normal web site for a business, an additional web site for a blog (to publicise the business), a forum site for customer support, etc.

Each of which could be hosted with a different hosting company!

Most of these “other” DNS record types are not of interest if all you are interested in are web sites, but one – the CNAME – may be useful. It allows you to give an “alias” to another name – i.e. make www.zonky.org point to zonky.org. If you have a web site with multiple names – for example a web site that responds at your domain name (zonky.org), and your domain name with “www” added to the front (www.zonky.org) – then it may be better to use a CNAME for the “www” to point to your domain. This is simply so that you only have to enter the network address of your web site in one location, and only update it once.

However “aliases” can only exist as aliases … there can be no other additional records associated with that name. Your domain name (zonky.org) has at least one other DNS record associated with it, so you cannot use an alias here.

Cameron’s Confused Plans for a “Cleaner Internet”

Computing, Politics, Security No Responses »

Jul 232013

Sign me up for the perv’s list … I won’t trust a politician to come up with a sensible method of censorship, and neither should you.

Ignoring the civil liberties thing: That politicians with a censorship weapon will tend to over use it, to the eventual detriment of legitimate debate.

How is Cameron’s censorship thing supposed to work? It appears nobody has a clear idea. Probably not even Cameron himself.

It seems to be two separate measures :-

Completely block “extreme” porn: child abuse images, and “rape porn”. Oddly enough, he also claimed that “50 Shades of Grey” would not be banned although there are those who categorise it as rape porn. Interestingly this is nothing new as child abuse images have been blocked for years ineffectively.
An “optional” mechanism for blocking some other mysterious category of porn – the “family filter” mechanism.

Now it all sounds quite reasonable, but firstly let’s take a look at the first measure. Blocking child abuse images sounds like a great idea … and indeed it is something that is already done by the Internet Watch Foundation. Whilst their work is undoubtedly valuable – at the very least it prevents accidental exposure to child abuse images – it probably doesn’t stop anyone who is serious about obtaining access to such porn. There are just too many ways around even a country-wide block.

Onto the second measure.

This means that anyone with an Internet connection has to decide when signing up whether they want to be “family friendly” or if they want to be added to the government’s list of perverts … or possibly the ISP’s list of perverts. Of course, how quickly do you think that list will be extracted and leaked? I’m sure the gutter press is salivating at the thought of getting hold of those lists to see what famous people opt to get all the porn; the same gutter press that won’t be blocked despite publishing pictures that some might say meet the criteria for being classified as porn (see Page 3).

And who decides what gets onto the “naughty list” of stuff that you have to sign up as a perv to see? What is the betting that there will be lots of mistakes?

As we already block access by default to “adult sites” on mobile networks, I have already encountered this problem. Not as you might imagine, but whilst away on a course I used an “app” to locate hostelries around my location. On clicking on the link to take me to a local pub’s web site to see a few more details, I was blocked. The interesting thing here is that the app had no problems telling me where the pub was, but the pub’s web site was blocked. Two standards for some reason?

And there are plenty of other examples of misclassification such as Facebook’s long running problem with blocking access to breast feeding information, hospitals having to remove censorship products so that surgeons could get to breast cancer information sites, etc. I happen to work in a field where sales critters are desperate to sell censorship products, and I’m aware that many places that do install such products have the endless fun of re-classifying sites.

And finally, given this is all for the sake of the children, who thinks that children will come up with ways to get around the “family filter” anyway? It is almost impossible to completely censor Internet access without extreme measures such as pulling the entire country off the Internet – even China with it’s Great Firewall is unable to completely censor Internet activity. Solutions such as proxies, VPN access, and Tor all make censorship impossible to make totally effective. If you are thinking that this is all too technical for children, you are sorely mistaken … for a start it does not take many children able to figure this stuff out as they will distribute their knowledge.

This not to say that a censorship mechanism that you control is not a sensible idea. You can select what to censor – prevent the children getting access to information about the Flying Spaghetti Monster, but block access to other religious sites, etc. And such a product has to be network-wide, to prevent someone plugging in an uncensored device; such as using the OpenDNS FamilyShield (although I have never used it, I believe it to be a good product from independent reports). Of course even DNS blocking can be worked around, but it’s a reasonable effort.

What’s Missing From The New Mac Pro

Computing No Responses »

Jun 122013

Apple’s teaser of their replacement for the venerable Mac Pro has raised quite a few hackles “out there” amongst a certain kind of Mac Pro prospective customer. They’re wrong.

It is quite possible that Apple has done some extensive research on whether internal expansion with storage and PCIe cards is necessary or not. And it is quite possible that most of the old Mac Pros had not been expanded in this way.

But Apple are wrong too (and of course I’m right whilst everyone else is wrong :-P): Internal expansion is important for some people, and they are quite possibly the sort of people that you don’t want to antagonise. Specifically the enthusiasts who would rather keep their storage internal, who want to add accelerator cards of one kind or another, etc.

Whilst the enthusiasts may not be the majority of Apple’s customers, they do have a certain amount of influence. People asking the enthusiasts at the moment may well get told to get an old Mac Pro right now so they are not limited by the expansion capabilities of the new Mac Pro.

And there’s a way that Apple could have done both; kept the neat design of the new Mac Pro, and allow the enthusiasts to have “internal” expansion. And it could be done by simply allowing the new form factor to expand the case through the base – allow it to “click” onto a PCIe expansion cage, or a two-drive enclosure.

Sure that would require some sort of special bus in the base, and a sensible way of attaching cases to the base in a secure enough manner. But it would also mean that the new Mac Pro was as expandable as the old without the use of the cable tangle that most external devices require.

Take a look behind most large tower PC’s and you’ll find a tangle of cables attaching screens, keyboards, mice, external drives, and odder devices. Apple’s new Mac Pro will just make this worse when they could have done something even more radical and showed the industry how to improve the situation.

The NSA and GCHQ Are Spying On Us …

Security No Responses »

Jun 082013

Which is news how exactly? Spying on us is what the NSA and GCHQ are for.

Over the last day or two, we have been hearing more and more of the activities of the NSA (here) and GCHQ (here) spying on “us” (for variable definitions of that word). Specifically on a programme called PRISM which monitors Internet traffic between the US and foreign nations, but not on communications internal to the US.

Various Internet companies have denied being involved, but :-

They would have to deny involvement as any arrangement between the NSA and the company is likely to be covered by heavyweight laws regarding the disclosure of information about it.
It’s also worth noting that they have asked the company executives whether they are involved in PRISM, but not asked every engineer within the company; it is doubtful in the extreme that any company executive knows everything that happens within their company. And an engineer asked to plumb in a data tap under the banner of national security is not likely to talk about it to the company executive; after all the law trumps company policy.
The list of companies that have been asked, and have issued denials is a list of what the general public think of as the Internet, but in fact none of the companies are tier-1 NSP; whilst lots of interesting data could be obtained from Google, any mass surveillance programme would start with the big NSPs.

What seems to have been missed is the impact of agreements such as the UKUSA agreement on signals intelligence; the NSA is “hamstrung” (in their eyes) by being forbidden by law from spying on US domestic signals, but they are not forbidden to look at signals intelligence provided by GCHQ and visa-versa. Which gives both agencies “plausible deniability” in that they can legitimately claim that they are not spying on people from their own country whilst neglecting to mention that they make use of intelligence gathered by their opposite number.

There is some puzzlement that PRISM’s annual cost is just $20 million a year; there is really a rather obvious reason for this … and it also explains why none of the tier-1 NSPs have been mentioned so far either. Perhaps PRISM is an extension of an even more secret surveillance operation. They built (and maintain) the costly infrastructure for surveillance targeting the tier-1 NSPs and extended it with PRISM. In particular, the growing use of encryption means that surveillance at the tier-1 NSPs would be getting less and less useful (although traffic analysis can tell you a lot) making the “need” for PRISM a whole lot more necessary.

As it turns out there is evidence for this hypothesis.

But Are They Doing Anything Wrong?

Undoubtedly, both the NSA and GCHQ will claim what they are doing is within the law, and in the interests of national security. They may well be right. But unless we know exactly what they are doing, it is impossible to judge if their activities are within the law or not. And just because something is legal does not necessarily make it right.

Most people would probably agree that a mass surveillance programme may be justified if the aim is to prevent terrorism, but we don’t know that their aims are limited to that. The surveillance is probably restricted to subjects of “national interest”, but who determines what is in the national interest? Just because we think it is just about terrorism, war, and espionage doesn’t mean it is so. What is to stop the political masters of the NSA or GCHQ from declaring that it is in the national interest to spy on those involved with protests against the government, or those who vote against the government, or those who talk about taxation (i.e. tax avoidance/evasion)?

Spying is a slippery slope: It was not so very long a ago that a forerunner of the NSA was shut down by the US president of the day because “Gentlemen do not read each other’s mail.”. But intelligence is a tool that is so useful that more and more invasive intelligence methods become acceptable. It is all too easy to imagine how today’s anti-terrorist surveillance can become tomorrow’s 1984-like society.

That does not means that GCHQ should not investigate terrorism, but that it should do so in a way that we can be sure that it does not escalate into more innocent areas. Perhaps we should be allowing GCHQ to pursue surveillance, but that it should be restricted to a specified list of topics.

Older Entries Newer Entries