February 2008 Archives

A few entries ago, I said that the choice between pushing the life of IPv4 and deploying IPv6 comes down to economics:

Ultimately, I think this is an economic question: In the long-term is it cheaper to (more aggressively) deploy NAT and NAT-enable our apps, or to deploy IPv6 and v6-enable our apps?
I listed several factors that influence the cost of both IPv4 and IPv6. But I forgot one big one: If you deploy IPv6, you have to keep your IPv4 network around. For a long time. I'd like to explore that cost in this entry.

I don't think there are any capital costs involved in maintaining IPv4 support. Any networking equipment we have already supports IPv4, and will continue to do so for the foreseeable future. All network software supports IPv4, and will do so for the foreseeable future. No, I think the costs are labor costs. For example, everytime we deploy a service, we have to configure it twice, on both IPv4 and IPv6. This is called dual-stacking our services. We will have to dual-stack our DNS, Kerberos, directory, mail, and web servers, as well as the firewalls that sit in front of them.

This does have costs. Not in terms of configuration, that's usually trivial. The first cost is mindset -- we have to convince our network and system administrators to think about the word "network" differently. They will have to learn that "network" means "IPv4 and IPv6." Programmers will have to learn this same lesson. It's a habit that takes a while to develop. The second cost is testing, and this will be a big cost. Many systems have no problems with IPv6 and dual-stacking. But a minority of them do. Isolating and fixing those problems is time-consuming. This process is confounded by programming languages that aren't amenable to IP-agnostic development.

For example, there are anecdotal reports that when some sites turn on IPv6, 10% of their web traffic disappears. This doesn't happen for everyone (I've not experienced it in ET), but a few people report it. What's happening to those missing packets? Are they lost in a blackhole route somewhere? Does some client have broken DNS software? It's a laborious process to find out. And it is not always the case that the problem is solved simply by upgrading to the latest software at both ends.

So there's a lot more to deploying IPv6 than turning it on on your routers and firewalls. That's only a very small tip of a very large iceberg. To paraphrase a common line: It's the apps, stupid.

This is a serious cost, but it's one I think IT needs to bear. The good news is that everyone who is deploying IPv6 has to bear it, which helps by distributing the cost over many organizations. And people who have deployed IPv6 are willing to help one another out. Two good places for IPv6 help are the ARIN IPv6 wiki and the ITS Wiki IPv6 space. The latter is not currently accessible to non-Penn State users, but we hope to rectify this in the near future.

Two more down

| | Comments (0) | TrackBacks (0)

We're down two more /8 blocks in the IANA free pool. About two weeks ago 173/8 and 174/8 were allocated to ARIN. We're down to 42 blocks free:




This really highlights the need to develop a long-term solution to the IPv4 addressing crunch. In 2007, ARIN managed to reclaim three legacy blocks. It took a lot of work, and ARIN isn't optimistic about reclaiming the remaining 41 legacy blocks.

Bottom-line: In one month, we've used up two-thirds of what it took a year to reclaim. That's not sustainable, folks.

TNS has fixed a long-standing bug with otc2, one of our authoritative DNS servers. As of today, recursive DNS queries will now be answered for users within Penn State's IPv6 network (2610:8::/32).

IPv6 over WiMAX

| | Comments (2) | TrackBacks (0)
The IETF just approved RFC 5121, which defines how to send IPv6 packets over 802.16 (commonly called WiMAX). For those of you not familiar, WiMAX is a next-generation high-speed wireless networking technology. WiMAX isn't heavily deployed yet, but it's good to know that there will be IPv6 support for it.
Last week, six of the DNS roots got IPv6 addresses. This begs the question: How well does the IPv6 internet perform? What's the network latency to reach the root servers over IPv4 versus IPv6?

I did some ping tests from AS3999. The F root isn't too much slower over IPv6, J and M are actually faster over IPv6, and K is a lot slower. It isn't surprising that the K root is slower, since it has only one host in North America (the rest are in Europe and Asia).

Contrary to popular belief, IPv6 isn't always slower than IPv4. In some cases, it's significantly faster!

Here's the raw data:

F server:
IPv4: round-trip min/avg/max = 75.096/75.122/75.142 ms
IPv6: round-trip min/avg/max = 88.437/89.268/90.966 ms

J server:
IPv4: round-trip min/avg/max = 120.905/121.093/121.335 ms
IPv6: round-trip min/avg/max = 90.373/91.438/93.089 ms

K server:
IPv4: round-trip min/avg/max = 91.265/91.540/92.323 ms
IPv6: round-trip min/avg/max = 171.145/171.990/172.969 ms

M server:
IPv4: round-trip min/avg/max = 174.806/175.067/175.411 ms
IPv6: round-trip min/avg/max = 89.096/90.527/92.050 ms

The A and H servers aren't pingable over IPv4, so I excluded them.

(As an aside, notice that above I said "DNS roots" not "servers." There are 13 addresses for the DNS root servers. But there are a hell of a lot more than 13 root servers.)
Yesterday evening I reported that GMime didn't handle IPv6 addresses in URLs. (Note the past tense).

As of this morning, the bug is already fixed in svn.

Wow. That's the fastest turn-around I've ever had for a IPv6 bug. Hell, that's incredible response time for any bug. Hats off to Jeffrey Stedfast for great work!
Severity: Normal

Today's bug - GMime doesn't parse URLs with literal IPv6 addresses. This has been reported to them as bug 515088.

GMime is a popular library for handling MIME. Penn State's WebMail system uses it. It has a very useful module that converts plaintext email containing URLs into HTML. WebMail uses this capability. Unfortunately, it doesn't support URLs with IPv6 addresses, such as http://[2610:8:6800:1::7]/. This is a valid URL, per RFC 2732.

From what I hear, the GMime developers are very responsive to bug reports. I look forward to this being fixed. It'll be one small step to making WebMail IPv6-capable. :)
I've been looking into what it would take to get several of our applications IPv6-enabled. In some cases, it's trivial. In others, it's going to be hell. This post is about how several programming languages have added IPv6 support to their standard libraries. Well, it's not really a post. It's more of a rant. Because I'm angry. I'm angry that two of the most popular languages appear to have put no forethought into future-proofing their networking libraries.

PHP and Perl are excellent examples of how not to add IPv6 support to a library. The status of IPv6 in both systems is an utter mess that will take years to fix.

We have many apps written in PHP and Perl, which we will eventually need to IPv6-enable. Given the abysmal state of IPv6 in these languages, I'm not optimistic about success. Critically, it not possible to write applications in PHP or Perl that use a single API and will work on IPv4-only, IPv6-only and IPv4/IPv6 dual-stacked hosts. The inability to do this is dooming.

Non-programmers can skip to the last paragraph. What follows is techy.

Let's look at PHP. It got IPv6 support in version 5, but its gethostbyname() function still doesn't support IPv6. Of course, the documentation doesn't mention this fact. Instead, users should use dns_get_record(). (As an aside, why does dns_get_record() still support A6 records, which were deprecated years ago?). Alternatively, developers may use PEAR's Net_DNS class to resolve names. It appears as if there are cases where Net_DNS_Resolver::query() can fail to use IPv6 to connect to a DNS server, however (if using UDP and the "enhanced sockets library").

Then there are the IP address utility libraries: Net_IPv4 and Net_IPv6. First question: Why are there two of them? Why is there not a single network address library that can handle both formats? But it gets worse: The two libraries have different APIs. There is considerable functional overlap between them -- there are functions to apply a netmask to an address, parse CIDR-formatted addresses, etc -- but the function prototypes look nothing alike. For example, some of Net_IPv4::parseAddress()'s abilities are duplicated in Net_IPv6::getNetmask() and Net_IPv6::removeNetmaskSpec(). Both Net_IPv4 and Net_IPv6 have functions to verify if a string is a valid address, but they have different names: Net_IPv4::validateIP() vs Net_IPv6::checkIPv6(). There are other examples of this sort of mismatch.

I'll only mention briefly that there are two other modules for validating IP(v4) addresses: Net_CheckIP and Net_CheckIP2. Both of them only handle IPv4 and have different function names than the equivalent functions in  Net_IPv4 and Net_IPv6. This only serves to increase programmer confusion and encourage the development of non-IPv6-capable code.

And that's just the low-level code. There are still the higher-level libraries for dealing with IMAP, SMTP, HTTP, etc. I haven't begun an audit of PEAR's Networking library to check for IPv4-only code. I'm too afraid.

Perl isn't any better. For reasons that escape me, Perl decided to fork its network libraries into IPv4-only and IPv6 halves. Specifically, there are Socket (and it's object-oriented cousin, IO::Socket::INET) and Socket6 (and IO::Socket::INET6). Of course this means that developers have to change their code to get IPv6 support. Almost no OSes bundle Socket6 or IO::Socket::INET6, so developers are loathe to make such changes. (Kudos to Mac OS X 10.5 and Ubuntu 7.10 for supporting them).

Because the modules are separate, you have to write a bunch of ugly code. When your code starts, you check if you have one of the IPv6 socket modules installed. If so, set a flag. Then every single time you need to talk to the network, you check this flag and branch, either to Socket6 or Socket. This is a) ugly b) inefficient c) unlikely to be adopted by virtually all programmers d) all of the above. Check the source to the Net::DNS module for an example. I am not looking forward to submitting patches that do this. But someone is going to have to.

I contrast this to Java, Python and C which have sane, clean, standardized APIs that support both IPv4 and IPv6. In Java, it's unlikely that developers will have to change code. In Python, the changes are usually two or three lines. In C, it's slightly more lines than that, but generally not much more. I'll elaborate more on these languages in a later post. Readers at Penn State may want to look at my IPv6 programming notes in the ITS Wiki (which is IPv6-enabled).
Penn State is a multi-campus institution. The University Park campus is commonly thought of as the "main campus," but we have 26 other commonwealth campuses. We also have an affiliate relationship with the Pennsylvania College of Technology. I have a personal connection to Penn College -- both my mother and step father worked there for over a decade, and both were deans there for several years.

Penn College isn't part of Penn State. ITS does not provide networking services for them, and Penn College has its own IP address space. I was trolling around whois last week and noticed that Penn College has IPv6 space! And they're using it: The Penn College chapter of the ACM has IPv6 on their web site.