Living with IPv6

More IPv6 progress around the world:

• The Linux NFS community released a working prototype of an IPv6-capable NFSv4 client and server. The linux-nfsv4 list has more details. I suppose I should point out that Solaris and AIX have supported NFSv4-over-IPv6 for a few years.
  


• The Amsterdam Internet Exchange (AMS-IX) saw a new high for IPv6 data: 0.7%. AMS-IX is the largest internet exchange in the world.


• Tajikistan's TLD DNS servers are IPv6-enabled. With this addition, 74% of the top-level domains are reachable over IPv6. Of course, few domains accept AAAA glue for child-domains, which limits the ability to deploy IPv6 more fully.


• Speaking of DNS, Iowa State University v6-enabled their DNS servers. At the DNS root, Netnod is working to v6-enable the I root.


• At Penn State, the Center for Comparative Genomics and Bioinformatics did a thorough IPv6 deployment. They v6-enabled their web, SMTP, IMAP, and DNS services. The College of Arts and Architecture got its first IPv6 allocation. I'm anxious to see what they'll do with it. In ITS, we fixed a long-standing bug with the ITS Wiki's RSS feed; it's now reachable over IPv6. ITS also v6 enabled another of PSU's nameservers, so now one-third of them are IPv6-enabled.


• I had my first patch accepted to WireShark. It adds support for parsing IPv6 addresses in Kerberos tickets.


• Sun fixed a (very) long-standing IPv6 bug in Java. This should significantly improve IPv6 support for modern I/O calls.

There has been a lot of IPv6 activity since my last blog post. I wanted to give folks an update.

• Belize's top-level-domain servers are reachable over IPv6.
• The University of California, Santa Barbara v6-enabled their DNS servers. UCSB does secondary DNS for Brown and Oregon State, and they got dragged along too.
• ClamAV added IPv6 support in version 0.94.
• IPv6 traffic at the Amsterdam Internet Exchange, the largest IXP in the world, continued to increase. The percentage of IPv6 traffic (compared to IPv4) hit a new peak, 0.4%.
• Several Nokia firewalls received IPv6 Ready certification. Penn State uses Nokia firewalls for the ITS firewall service, so this is good news for us.
• On a personal note, I had an IPv6 patch accepted into OpenSolaris. It added IPv6 support to the whois(1) utility.

The most interesting event was the APNIC 26 conference. Google's talk made the case for IPv6. We've long known that there are limits to NAT's scalability. Google is concerned that "AJAX applicatiosn break behind excessive NAT" due to ephemeral port exhaustion. Essentially, modern web apps create lots of connections, and that limits how many clients a NAT box can handle. This claim was echoed by NTT in a related talk. NTT tried to put some numbers on how many clients can be placed behind a single public IP address. They measued the number of concurrent TCP sessions created by popular "Web 2.0" services (YouTube, iGoogle, iTunes Music Store, etc), and used this to get some ballpark numbers for how many clients can be comfortably NATed. The numbers aren't encouraging.

Google had some good advice on how sites could do IPv6 deployment. They emphasized building testing and pilot services, and stressed that IPv6 doesn't have to be at parity with IPv4 on day one. I was glad to see this, since I frequently encounter defeatist thinking regarding IPv6: "Our {firewall, load-balancer, etc} doesn't support feature X for IPv6, therefore we can't do anything with IPv6!" This is simply false. IPv6 isn't an all-or-nothing proposal. While there are certainly problems, many of them aren't show-stoppers. For example, at Penn State, the ITS Wiki is IPv6-enabled, but it's not at feature-parity with IPv4. It's not load-balanced on IPv6, and the backend database isn't IPv6-enabled. The web double-sign-on system protecting it isn't IPv6-enabled either. We'll get to those issues later. For the moment, it was more important for us to do something rather than nothing. It doesn't have to be perfect on day one.

This week, Arbor Networks released a report on the status of IPv6 deployment. There's also a well-written blog entry on the subject by the author. They claim it is the "largest study to date on IPv6 traffic on the Internet."

The results are not encouraging.

We've known for some time that IPv6 adoption is lagging far behind where it should be. It's been well-reported that IPv6 traffic makes up a tiny fraction of total Internet traffic. But Arbor's numbers are even smaller still.

Arbor claims that less than 0.01% of Internet traffic uses IPv6:

While this number is sobering (perhaps shocking is a better word), I think Arbor's numbers are low. The study has a few issues which I think led to undercounting of IPv6 traffic. The Arbor study was heavily biased towards North American and European ISPs. Of the 98 providers in the study, only 6 were in Asia. A fair amount of IPv6 adoption is in Asia, so it seems odd to exclude that region.

Ignoring the regional issues, Arbor's report is a little confusing on the issue of native versus tunneled IPv6. They claim to have collected data on both, but their graphs only show tunneled traffic. It also claims that they have difficulties extracting data on native IPv6 due to equipment limits at participating ISPs/IXPs:

Our measurements showed a negligible amount of native IPv6 traffic. However, since we do not currently have a good estimate for the number of monitored routers in our study that are capable of exporting flow records for native IPv6 traffic, we are unable to conclude whether this is due to a true lack of native IPv6 traffic or simply due to the limitations of the network reporting infrastructure itself.

Arbor tried to adjust their figures to include native IPv6, but there are no good estimates for the ratio of native to tunneled IPv6 (and Arbor admited this in their report). I wish they had provided more detail about how many participants provided data on native IPv6. It would also have been very interesting to see the data broken down by RIR region. Hopefully this data will be forthcoming in a follow-up report.

Nevertheless, if you know where to look, you can find IPv6 usage that's higher than Arbor's findings. The Amsterdam Internet Exchange (AMS-IX) is one of the largest Internet Exchange Points (IXPs) in the world. They provide real-time, public data on IPv6 traffic:

According to AMS-IX, native IPv6 makes up 0.1% of its traffic. This is still far too small, but it's 10x greater than Arbor's value for tunneled IPv6 (0.01%). Granted, this is only one data point, but it's an encouraging one.

There are other reasons to be optimistic. If you look ad application-domain-specific statistics, you see numbers that are higher than Arbor's. If you look at DNS, for example, you see about 1% of the traffic at the root is over IPv6. Again, this number is far lower than it should be given than the IANA pool will run out in three years, but it's much higher than Arbor's numbers.

Geoff Huston, chief scientist at APNIC, reported earlier this year that 0.4% of traffic to the APNIC website is over IPv6. Again, while this number is still frighteningly small, it is orders of magnitude larger than Arbor Network's findings.

(As an aside, I never thought I'd be using these dismal numbers to defend IPv6 deployment).

Why are these numbers so low? Several reasons. First, relatively few DNS registrars accept AAAA glue, making it harder for sites to make themselves reachable over IPv6. Second, many sites are deploying IPv6 in stages. Usually they get v6 DNS
first, and only much later move on the higher-level services like HTTP. For those that do have IPv6 HTTP, I have to wonder how many of them have it on their "www" name. Neither Google or the 2008 Olympics do. Nor does freenode. Anything that requires a user to go to a different URL is going to reduce the amount of traffic it gets.

But there is reason to be optimistic. The amount of IPv6 traffic is increasing. If you go back to AMS-IX's stats, you see this for IPv6:

There's been quite a surge in recent months. You see similar results for DNS traffic. From the H-root:

Given this recent surge, I'm still cautiously optimistic. It appears as if the recent media coverage of IPv6 has been paying off. I'm very anxious to see if this increase in traffic continues through year's-end.

Last Friday I gave a talk on IPv6 for the Penn State Mac Admins group. It was received well. Surprisingly, there were a lot of questions about IPv6 tunneling for home use. Fortunately, Apple makes it very easy to setup 6to4 tunneling -- it takes 5 mouse clicks on Apple's Airport products. The talk was mostly praiseworthy for Apple, who I think has done a really good job integrating IPv6 into most of their products.

Slides are here, if you're interested.

There's been a lot of great progress on IPv6 this summer. I'd like to highlight some recent news items:

Google

A few months ago, Google launched a preview IPv6 service. That made their search engine, maps and a few other services reachable at a special URL: http://ipv6.google.com/. They've recently expanded this service to include their cache (note the IPv6 address in the URL):

This is a good missing piece to fill in. I'm still waiting for IPv6 at www.google.com, though.

Wikipedia continues IPv6 testing

Wikipedia has been testing IPv6 throughout the summer. They've already v6-enabled several of the secondary services (mail, bug tracker, Subversion repository, etc).

Many organizations are still afraid to use IPv6 on their main web site. There's a lot of FUD and myth that IPv6 will break clients and drive away visitors. Wikimedia is empirically testing these claims. The preliminary results look encouraging -- less than 1% of their test clients show problems.

.org

The .org domain just announced that it will accept IPv6 records. They've had AAAA glue in the DNS root for a while, but until this week they didn't allow .org's to have AAAA glue. This is really great news. 70% of the top-level domains have IPv6 glue in the root, but not many of them accept AAAA glue themselves.

nmap

nmap, the venerable port-scanning tool, now includes IPv6 support in its Windows version. This is great for security conscious sysadmins who want to test their systems.

Some admins think they're safe from port-scanning because IPv6 subnets are so big (18 million million addresses per subnet). This just isn't true. There are many ways for an attacker to reduce this search space. I highly recommend reading RFC 5157 for guidance on IPv6 port scanning.

Huge surge in IPv6 DNS traffic

In February, 2008, the DNS root enabled IPv6 transport. During spring 2008, IPv6 traffic at the root was relatively flat. But starting in late June, IPv6 traffic has steadily increased, and is now nearly four times higher than the spring average:

Independently, there has been a significant increase in AAAA queries. At one of the largest ISPs in Japan, almost 20% of the DNS traffic is IPv6-related:

I took the graph from a presentation at the 2008 DNS Operations Workshop.

I just got back from the Summer 2008 Internet2 Joint Techs Workshop in Nebraska. There was a heavy focus on IPv6 at the workshop. The IPv6 Working Group announced its IPv6 Challenge. This is a challenge to Internet2 members to IPv6-enable several aspects of their networks. Joe Nasal from TNS participated in a very interesting panel discussion on campus IPv6 addressing plans (video here). Both Penn State and Stanford were on the panel. We both have a /32 from ARIN, but we've chosen to use them in very different ways. I found that rather interesting. As usual, DREN gave a useful update on their IPv6 deployment in the Department of Defense.

I gave a lightening talk on a quick way to IPv6-enable lot of DNS servers (slides here). Essentially, there is a "clustering" effect in DNS, where one server will provide authoritative DNS for three or four other domains. For example, UC Berkeley provides DNS for Columbia, UC San Francisco and UCLA. So v6-enabling that one server provides considerable extra benefit. If you map out these "clusters" in DNS, you get a list of the most beneficial servers to target. It turns out that by v6-enabling 11 extra servers, you would give 31 domains v6-reachable DNS. Put another way, that's 15% of Internet2 members, but requires upgrading only 1.5% of its nameservers.

In non-IPv6 news, there was a good DNSSEC talk from NIST. This was especially interesting in light of the recent Kaminsky DNS attack. Suffice it to say, there are still a fair number of hurdles to integrating DNSSEC.

I just got the email. OpenSolaris just got support for NATing IPv6 connections. To make matters worse, this was done in order to support transparent proxies.

Now I definitely need a drink.

iPhone 2.0 is out. It still doesn't enable IPv6. I wasn't optimistic that it would be added in iPhone 2.0, but I'm still sad to see Apple abdicate leadership on this issue, especially since Mac OS X has had solid IPv6 support for years. Many handset OSes already support IPv6.

Qualcomm's proprietary BREW platform has supported IPv6 since version 3.1.5. Nokia's snazzy new N78 and N96 both support it as well. Actually, all of Nokia's E-Series, N-Series, and most of their Communicator, 6000- and 7000-Series phone have supported IPv6 since 2005. Microsoft added support in 2003. Symbian, VxWorks, QNX and J2ME also support it.

The good news is that much of Apple's SDK is IPv6-clean, so 3rd-party apps should be IPv6-capable. Hopefully it will just be a matter of Apple enabling v6 on a later software update.

I've recently been handed an assignment that requires me to write some Perl code. I'm not a fan of Perl anymore. I haven't used it routinely in almost seven years. It certainly filled a very useful niche in the '80s and early '90s, but by 2008, I think it's been superceeded by Python or Ruby. Frankly, I think it's an anachronism. But this isn't a blog about programming.

I've griped about the shoddy IPv6 support in Perl before. This post is in the same vein.

This code needs to use Perl-LDAP. Natually, I checked if that library supports IPv6 (remember, kids, in Perl, you have to check if every single library supports it!). Fortunately, it got support a few months ago, in version 0.35. All's well, right? Not so fast, kiddo. The changelog says it "add option to support IPv6". What does "option" mean?

Looking at the docs, I found this gem:

So let me get this straightish. To get IPv6 support in my LDAP apps, I have to pass a special "no, really, use IPv6" flag to every LDAP object I create? What happens if the hostname resolves to both an IPv6 and an IPv4 address? And I better just hope that I'm not using SSL. This is just charming. But it gets better. If you follow the link to IO::Socket::SSL, you find this:

Gah! You can't possibly be serious. Java has supported SSL-over-IPv6 and mixed v6-v4 apps for 5 years! Hell, it's supported it for so long that the first version to get support is now EOL. Python has had support for the same amount of time.

Folks, this is not the way to design a maintainable, supportable, enterprise language. How people continue to use Perl in production totally escapes me. And this does not bode well for IPv6-enabling the large mountain of legacy Perl scripts that hold this University's IT systems together.

A week-and-a-half ago, I commented that 10% of Internet2 schools had IPv6-reachable DNS. It's time to add one more: Georgetown. And, unlike many schools, Georgetown isn't piggybacking on someone else's DNS server.

For those of you keeping count, here's the updated list:

IPv6-reachable DNS in Internet2

• Columbia University
• Georgia Institute of Technology
• Georgetown University
• Indiana University
• Internet2
• Ohio University
• Pennsylvania State University
• Portland State University
• Princeton University
• University of California, Berkeley
• University of California, Los Angeles
• University of California, San Diego
• University of California, San Francisco
• University of Delaware
• University of Illinois, Urbana-Champagne
• University of Iowa
• University of Notre Dame
• University of Oregon
• University of Pennsylvania
• University of Rhode Island
• University of South Florida
• Virginia Tech
• Wichita State University
• Worcester Polytechnic Institute