Recently in Penn State Category

Oy, it's been a while since my last update. Lots of news to report. I'll spread that out over several posts.

Last month, I gave a talk on Campus IPv6 Deployment at the Winter 2009 Internet2 Joint Techs workshop in College Station, TX. The gist of the talk was that the main barrier to IPv6 deployment is attitude and perception. Although there are plenty of technical issues to overcome, the technology is good enough for initial deployment, and we have to start doing that now. The hurdle to overcome is attitudes like "no one is using IPv6" and "IPv6 is a network issue."

People are using IPv6. And there's a lot more to a successful IPv6 deployment than upgrading some routers and firewalls.

I've never seen IPv6 as a network issue. Maybe it's because I'm not a network engineer. Maybe it's because I've always thought the OSI model was a fiction. But I approach IPv6 deployment from the perspective of services not devices. E.g., what services do we need to IPv6-enable to make IPv6-only machines viable from a policy compliance perspective?

We have some good news to report on this front. At Penn State, we've got a fair bit of buy-in about the importance of IPv6 from the senior leadership in central IT. Our deputy CIO has tasked me with leading a team to develop an IPv6 deployment plan for central IT. We're only about two months in, so we don't have much to report yet. I should say that we are only charged with developing a plan; we're not yet implementing it, although I am pushing for little bits of deployment whenever I can. We should have the final report done in a few months, and I should be able to post more then.

.edu gets IPv6 glue!

| | Comments (0) | TrackBacks (0)

Last night, the .edu domain got IPv6 glue in the DNS root. I'm thrilled beyond words at this.

Most top-level domains (e.g. .com, .org, .uk) already have IPv6 glue. I'm glad Educause made this step. It's been a long-time coming.

Educause operates .edu under contract from the US Department of Commerce. In turn, they sub-contract the day-to-day operation to Verisign, which also runs the .com and .net root servers. From what I understand, Verisign is migrating .edu over to the .com and .net cluster, which will give .edu IPv6 support.

Educause has allowed subdomains to register IPv6 glue for a while ( has had IPv6 glue for at least year). The DNS root got IPv6 support earlier this year. But .edu was always this big v4-only gap in DNS. I'm really happy that it's being fixed.

$ dig -t NS edu.

; <<>> DiG 9.4.2-P2 <<>> -t NS edu.
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17220
;; flags: qr rd ra; QUERY: 1, ANSWER: 8, AUTHORITY: 0, ADDITIONAL: 9

;edu. IN NS

edu. 144225 IN NS G3.NSTLD.COM.
edu. 144225 IN NS H3.NSTLD.COM.
edu. 144225 IN NS L3.NSTLD.COM.
edu. 144225 IN NS M3.NSTLD.COM.

A.GTLD-SERVERS.NET. 172700 IN AAAA 2001:503:a83e::2:30
M3.NSTLD.COM. 172700 IN A

;; Query time: 21 msec
;; SERVER: 2610:8:6800:1::4#53(2610:8:6800:1::4)
;; WHEN: Thu Nov 20 10:07:57 2008
;; MSG SIZE rcvd: 334

I mentioned recently that I gave an IPv6 poster session at the Penn State WebConference. I've put the poster and handout online. I spoke to a couple of interesting people at the conference, and learned a lot about the web infrastructure at the University (there are more IIS and ColdFusion shops that I realized. Fortunately, both supports IPv6.)

In DNS news, the .dk (Denmark) and .nk (Saint Kitts and Nevis) top-level-domains are now reachable via IPv6. This brings the total to 190 of 269 TLDs that are reachable over IPv6.

TNS has fixed a long-standing bug with otc2, one of our authoritative DNS servers. As of today, recursive DNS queries will now be answered for users within Penn State's IPv6 network (2610:8::/32).
Penn State is a multi-campus institution. The University Park campus is commonly thought of as the "main campus," but we have 26 other commonwealth campuses. We also have an affiliate relationship with the Pennsylvania College of Technology. I have a personal connection to Penn College -- both my mother and step father worked there for over a decade, and both were deans there for several years.

Penn College isn't part of Penn State. ITS does not provide networking services for them, and Penn College has its own IP address space. I was trolling around whois last week and noticed that Penn College has IPv6 space! And they're using it: The Penn College chapter of the ACM has IPv6 on their web site.
In my previous blog entry, I reported at PSU has about 300,000 IPv4 addresses assigned to it, and that ITS manages a little more than half of those. ITS projects that it will exhaust its IPv4 pool sometime in late 2009 or early 2010. This graph is based on data from TNS in January 2008:

The text on the graph is hard to read. Click on it for a larger version.

When we finally exhaust our pool, will we able to go ARIN and get more? That depends on the status of ARIN's IPv4 pool and on any new policy ARIN has adopted. Several of the RIRs have adopted policies which make it much more difficult to obtain additional IPv4 space. I believe that ARIN will still have addresses left, but we probably won't be able to get space due to policy changes. If that's the case, it's a pretty scary realization. Quite likely, we'll be stuck with the IPv4 space we already have.

It will be several years before ITS runs out of IPv4 space, but the end is closer than many people think. No, the sky is not falling, but I firmly believe that we have to start planning now for what to do when we do run out of space. IP addressing will be a serious issue during the University's next strategic planning period (2009-2014).

Several network managers on campus have already begun efforts to conserve IPv4 space. They're doing things like moving networked printers and internal servers to private address space. But there are only so many printers to move! These steps are important in the short-term to preserve business continuity, but ultimately, they only relieve the pressure temporarily. We're still left with the question: What is our long-term solution to the address crunch?

Some within the university have proposed using static NAT for desktops. This approach has issues, including potentially breaking certain applications. The number of apps broken by NAT has declined in recent years, in part due to protocols such as STUN, TURN and ICE, to name just a few. Frequently, NAT-enabled apps will have to support multiple such protocols. Adding NAT traversal support to an application can add considerable development and testing costs (see this paper on Skype for a good example of NAT-enabling a real-world app). Given the aggressive use of NAT today, many developers are already adding NAT support to their apps.

Others are proposing using IPv6. Of course, deploying IPv6 isn't free either. There are both hardware and software costs involved. New routers and firewalls have to be purchased. Of course, network equipment needs to be regularly replaced anyway to keep up with increased user demands. If IPv6 support is introduced as network equipment is replaced, deployment costs can be considerably reduced (but not eliminated). Software may need to be upgraded to use IPv6-safe APIs or to remove hard-coded IPv4-assumptions. There will certainly be testing costs. Fortunately, many programming languages (Python, Ruby, Java) and networking libraries (CFNetwork, Qt, NSPR, APR) are already IPv6-enabled. This support tremendously helps reduce programming costs. Even so, there will certainly be at least a few legacy apps which will be very expensive to convert to IPv6.

Ultimately, I think this is an economic question: In the long-term is it cheaper to (more aggressively) deploy NAT and NAT-enable our apps, or to deploy IPv6 and v6-enable our apps? Any measurement of cost should include the factors outlined above plus opportunity cost.

My vote is for IPv6. Why? Because the graph above scares me. I'm just not comfortable living in a network where we've used up most of our publicly routable addresses. Even with extensive use of NAT, I'm afraid that such an environment limits our ability to deploy new applications and services. Even if Penn State can reclaim significant portions of its IPv4 space, the story doesn't end at our border router. What about our partners? What about network operators in Europe and Asia who are already deploying IPv6? If we need to interoperate with them, we'll need an IPv6 story of our own.

I'm curious what others think. How do you answer the question?
A brief history of IP address allocation at Penn State, based on whois data.

March 31, 1986:

We got five Class C networks (256 addresses each):
We also got a Class B network (65,536 addresses):

October 5, 1988:

The Department of Computer Science and Engineering got its own Class B network (so another 65,536 addresses):
A good portion of this space is shared with the rest of the University. For example, about 3,300 of these addresses are in the University's dial-up modem pool.

February 26, 1991:

We got another Class B network (another 65,536 addresses):

August 7, 1991:

The Hershey Medical Center got a Class B network. This netblock was and still is managed by Hershey Medical Center, not by ITS.

September 25, 1991:

We got another Class C (256 addresses):

February 5, 2001:

We got a /17 (32,768 addresses):
This is used for the residence halls.

January 13, 2006:

We got our permanent IPv6 prefix (which can hold 4,294,967,296 sub-networks, with 18,446,744,073,709,551,616 addresses each):



Total IPv4 addresses assigned to Penn State: 296,448

A little over half of those addresses are managed by ITS. While it seems like PSU has a lot of addresses (and we do), ITS estimates that we'll run out in less than 5 years. More on that in a future blog entry.

WebAccess and IPv6

| | Comments (0) | TrackBacks (0)

If you are IPv6-enabling a website protected by WebAccess, you might need to make a change to your web server's configuration. Specifically, you might need to set CosignCheckIP never in your Apache config. Otherwise, connections over IPv6 won't be authenticated. You can read more about why this happens in the ITS wiki.

Penn State's WebAccess product is based on CoSign, an open-source web single-sign-on system. The CoSign source code is not IPv6-clean, so even if ASET were to assign IPv6 addresses to the WebAccess servers, we'd still have problems. I'm working on a patch to CoSign to make it IPv6-clean. After the break, I'll do some more testing and submit it to the CoSign project for integration.

I had some freetime this afternoon, so I whipped up a script to check IPv6 deployment at the Big Ten. I checked for AAAA records for authoritative nameservers and SMTP servers.

Only University of Illinois, Urbana-Champagne and Penn State had any IPv6 presence. We both have one IPv6-accessible authoritative nameserver.

Yes, is IPv6-accessible. It's answering DNS and NTP queries over IPv6. At the moment, it's not answering recursive queries over IPv6, but TNS is aware of this issue and hopes to have it corrected soon.

So, hats off to TNS for this. It's good to be ahead of the pack.

The ITS Wiki is now IPv6-accessible!

Both the web server and its authoritative nameservers are IPv6-accessible:

$ dig any

; <<>> DiG 9.3.4 <<>> any
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53202
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 5, AUTHORITY: 0, ADDITIONAL: 4


;; ANSWER SECTION: 21600 IN SOA 2007111600 3600 1800 1209600 3600 21600 IN NS 21600 IN NS 21600 IN A 21600 IN AAAA 2610:8:6800:1::a

;; ADDITIONAL SECTION: 21600 IN A 21600 IN AAAA 2610:8:6800:1::4 21600 IN A 21600 IN AAAA 2610:8:7800:9::4

;; Query time: 0 msec
;; SERVER: 2610:8:6800:1::4#53(2610:8:6800:1::4)
;; WHEN: Fri Nov 16 18:58:52 2007
;; MSG SIZE rcvd: 259