It's Cool for Me

I am currently on a committee to that is reviewing the viability of turning on Layer 3 capabilities on the switches that serve at the aggregation closets of our VoIP network at University Park. Today, those switches are essentially "dumb" Layer 2 only devices. This means that each switch only has one IPv4 subnet assigned to it, and that all VoIP phones downstream of that switch rely entirely upon it for their link to the world. This configuration also means that we have very large Layer 2 broadcast networks that can potentially cause issues with our end-points.

Once upon a time when the VoIP project at University Park was getting started, there were cost-benefit analyses performed to determine how close to the customers' phones we could cost-effectively extend network redundancy. At the time, which I would like to point out was 8 to 9 years ago at this point, there was not a cost-effective solution to provide fail-over switching outside of Penn State's Integrated Backbone (IB) Core. A decision had to be made, and as I have often commented, making a decision is always better than not making a decision. The decision was made to keep the network intelligence off the aggregation switches. That decision has served us well for almost 10 years. That's right...A technology design decision lasted for almost 10 years. I call that pretty darn good.

Fast-forward now those 10 years to 2009. The hardware provided under the RFP from most vendors now has capabilities that simply did not exist 10 years ago. One of the downsides of our previous design was that to mitigate the number of customer phones impacted by an aggregation switch failure (remember these are a single point of failure today), a limit was set on the number of phones permitted downstream of the switch. This limit forces us to stop using the switch in many cases before we can fully utilize the hardware. Another limitation of using flat Layer 2 networks is that we are unable to add redundant switches to the aggregation sites now that it is cost-effective to do so.

So our committee has set out to build a list of requirements and has submitted a recommendation to the directors of TNS asking that we move forward with extending the Layer 3 capabilities to the aggregation sites. I thought that it may be useful to others working on similar projects to share our list of requirements and our answers to those requirements. So after a long-winded introduction, here's the list.

1. Preserve as closely as possible the current level of reliability and supportability
• Any changes implemented should not compromise the previous standards of serviceability. For example, we must not introduce new single points of failure to the network.
2. Reduce current replacement costs
• Under the current economic climate where many sacrifices have already been made by faculty, staff and students at Penn State, we must exercise due diligence and fiscal responsibility with the resources entrusted to us.
3. Reduce demands on physical space at aggregation sites
• Just like our office buildings on campus, our telecommunications closets and server rooms constantly struggle with available space. The new replacement aggregation switches occupy 25% of the space as compared to the existing hardware.
4. Reduce demand for physical fiber resources between sites
• Most modern switches make use of new SFP modules that allow use of a single strand of fiber-optic cable for both transmission and reception. The new switches include this feature.
5. Reduce demands on HVAC at aggregation sites
• Fewer switches in the closets will reduce the demands on the HVAC systems that are currently deployed. This reduction could be used to buy valuable time to re-engineer the systems in the future.
6. Reduce direct energy usage at aggregation sites
• Just as fewer switches equates to lower HVAC usage, fewer switches will mean less demand for electricity. This reduction in demand is aligned with Penn State's Green Initiatives for Sustainability.
7. Reduce the number of PSU IB connections
• Reducing the number of switches will also reduce the number of backbone connections required to the PSU IB Core routers. This reduction impacts both overall hardware costs as well as fiber cabling resources.
8. Make more efficient use of IPv4 space
• Currently, VoIP at University Park makes use of RFC1918 private IPv4 address space. Ideally, we would prefer to deploy IPv6 on the LANs, but approximately half of the VoIP phones today do not support IPv6. As these phones are replaced, that number will drop, but to do so all at once would cost hundreds of thousands of dollars. As a result of this cost, we would instead like to make more efficient use of IPv4 subnetting techniques to simplify the current process of address management. Subnetting by building gives us access to administrative information more quickly than we are able to gather it today.
9. Plan for future extension of redundancy at the aggregation site
• At Layer 2 functionality, we do not currently have the option to build fail-over into the aggregation sites. By deploying the new switches with Layer 3 code, we open a door for a future service improvement. In the short-term though, it must be stated that we would be slightly increasing the number of phones dependent upon a single aggregation switch.
10. Plan for future requirements of Emergency Services (911)
• No discussion of phone systems can ever be complete without covering 911 services. Centre County 911 currently requires only the building ID. Today these records are updated daily based upon billing information. Our system is currently as accurate as is humanly possible, but we are always looking for ways to improve upon it. Nobody wants break the ability to call 911 especially considering what those phone calls can mean. It is conceivable that in the future, our PSAP may require more information from us. Subnetting our network by building or even telecom closet is a first step to providing locations to emergency services in real-time based upon the location of the caller.
11. Plan for future service enhancements that may benefit our customers
• Just as subnetting by building or closet can enhance our capabilities to report location for 911 in the future, it can also be the foundation for other network intelligence services. One example that comes to mind, is how a VoIP phone gets moved at University Park today. Today, phones must be moved after a TSR (Telecommunications Service Request) is submitted with appropriate budget numbers and approvals. See the price list here. The work is then scheduled and carried out by contracted technicians. This managed process is necessary today so that we do not have phones crossing building boundaries so that we can protect the integrity of our 911 data. With appropriate network intelligence, we could eliminate the middle man from this process and enable our users to move their VoIP phones within their own buildings. Not only does a cost-recovery not have occur in this case, but we also keep some money from leaving the University.

Like I said earlier in this post. Any design that lasts 10 years is a darn good one, and I really do mean that. However, technology changes, and we must be willing to at a minimum consider changing with it. Not all change is necessarily good, but it is often necessary. So what changes are you working on? Comment away.

Standards of practice
Measuring success depends
Choose the higher bar.

We are wrapping up the big upgrade project of the summer. Documentation is being written, and procedures for maintenance are being updated. Throughout the course of the project, our biggest challenge was getting that 100% allocation of staff for the work. Sure we had a project team of three people. However, those three people were still responsible for other projects and daily duties. My best estimate is that we had 60-70% from each of the three people. This still led to inefficiency in the project due to time lost for multitasking. There are numerous studies (professional and otherwise) about how multitasking is not actually working the way we thought it did. (Here, here, here, and here for starters)

The project was for the most part successful. We jumped forward 4 versions on one of our applications and three on the other. For the first time, a technical project in TNS was designed and implemented despite the silos of Network Planning & Integration, Transmission Services Group, and Network Services Group. This change in process was unofficial, but to the credit of management, they did leave us alone to do what we needed to do.

We are now faced with a new problem. That problem is that we have created three different specialists who among them cumulatively hold all the institutional knowledge. This post by Johanna Rothman sums up the problem with specialists. So vacations, sick time, other commitments, or departure from the organization would be a terrible cost to the University. How do we solve this new problem?

1. We start a new project for documentation specifically and then assign different people as the primary resources.
2. We silence arm chair quarterbacks who would rather complain than do the work. Give them the tasks in #1 to avoid this -->.
3. We assess our goals and requirements critically to make sure they haven't changed, then write them down again as commitments. Now we go and do what is written down.
4. We then empower our staff to make changes as necessary to accomplish the goals. Avoiding management debt requires employees acting without constant supervision or fear of retribution.

Does your team face similar issues? How do you deal with it? Can you deal with it?

The culture of "No"
Why should we put up with it?
You can make this change!

It's been a long time since I wrote something longer than a haiku on this blog, but yesterday I attended a seminar in Pittsburgh hosted by two vendors that have had long-standing relationships with Penn State. These vendors for completeness are Cisco and Verizon. Obviously I am looking into phone stuff again. Spending the day in a room with people who run businesses on budgets once again sparked my thoughts.

Every day that I come to work, I spend a large part of my day trying to figure out what is best for the University. I usually interpret what is best as how can I reduce the costs of running the University. I consider it my personal mission to figure out how to save Penn State money. I do not believe that being the most expensive public University is a badge of honor. Interestingly enough, I work with a lot of people who feel the same way, but like me have trouble figuring out where to begin to change the ways things have always been.

One thing we are good at in higher-ed is writing and re-writing and refining Mission and Vision statements. I only bothered reading one of these statements. It is Penn State's mission and specifically it lists our three responsibilities--Teaching, Research, and Service. (Go ahead and read it, I'll wait, and it's relatively short.) We are expensive because we have too many people doing the same thing all across the University system. I am not talking about cold-hearted layoffs, but instead making more efficient use of the people and money that are already here. As an IT professional I will confine my commentary to the areas that I know something about. Let's talk about technology.

Historically, telephony services had to be managed at each physical geographic location because that is how Ma Bell connected you. In 2009, voice, video, and data can and should be packets on the same wire(network). Security is available, and the complexity can be managed with proper research, design, training and management. Why then do the 24 commonwealth locations not pool telephony resources and work together for Penn State? This week, TNS which is part of Central IT at Penn State turned on a new data path for commodity Internet services. We are now using both 3Rox in Pittsburgh, and MAGPI in Philadelphia as our links. If we make those IP data networks redundant, then we can do the same thing with our other communication services.

Why do our over-burdened and under-funded campuses have to pay their own way and negotiate their own contracts? Why aren't we making our voice services as University services rather than campus-specific services? The only answer that I seem to get to these questions is "We've always done it that way, and each campus has its own budget for that." Well Office Barbrady, I would like to call shenanigans on that way of thinking. The world changes, and change is hard. However, if we are not changing, then we may as well not exist. Everything worthwhile that I have ever done in my life has been uncomfortable. Off the top of my head here's a partial list.

• Putting myself through college
• Getting married
• Having my first child
• Having my second child
• Taking any new job in my career

Now, I know what you may be thinking. "Well, Mr. Smart E. Pants, tell me how you would do it then if you are so smart." So here is my answer. We just do it. Instead of spending 4 years arguing about the virtues of doing it. Let's do a business analysis of inter-site communications. Let's put a price tag on it. Let's create a requirements list, and most importantly, let's put our silos aside and stop worrying about whose budget this project has to come out of. Let's instead focus on the business of why we are here--Teaching, Research, and Service. My job is to give you the tools that you need to deliver services to the students of Penn State and the people of Pennsylvania.

Why shouldn't President Spanier have on demand video conferencing capabilities in his office to communicate with the Dean's, campus chancellor's, or his peers at other universities? Not every question requires a 30 minute commute across campus for a 2 hour meeting. Now take the next step, and ask, "Why shouldn't our students, faculty, and staff all have the same tools?", because if we can deliver a service to one person, we can scale it to the rest. My challenge to anyone reading this post, is that you should be asking, "Why", and not take, "No" for an answer. If you are a manager or director reading this post, then you should be encouraging this change.

The alternative to more efficient services is working ourselves out of a job, because someday, Penn State may be too expensive for our students. If the students stop coming to Penn State, how much of a University will we have then?

**Empty Library 2 Photo by svenwerk

Project scopes creep
Someone makes the decision
Directors wanted

Meeting running long?
Your time is important too
Give these tips a try

Ten years of your life
You will never get them back
For better or worse?

J.J. Abrams film
Resistance to this movie
Is proving futile.
Trekkies Bash New Star Trek Film As 'Fun, Watchable'

Does this list have yours?
If that is so, shame on you
Are you a Spaceball?