« Report on American Competitiveness | Main | More coffee, more coffee, more coffee »

An ANGEL update

On Monday and Tuesday of finals week (fall '07) , Penn State's course management system experienced extreme performance problems, a surprising anomaly in a system that had seen few interruptions of service. These problems were very disruptive to faculty, students, and staff during finals week. Our staff worked feverishly over those trying 48 hours to make the system as usable as possible under unprecedented and unpredicted load. Still, the impact to some faculty and students was profound.

Since 2001, we have done rigorous analysis on a semester by semester basis and upgraded hardware, software, and processes accordingly to meet what we believe will be the coming academic year's demands. Heading into this semester there were over 75,000 students signed up for over 250,000 course section enrollments. This finals week, we saw a demand for use not seen before and it appears that there has been a fundamental shift in how it is faculty and staff are using the service as well. This combination of factors pushed demand over the headroom built into the systems.

ITS has been working non-stop since the problems occurred to make the system improvements necessary for a successful spring semester and beyond. What we need to improve falls into two categories: system performance and communications in emergency situations. We are in contact with all of the vendors that provide the various pieces of the system to aid in the analysis of finals week and put in place non-disruptive enhancements to dramatically improve the performance headroom heading into the semester and particularly for finals week. Additionally, we are having intense conversation about how it is we can better reach faculty and students should such a situation confront us all again. It is our goal to have an update on both aspects of the challenges before us on January 7, 2008.

Penn State's course management system is arguably our best supported application/service. There are a number of first-rate personnel that support the service from the technical plumbing to training and to support in the classroom - there isn't an aspect of the service that we don't put a lot of energy and passion into, and we've been doing it for years. And even with that commitment, we've been humbled by the intensity and volume of the anger and frustration voiced by those faculty and students that were most greatly impacted. We deeply regret the disruptions that were caused during finals week, and we are doing all that we can to make the spring a great success so that faculty, staff and students can teach, learn, discover and not have the technology get in the way.

It takes years to build trust and confidence and only days or even moments to lose it. We understand it will take time to regain trust and confidence, and we've tightened up our boot straps to take that journey, however long it may be. We hope that recognition of our historic commitment to ANGEL will reduce the time it takes to regain that trust, but if it doesn't we'll keep making it better until we get back to where we were and beyond.

UPDATE, January 6, 2008:

Staff in ITS have been hard at work to continue to address the issues of finals week.

We are increasing the computing capacity for web transactions by adding 100% more Web servers. Additionally, we are in receipt of a larger, faster database server and will begin acceptance testing as soon as we possibly can.

We are working with our vendors to develop methodologies for increasing computing capacity for finals week at the end of spring semester. There have been countless emails and two conference calls to get to the bottom of the issues. There are tentative plans to conduct a summit of sorts between all parties, in the coming weeks - when the effectiveness of telephone calls has been reached

We are refining technical mitigation strategies should system degradation reoccur. Stay tuned to the ANGEL log on page for a description of those strategies.

We are developing guidelines for faculty about importing their courses and course materials prior to the start of a semester, to spread out the system load due to import/export during the first few days of classes.

We are outlining a processes for rapid crisis communications, proactively using a variety of vehicles. We also plan to meet with a subset of ANGEL users so we can better anticipate their usage at critical periods during a semester. Our future crisis communications also will outline ways the community can minimize the load, including alternatives to accomplish some tasks outside of the ANGEL system.

Technorati Tags: ,

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

About

This page contains a single entry from the blog posted on December 22, 2007 1:23 PM.

The previous post in this blog was Report on American Competitiveness.

The next post in this blog is More coffee, more coffee, more coffee.

Many more can be found on the main index page or by looking through the archives.

Creative Commons License
This weblog is licensed under a Creative Commons License.
Powered by
Movable Type 3.33