Recently in PSU-IT Category
On Monday and Tuesday of finals week (fall '07) , Penn State's course management system experienced extreme performance problems, a surprising anomaly in a system that had seen few interruptions of service. These problems were very disruptive to faculty, students, and staff during finals week. Our staff worked feverishly over those trying 48 hours to make the system as usable as possible under unprecedented and unpredicted load. Still, the impact to some faculty and students was profound.
Since 2001, we have done rigorous analysis on a semester by semester basis and upgraded hardware, software, and processes accordingly to meet what we believe will be the coming academic year's demands. Heading into this semester there were over 75,000 students signed up for over 250,000 course section enrollments. This finals week, we saw a demand for use not seen before and it appears that there has been a fundamental shift in how it is faculty and staff are using the service as well. This combination of factors pushed demand over the headroom built into the systems.
ITS has been working non-stop since the problems occurred to make the system improvements necessary for a successful spring semester and beyond. What we need to improve falls into two categories: system performance and communications in emergency situations. We are in contact with all of the vendors that provide the various pieces of the system to aid in the analysis of finals week and put in place non-disruptive enhancements to dramatically improve the performance headroom heading into the semester and particularly for finals week. Additionally, we are having intense conversation about how it is we can better reach faculty and students should such a situation confront us all again. It is our goal to have an update on both aspects of the challenges before us on January 7, 2008.
Penn State's course management system is arguably our best supported application/service. There are a number of first-rate personnel that support the service from the technical plumbing to training and to support in the classroom - there isn't an aspect of the service that we don't put a lot of energy and passion into, and we've been doing it for years. And even with that commitment, we've been humbled by the intensity and volume of the anger and frustration voiced by those faculty and students that were most greatly impacted. We deeply regret the disruptions that were caused during finals week, and we are doing all that we can to make the spring a great success so that faculty, staff and students can teach, learn, discover and not have the technology get in the way.
It takes years to build trust and confidence and only days or even moments to lose it. We understand it will take time to regain trust and confidence, and we've tightened up our boot straps to take that journey, however long it may be. We hope that recognition of our historic commitment to ANGEL will reduce the time it takes to regain that trust, but if it doesn't we'll keep making it better until we get back to where we were and beyond.
UPDATE, January 6, 2008:
Staff in ITS have been hard at work to continue to address the issues of finals week.
We are increasing the computing capacity for web transactions by adding 100% more Web servers. Additionally, we are in receipt of a larger, faster database server and will begin acceptance testing as soon as we possibly can.
We are working with our vendors to develop methodologies for increasing computing capacity for finals week at the end of spring semester. There have been countless emails and two conference calls to get to the bottom of the issues. There are tentative plans to conduct a summit of sorts between all parties, in the coming weeks - when the effectiveness of telephone calls has been reached
We are refining technical mitigation strategies should system degradation reoccur. Stay tuned to the ANGEL log on page for a description of those strategies.
We are developing guidelines for faculty about importing their courses and course materials prior to the start of a semester, to spread out the system load due to import/export during the first few days of classes.
We are outlining a processes for rapid crisis communications, proactively using a variety of vehicles. We also plan to meet with a subset of ANGEL users so we can better anticipate their usage at critical periods during a semester. Our future crisis communications also will outline ways the community can minimize the load, including alternatives to accomplish some tasks outside of the ANGEL system.
"If history teaches any lesson it is that no nation has an inherent right to greatness. Greatness has to be earned and continually re-earned."
"Only by providing leading-edge human capital and knowledge capital can America continue to maintain a high standard of living - including providing national security - for its citizens."
- Norman Augustine
A review of the report titled, "Is America Falling Off the Flat Earth?" is compelling reading, as is a summary of the report's recommendations. Mr. Augustine also recently briefed congressional staff on the report. His testimony can be found on the National Academies Press web site.
Let's hope our government is listening. There are strong cases made for new (or renewed) investments in education, research and infrastructure - all of which would have an impact on research institutions and their IT organizations.
Technorati Tags: cyberinfrastructure, psuits
In our last all-staff meeting one activity I foreshadowed was an upcoming presentation to our University Research Council that Vijay Agarwala and many others developed. Vijay did 95% of the talking, I just did some set up and some closing. The presentation is laid out in three sections - who we are, what we've been doing, and the results and early analysis of a recently done survey of current customers of ITS research computing and research cyberinfrastructure (or cyberinfrastructure or cyberinfrastructure or cyberinfrastructure or cyberinfrastructure - you get the point) services. The early returns have been pretty good from the members of URC. As I said in the all-staff meeting, this is a general overview but also tailored specifically for that audience. If something doesn't make sense, feel free to post a question here or ask Vijay or me directly.
Technorati Tags: cyberinfrastructure, psuits, puppy
On October 24, 2007 I attended "Forum for the Future: An Energy and Environmental Discussion."
The keynote was delivered by Dr. Richard Alley. I had somehow managed to miss numerous other opportunities to hear him speak about climate change and was thankful to hear him at this forum. If his passion and knowledge don't grab you, the topic will. I highly recommend seeing Dr. Alley if you get a chance.
I learned lots during the forum. The big takeaway for me was the commitment I made to see what I can do personally, as a family, and as an organization to reduce our contributions to global warming. I know this is sometimes a touchy political topic but I am scientifically convinced - we've got a problem. We've put sustainability in our strategic planning conversations now as something to take into consideration in all we consider.
There's a lot going on in sustainability at Penn State and a new web site was developed to act as a landing place for the many web sites that portray the working going on regarding various aspects of the energy and climate discussion.
More recently, I've been spending some time thinking about the "other side" of thinking green. More often than not, when I'm pulled into this conversation people want to talk about duplex printing, using recycled paper in printers, deploying new techniques in remote system management to allow for desktop computers to be shut off at night, buying more efficient power supplies in servers, embracing virtualization, etc. These are all important aspects of thinking green in IT, but I think there is lots of impact to be had by applying IT solutions to make more green academic and business processes. If you create 10 pounds of IT CO2 by running a service that keeps people out of cars and airplanes, the net improvement could be multiple orders of magnitude improvement.
I asked for a back of the envelope calculation with whatever assumptions someone wanted to make about how it is the Adobe Connect might have positively impacted our CO2 production. Here's an excerpt from the exchange:
"I'm thinking that if we assume 1/4 of the meeting sessions save just one person traveling 300 miles (that would be 150 miles both ways), then in October, which shows over 4,000 meeting sessions, Connect would have helped save:
1000 meetings * 300 miles = 300,000 miles
and if you expect 30 miles to the gallon, that would save:
300,000 miles / 30 miles/gal = 10,000 gallons.
A gallon of gasoline is assumed to produce 8.8 kilograms (or 19.4 pounds) of CO2 (I found this at http://www.epa.gov/otaq/climate/420f05004.htm).
If we save 10,000 gallons of gasoline in October alone, we save:
10,000 gallons * 19.4 pounds/gallon of CO2 = 194,000 pounds or 97 tons of CO2"
I know more people are doing similar analyses of how these kinds of tools can help out.
On another front, I know The Computer Store has been aggressively pursuing incentive based recycling programs. What other activities are going on to help us better manage scarce energy resources that are getting more expensive? What are we doing to reduce our creation of harmful emissions? Recycling? We're doing a lot, we could do more. Some people like to cloud the conversation with trying to establish priorities. In my opinion, our number one priority is shifting to a culture of thinking about energy consumption and emission production in all that we do - that's #1. A pretty good way to get going on that is to start doing something.
Technorati Tags: climatechange, energy, sustainability
For the purpose of this thought:
Our = information technology.
Threat = a condition, environment or stimulus that jeopardizes information technology organizations from being successful, agile, flexible.
And a big caveat here - there's lots to quibble with in here and to write it up right would take a very, very long time. I'm tired of sitting on a draft entry and want to get the idea out there.
Some months ago, I had lunch with a Penn State colleague (not in IT). We had a free ranging discussion about many matters regarding Penn State, his line of business, IT, parenting, etc. It was a really nice lunch. I had asked for the meeting because I had long been impressed by his ability to embrace, succeed and apparently enjoy a career and a position despite its significant challenges and unending supply of controversy and stress.
This colleague has a degree in law and while he isn't in a practice at the moment, he does call on his degree and experience on a daily basis. We got to talking about significant professional, personal, and educational influences in our lives. That's where I really pressed for insight and he was wonderfully open.
A professor received a bulk of the credit of his work ethic and perspective. It was then that we drifted in a discussion about an "ethic" that seems to be pervasive in the legal profession (I need some vocabulary freedom here as I'm not quite sure it's an ethic, an ethical framework or maybe even culture. I hope you'll allow me to interchange "ethic" and other words with the understanding that I realize there might be a better word or phrase).
I've watched "For a Few Good Men", "Law and Order", "Perry Mason", and "12 Angry Men" so I know a thing or two about law and lawyering. In these dramatic portrayals of life in the legal profession, we learn over and over again (and you can probably recall having seen this in your own life) that lawyers can and will change jobs over the course of their career - spending years on the prosecution side and then spending years on the defense side of the system. That's pretty interesting to me so I've lingered on this for awhile.
A lawyer isn't judged by whether or not they are on one side of the courtroom or the other, but rather on the skill, manner, and success with which they execute their parts of the legal processes. Prosecution isn't better than defense, defense isn't better than prosecution. The framework depends on both sides doing all they can the best they can and the culture/ethic among the players is such that doing one or the other isn't really a big deal seemingly. The framework helps as does the shared common goal of justice.
So what is our "justice" in information technology? I've concluded that it's service. Delivering the service is the woman with the blindfold over her eyes. Whatever you do in IT, you should be driven to deliver the service. By zeroing in on service, the service area and constituency is neutralized - and priority setting is in another layer of the discussion. If you think that this is wrong, close this tab and move on to the next wave in your surfing because you won't agree with the rest either.
In IT we don't appear to have an analogy for prosecution and defense in our framework for service delivery. And we do have multiple players in the game: engineers, developers, trainers, writers/communicators, managers, administrators, security specialists, etc. Lots of animals are necessary in our zoo so that we can facilitate service delivery.
In legal training and practice (it seems to me), there is only respect for both sides of the framework. It makes no sense to have it any other way. When programmers are learning to program, are they taught about the other sides of service provisioning - all sides of it - and that a respect and understanding of those aspects is critical for programming success AND service delivery success? How about system administrators, engineers, etc.? We're taught to do what we do and it is implied and in fact sometimes even spelled out that "my" component is more important than another. A case could be made that we're trained and encouraged to have a narrow pride and there's little room for discussion of a blind devotion to the larger common goal of service.
How do we introduce behavior that leads to cultural adoption of our own blind justice? How do we get to a place where we expect each other to do the very best we can in what we do and do so with only deep respect for what has to be done "on the other side." It probably won't happen in the classroom to the degree it would happen in law school - so perhaps this is something left up to the organization.
More, sometime soon.
I have only begun to scratch the surface of all of the ITS blogs that have sprung up since the pilot was opened up. And with even just a little scratching the possibilities continue to astound me.
I am better connected to what's happening in the library world, for example. I currently subscribe to Mairéad's, Mark's, Lisa German's, John Meier's, Peter Brantley's (UCB), and Lorcan Dempsey's, blogs . Not all are at Penn State and part of the local pilot, but all are related because the "outside" blogs were found by reading others' blogs.
I've been getting a lot out of Mark Linton's blog from TNS.
One of the newer one's I've subscribed to that I've enjoyed is Chris Stubbs' in TLT.
I know there are many more out there and that I can use the directory to try and find them. I was just calling out how much I'm getting out of it personally without even having gone on much on a hunt. Please feel free to comment with a pointer to your own blog or if you have any advice that goes beyond how to search our blogging directory.
After I make some sense out of how I'll organize the ITS blogs I read, I can see that I've got a larger process to undertake to organize the many faculty blogs that are quite active.
Continued thanks to all involved in making this effort go.
I'm always on the lookout for activities to personally participate in or learn from to better help me understand what life is really like for faculty and students - and staff in other units. And I also think about ways we can systemically improve ITS' overall appreciation for the daily life of others because I strongly believe that that perspective is fundamentally critical to designing and deploying useful services. That's the thinking behind trying to organize a fairly sizable effort for move-in weekend, for example.
Someone shared with me a youtube video this week made by some students at another university. While all story tellers exercise editorial control over their material, this video is sobering even if one takes the most cynical of views - in my opinion. I highly recommend watching it as we think about how students' perspectives and needs should be reflected in our strategic planning discussions.
I had the good fortune to attend the last few minutes of the first open house for Penn State's new e-testing facility in Pollock. You may have missed the news story about it, but don't miss one of the remaining open houses,
* October 25th, 11:00 a.m. - 2:00 p.m.
* November 13th, 11:00 a.m. - 2:00 p.m.
The facility is really impressive. There's lots of technology at play but it isn't in the way. When the facility is not being used for testing, it will act as a general purpose computer lab at the flip of a switch. The layout is inviting and functional.
About 40-50 people came to the first open house yesterday and from what I was told, enthusiasm was almost as high as interest. I highly recommend checking out this facility as it will be one that many faculty and students associate with their IT experience at Penn State.
Hats off to all involved in this project. And please, to anyone I spoke with yesterday (Chris, Jonathan, Brian, et al), jump in here and highlight anything I missed in this very brief overview of the facility.
This post would have better pictures if it were The Hulk vs. Mr. T, but then I'd probably end up violating someone's copyright.
A little over a year ago, I subscribed to MAKE magazine. I learned of it from a now defunct podcast published by O'Reilly. The driver behind the magazine is that with information technology as pervasive as it is now, there is much more surface area between IT and the analog world. MAKE set out to capture this phenomena and I think it does a great job of it. A week doesn't go by when my son and I aren't talking about some cool project we've see on on MAKE. We even set some time aside this past summer to make one of the projects we saw and it was really fun.
MAKE has also been a force behind the establishment of events called Maker Faire's. You can think of Maker Faire as the Grange Fair but with technology. Instead of showing off the pig, you show off the kite with the digital camera attached to it for taking pictures several hundred feet off the ground, for example. Or the Altoid can mod that becomes the case for your latest iPod. You get the idea.
Aside from the fact that this kind of innovation is fun and fascinating to track, it gets me wondering about what role IT thinks it has in this space. There are times I think we're (IT as much as ITS, but probably more of an issue for ITS IMO) hyper-concerned from the browser to the backbone, and for good reasons. But what about the plethora of cameras and digital signage on campus, or instrumentation in research labs, or the cornucopia of technology that now resides in residence halls rooms. Keeping in touch with how faculty, students, and peer staff are changing their IT environments to accommodate all of these new devices/agents/etc. is only going to become more important as the number of devices equals or greatly exceeds the number of computers running clunky browsers. We don't have too many first hand examples of this in ITS but we do have some. I'm wondering about how we get closer to what's going on at this IT vs. T interface.
Maybe we should co-sponsor a residence hall Maker Faire for residence hall students along with IST and Student Affairs? Maybe we could sponsor a faculty oriented Maker Faire so they could show off cool IT/T solutions they've come up with in their offices, homes, labs.
Even better, it would be great to have a place (and excitement) large enough to have both going on at the same time. 4H for technology at it's most basic level - getting things done.
Doing anything cool that you'd be willing to share here? Have any ideas about how we can raise awareness about our own Penn State MAKE attitude, so we can better prepare for the future (because it's here now)?
I am excited to announce that the Blogs at Penn State
(http://blogs.psu.edu) pilot is now officially open to all ITS
Employees. The Blogs at Penn State toolset is designed to eventually
provide all staff, students, and faculty of the University with an
easy to use web publishing toolkit. During the Fall semester, the
Blogs at Penn State project is in a limited pilot that will reach
about 1500 faculty, staff and students.
I would now like to invite you to join us as we investigate how blogs
can be used effectively across an organization like ITS. Remember,
the single biggest change here is shifting from a culture of
consumption and fighting the information glut to a culture of
contribution and finding discoverable, relevant information. Create
a blog - and start sharing what you know about the art of your work,
working at Penn State, serving faculty and students, collaborating
with colleagues across the the university, the country and the world.
Here are some details:
* If you have not already been provided an account, you can activate
your Blog Account by going to http://blogs.psu.edu/pilotenroll.html
and clicking the "Activate" link at the bottom. You will be asked to
make sure your Personal Web space is active before being allowed into
the Blogs system.
* If you have been provided a Blogs at Penn State Account at
blogs.psu.edu, you do not need to do anything further.
Since this is a pilot, support is currently limited. Additional
Instructions on how to get started using the blogs, podcasting,
controlling comments and how to customize the appearence are
available at the Blogs Community Hub at: http://blogger.psu.edu. I
have personally found this to be a great place to go for help and
insight.
For advanced web users of the years - a note an Access Control Lists
and your personal web space. Blogs at Penn State publishes content
to your www/blogs/ directory. If you have have modified ACL's
permissions within your Personal Web Space, you may need to reset
ACLs to the defaults. See http://blogger.psu.edu/help/advanced/acl
for more details
If you have any questions about the pilot program you can post
questions at the Blogger Forum at http://blogger.psu.edu.
BASIC FEATURES
* Multiple blogs per user
* Customize sidebar and style themes
* Supports tags, categories and monthly archives
* Published in your personal web space (you can blog about whatever
topic you want)
* Comments from anyone
* Generates (RSS) news feeds and podcast feeds
* Ideal for announcements, journaling, portfolios, note taking, and
discussion.
* You can register your blog at http://blogger.psu.edu/directory to help
others find your blog
NOT ENABLED YET
* Multiple authors in one plog - only one editor per blog (but
anyone can comment)
* Private blogs - all blogs are visible to the public
Start sharing and tagging!
Thanks to the folks who have been working so diligently on getting
this new service off the ground.
