HP Tech Forum

| | Comments (0) | TrackBacks (0)
Mark Campbell and I made the trek to the HP Tech Forum this year in Las Vegas.

At the opening keynote addresses Mark Hurd, Randy Mott and Ann Livermore CEO, CIO and EVP had very compelling stories of direction shifts becoming increasingly nimble and incredibly agile in change.

I want to start by saying I try to stay as vendor neutral as possible but HP definitely eats its own dog food. 

I'm sure most have a heard of HP data center consolidation efforts over the past couple of years starting in 2005. Well the story is they'll be done in another few months.  That's taking 86 data centers in 29 countries and consolidating into 6 in 3 US locations.  I shouldn't have to do the math but it's in 3 years!  The big take away here is that they were able to effect change in a very positive way and become the leading server vendor last quarter to boot!  Yep beat "Big Blue".  Now shipping a server every 12 seconds.  Sorry Steve!

HP still has a substantial footprint in that they have about 21.6 football fields worth of raised floor space (that's 16.7 football pitches for Kevin). They also are very green, they have 40% less servers with 250% more processing power.  Have reduced power consumption by 60% and I think they said will save what is equal to a 350 acre forest annually.

But back to change, the fact that HP transformed their data center operations in 3 years is still amazing to me.  Randy Mott said that the biggest mistake most organizations make is to plan over time.  Change needs to be transformational and not incremental.  This leads to uncertainty with many different choices, discussions, that don't need to happen and ends up watering down the goal.  "Ongoing Choosing is Loosing", was another take away for me!

Risk only comes from not knowing what you're doing, says Warren Buffet. 

As a side note that may be interesting to some is that EYP Mission Critical Facilities did much of the design for the data centers and consolidation efforts.  They did such a good job HP bought them and are now selling their services.

All in all a very good conference and some new products that sound as if they will be very competitive.  One such product that caught my eye is the new Extreme Storage Solution (ExDS) StorageWorks 9100.  This seems to hit the price mark with many features that includes manageability as well as a road map for hardware life cycle and ILM. This is not just a bunch of spinning disks, you can actually manage your destiny.

I'm still pretty vendor neutral (Really!)
 


Data Center World, Spring 2008

| | Comments (0) | TrackBacks (0)

The following are notes that I made during two keynotes at Data Center World.  I didn't do a great deal of editing so please let me know if you have any questions!


Data Center World, Las Vegas

Keynotes

04-01-2008

 

Michael Manos,
Sr. Director of Data Center Operations, Microsoft

 

Microsoft Challenges

  • Microsoft increases the number of servers 15,000 – 20,000 per month
  • They double the number of servers each year
  • Power
  • Tens of megawatts of power usage

Why Power matters:

  • Federal government will not be able to build power stations fast enough
  • Sustainability regulations are coming
  • How efficient are you and can you back up the numbers
    • Monitoring will need to be in place
  • Building a program to monitor and measure facilities is not an overnight process
  • Cost of 1U server is much less than the power to run them.

Not solving the problems together?


Economics are changing

  • Power bills may not be seen by DC managers
  • Power is a more significant component
  • 30% of DC managers don’t think power is an issue

Industry Challenges/Minefields

  • Sustainability reporting/ efficiency reporting
  • Data center inventory “Globally”
  • Increasing power densities at the rack level
  • Power densities per rack keeps growing
    • 8 – 36kw per rack
    • Power is driving the square footage costs
  • Power Costs
  • Green-washing
  • Expertise shortage
  • Organizational structures
  • Increasing capital cost barriers
  • Innovation hoarding
  • Heterogeneity versus homogeneous mindsets

 

To drive a .1 PUE decrease, decreases 7 million kw (Microsoft)

  • MS could spend up to $500M per data center

$11 – $18M cost for construction of a new data center and will grow 10% per year.

  • Not based on housing industry
    • Completely different skills involved
    • Mostly electrical and mechanical (80%)

Data centers at MS

  • Bring Moore’s law to the data center
    • Unfortunately the data centers have not kept up
  • Challenged Microsoft team to bring it to the DC
    • Innovation hoarding
    • MS will share its knowledge

 

Data Center Packman  (Typical DC Costs)

  • 2% Land
  • 9% core and shell
  • 7% architecture
  • 82% Mechanical and Electrical

Density Questions

  • If average rack now is 16kw how can we plan for the future
  • How do we balance for high density VS less densities
    • OK, if you are consuming all the power

Microsoft Best Practices

  • Building 500,000 sq ft data centers on average
  • 40k – 100k systems per building
    • Highly optimized
    • 10k – 20k growth per month
      • 35 people spread across MS to manage deployment
      • Gone is the days of unpacking servers
        • DRIVING AUTOMATION IS KEY
        • Discovery of the servers and imaging
  • Site selection is critical for new data centers
    • MS is growing their numbers
      • Some are consolidating
  • Carbon footprint is already set if you currently have a data center
  • Look at where the internet is and power pricing
  • Environmental
    • How can it be optimized in certain areas
    • Going green is good for the bottom line
    • Construction costs
    • IT costs and availability in area
    • Power cost in the area
    • Where is the carbon emission factor in the area
    • Mobile users
    • Tax climate or breaks

 

PUE VS DCIE

  • MS uses PUE and are reciprocals of each other
  • Engineers love DCIE
  • PUE shows the inherent overhead
  • Doesn’t matter but must drive efficiency
    • MS drives these targets internally

If I can half the distance by 1 each year brings Moore’s law into play

By 2012 PUE below 1.2 for Microsft


MS uses a tool they call scribe to evaluate their efficiency

  • On average of 1M points of monitoring per each data center
    • CPU, memory, overhead temps, etc…
    • Sensors can tell what is going on in each rack and replay events in the DC
  • Measure an environment as
  • Cleaning the roof on one of MS’s DC’s was able to reduce PUE (very large facility)

Top ten best practice that have been published by MS


MS Datacenter in Chicago is modular in design

  • They will have standard containers that will hold up to 2,000 server that they back into a data center and hook up power, liquid for cooling, network and ground.  Everything else is pre configured at the vendor site

Could we use IE students or project to model a data center build??


What’s next

  • MS participating in Energy Star Forum
  • Share with industry

 


Hu Yoshida

VP and CTO – Hitachi Data Systems

 

Sustainable Data Storage is more than Green Technology

Storage can’t be moved since it’s stateful and will be destroyed if not handled properly.  Typically server are stateless since they can be plugged and unplugged without loss.

The Restriction of Hazardous Substances (RoHS) Directive 

Move data off current storage

  • Consolidation
    • Fewer footprints, fewer LOCATIONS
  • Increase utilization
    • Of server, storage, network, power and cooling resources
  • Eliminate redundancies
    • Delete, compress, dedupe, store and process once
  • Reduce the working set
    • Archive stale data from production systems
      • Technologies coming that will scale to petabytes

 

Management software

  • Common user interface
  • Efficient management
  • Better resource utilization
  • Reduce staff training
  • Improved TCO

Archiving as an environmental tool

  • 70% of data over 60 days old is rarely referenced again
  • This is toxic waste, wasting time, money and fossil fuels
  • Keep one gold copy in a single place

Optimizing storage capacity

  • Thin provisioning
    • Unallocated space is huge
    • Growing storage as needed, giving people what they want but not allocating it until it is touched or provisioning only when used


Thin provisioning coupled with virtualized disks you can gain up to 60% efficiency

Virtualization can help move storage as needed without disrupting anything

Tiered storage reduces power and costs

Not all data needs same performance

Intermediate copies used for backup

Massive array of idle disks (MAID)

De-duplication

 

Which storage is more efficient

  • Modular storage
  • Monolithic storage
    • Since there are fan’s at the top and bottom instead of in each module this storage tends to be more efficient

Look to Storage Neworking World (SNIA) http://www.snia.org for metrics on storage efficiencies

ISO 14001 references sustainability

 

 

Building a Data Center Glitz or Glory?

| | Comments (0) | TrackBacks (0)

I've gone over this in my head and been involved in discussions about what happens when and "IF", we decide that we need a new data center. There are several approaches that seem very efficient, effective and will produce huge results. And, then there is the "Glitz", factor that I think has as much impact as the best engineered, green, sustainable data center design.

Let's look at a couple of scenarios:

As many know I'm deeply involved with the RFP that will ultimately produce a master plan that will suggest what we need to do with data centers at Penn State. Presuming we are told that we need at least one data center, what are we to do? Should we be engineering geeks and squeeze the nickel till the buffalo (you know the rest). By this I mean take the approach of say a less expensive piece of real estate (if that exists) that's not in the middle of campus. Say USB3 and build another steel building that has the best power, cooling, backup etc, etc...... What do we have? We have a great data center? Right?

Well, yes, BUT, who will want to bring there project here or who will want to put their NAME on it? I don't see any names coming up to replace USB 1 and 2?

The point of it all then becomes how "pretty", do we have to make it so it sells!??

I think we'll have to take a $40 million dollar building and make it a $50+ million dollar building so it will bring the research and the development monies to the University. I'm not saying by any means this is a bad thing but I'm betting that researchers would rather say that they are doing their research in the "John and Jane Doe" Research and Engineering Data Center at Penn State! The beautiful facility that it is overlooking the duck pond and arboretum. Rather than saying I'm doing my research at USB 3 behind OPP at PSU. You get the picture.

Of course we're a long way from any of this becoming reality but It's been on my mind so let me know what you think?

By the way, I do think option 2 is the right thing to do in the long run!

ITS Climate Survey

| | Comments (0) | TrackBacks (0)

As you can see, it's been a while since something has struck me in a big enough way to take my Saturday evening (yes it's Saturday at 20:02)

I'm going to stick my neck out yet again and comment on the ITS Climate Survey. I'm going to be general at first and offer help at the end!

I think I hate surveys as much as anyone but in this case and with the size of our organization it's difficult for a person or persons to personally contact each one of us to get the job done. I saw Kevin's note regarding reluctance of some folks in wanting to fill it out. From my point of view, I don't understand why? From Mairead, to Jeff to Kevin I truly believe they are trying to do what's right!! Not just for them but for ITS and The University. This survey is important to see where the pain points exist and there's no other way to get the answers and take corrective action unless you ask, is there?

So, for those folks that have pushed it aside, Please fill it out. Here's the link: https://online.survey.psu.edu/itsclimate? You have to put your survey number after the question mark!!

Now for the helping and sticking my neck out part!
If anyone feels that there may suffer untoward consequences for filling out the survey truthfully, please contact me and I'll do everything possible to get your comments to the proper place anonymously! Yes, I'll know who you are, but you got to trust someone if you want to be heard!

I know only a handful of folks read this but if you think it's important pass it along to those that may be hesitating to send the survey and I'll do all I can to help!

Green Data Centers

| | Comments (0) | TrackBacks (0)

Many of us are acutely aware of the costs associated with the operation of a data center. Not to mention the exponential demand for more processing power and even MORE storage.

Just returned from a "Green Data Center" briefing with my OPP colleague Ian Salada.

What many are not considering are the strategies around taking advantage of opportunities to be more green and energy efficient in current installations.

At the briefing there was a great deal of emphasis on virtualization rather than just simply adding an additional server to our never ending collection. By virtualizing our current server farms we can essentially cut energy costs nearly in half. This holds true a bit more on the wintel side of the fence because of the proliferation of a singe server for each application. This produces typical processor loads in the less than 10% range thus wasting a subsequent energy load. If for example 5 or 10 machines are virtualized; There would be a need for a more powerful server but only two power supplies are required instead of 10 or 20 and this may be accomplished with less than half the processing power than the original configuration. There may be a fallout savings in database or other per processor licensing too. Nothing new just restating to underscore the importance of thinking about it as we grow.

Here's some stats:
* Between 2000-2010 there will be a 6X server growth, 69X storage growth
* The per square foot energy cost is up 10 - 30%
* Data centers have doubled their energy consumption in the past 5 years.

Where does the energy go?
* 30% is typically used by the IT equipment
* The rest is used by chillers, UPS, CRAC units and power distribution

There was an EPA Report to Congress August 2, 2007 where EPA lists 3 areas or types of improvement: (executive summary worth reading)
* Improved efficiency (low hanging fruit, moderate energy conservation)
* Best Practices (aggressive server consolidation and all of the above)
* State of the Art (aggressive virtualization with cutting edge cooling)

This underscores the need to look at the entire infrastructure of the physical plant and not just what's in the data center.

There was some really cool stuff (literally) regarding techniques of stored cooling. The most interesting was the "Cool Battery". This is a device that uses a synthetic phase change material (don't ask me what it is) that freezes at up to 43 degrees F. Basically a big honkin ice cube. Up to 3000 ton/hr units are available. The material can be frozen by your chillers during off peak times and the chillers can be turned off at peak times thus saving on energy costs or credits the institution would be in line to receive.

There are also techniques of utilizing waste heat that is generated constantly by the IT equipment. This could be used to heat offices around the data center.

There are some very compelling reasons to keep this in front of us but we have to be very careful in our selection criteria. As improvements in IT equipment continue there will be a point where techniques to further heat or cool the areas may no longer be cost effective to implement.

Lots to think about!

Data Center RFP

| | Comments (0) | TrackBacks (0)

After many meetings and quite a bit of work from many dedicated team members the Data Center RFP hit the street Tuesday. If you don't follow this blog, (I know not much to follow), we've been working on an RFP that will bring in a consulting firm to study the data centers and server rooms at Penn State.

The team members are: Ian Salada, Co-Chair, Mark Saussure, Co-Chair, Vijay Agarwala, Lisa Berkey, Dave Beyerle, Deborah Blythe, Ron Dodson, Ken Forstmeier, Ben Grissinger, Tom Irwin, Mike Kauffman, Mark Miller, Kevin Morooney, J. Daniel Morris, Neal Vines, Jeff Wolfe, Jeff Kuhns, Meg Harpster, Monica Reed

The current plan is to have a pre-bid conference in a couple of weeks to allow potential bidders to ask questions and for us to answer as best we can regarding the scope of the project and deliverables among other things. Meg Harpster from purchasing has been a tremendous help in getting this document together and advising on what we can and can't say. Now and in the future while working through the process.

I've been on several RFP committees but never as co-chair. There are many logistical issues that you can imagine that must be handled, none greater than getting Kevin and Jeff in the same room at the same time! At any rate once we have our pre-bid conference at UP we will be traveling to Altoona to allow the potential bidders to get a flavor for one of our campuses. Of course Altoona being one of the largest we'll explain some of the challenges as well as the efficiencies of those smaller.

All of IT at Penn State will be involved in this process. The survey that I alluded to in last months blog will be just a guideline. There will be many site visits required by the winning bidder to best understand the parameters of each server space at Penn State. I believe the range of server spaces described in the RFP echo what we all know as good raised floor areas to closets that most likely shouldn't be used for servers at all. Of course it doesn't say it like that but we want the best spaces evaluated along with the worst to be sure we have the proper picture.

I'm not sure if I've said this publicly but I truly believe this is one of the single most important projects that will shape the future of IT at Penn State. If I can speculate on the results, and subsequent building projects, this will allow us to be more flexible, responsive, and collaborative than ever before.

I have many thoughts, aside from the money it will take to make this come true. Stay tuned!

ITS ITANA and Data Center Activities

| | Comments (0) | TrackBacks (0)

Looking into the future I think I'm currently involved in two groups that pretty much reflect the crown jewels of the decision makers in IT at Penn State. (Granted there are a few more)

ITS ITANA was modeled by Kevin and Jeff after the national ITANA organization that stands for Information Technology Architects in Academia. The local group is charged with first getting to know each other in terms of their respective organizations and bringing to the table issues and suggestions to improve our collaboration and over time improve our efficiencies in ITS. I think the group is a great mix with no one having any preconceived notions! I think this is a huge plus!

It was interesting that Jeff Reel, Steve Kellogg, Scott Smith and I had already planned to meet but this fits in perfectly with ITANA since all four of us happen to be members. We met for the first time last Friday and it became apparent that even if we don't actually form a sub-committee as part of ITANA that we should continue to meet to discuss issues such as Disaster Recovery and Business Continuity along with just sharing ideas to become more informed and efficient. Not to mention diving deeper into how we may be able to better understand our respective groups and produce services or share spaces that make sense. An example being: Steve and I trading computer spaces for DR. Well at least Steve tried until he figured that the transformer would probably glow a bright red if I plugged my stuff in. Thanks to Jeff Wolfe and Vijay we did some more trading. Isn't that how it's supposed to work, doesn't Penn State sign all of our paychecks?

At any rate that's a perfect segue into talking about the future of our Data Centers at Penn State.

Kevin asked me along with Ian Salada (OPP) to co-chair an RFP committee to commission a study of ALL Penn State data center spaces (yes all). Without going into great detail you can imagine the scope and impact of this project.

IT organizations across Penn State were asked to fill out a survey at the beginning of the calendar year. The data received will be used to help the successful consulting firm gain knowledge about us and our current practices. With that said, it will be somewhat open ended for the consultant to make recommendations regarding future construction, business continuity, tactical and strategic planning to name a few.

I believe this project (along with subsequent activities) (and a LOT of money) will ultimately shape the way Penn State is seen worldwide as an institution positioned to take on the challenges of IT for generations to come.

I'm just happy to be associated with these groups!

Storage, Repositories and XAM

| | Comments (0) | TrackBacks (0)

I'm sitting in the airport (now on the plane), returning from SNW (Storage Networking World) with Ben Grissinger. No Steve it doesn't stand for Still Not Working!

At any rate we attended to be present when SNIA announced the new XAM (eXstensible Access Method) SDK and most importantly demonstrated the interoperability between HP, EMC and Sun storage. They also demonstrated it on top of their respective ILM platforms. HP's IAP (formerly called RISS), EMC's Centera and Sun's ST5800 (Honeycomb). All we saw worked smoothly!

At a breakfast yesterday the CTO of EMC Centera made several comments regarding this standard and was acknowledged by both HP and Sun that after this announcement they will be incorporating the XAM standard into their road maps and in the case of Sun into their OS. Without going into great detail regarding the XAM SDK we talked about how the community needs to be built. This seems to be the best attempt out there to allow interoperability as well as the ability to bring silo'd repositories under a single searchable storage platform. An analogy was made to that of NFS when it was first introduced. No one can dispute the uptake and mainstream of NFS. SNIA is asking that we do our part to get the word out regarding XAM and the rich features of it's SDK.

We have been talking extensively with all three vendors and how we could play a role in further development. Of course our interest would be the ability to bring several of our repository products under one searchable storage platform. Much of the functionality we're looking for is part of the ILM platform we ultimately choose. Having the ability to create and extend the metadata of fixed content objects will be huge for future initiatives and development. An example today of having thousands of .pdf files in a "repository" system that one can search but may not be able to figure out the version or have the attributes to determine what version of Acrobat the object was created. With XAM and an ILM solution we could ultimately be able to reach into the archive, save the original copy for any compliance reasons, then extract a copy and transform it to the latest version. Thus creating a path for our archived data to be literally preserved as well as being readable for the foreseeable future!

An ILM platform creates a system that is very robust, scalable and has the ability to take care of compliance issues as well as simplified life cycling of the actual hardware, now becoming agnostic to what storage medium offered over time. Once the data is ingested you will never have to touch it to move it from one storage platform to the other. Resulting in near zero down time seen by end users.

Yes, it's expensive but one must factor in the net worth it in respect to the preservation of institutional, intellectual and historical data over time and the inherent protections this system offers.

Let me know what you think?

Common Solutions Group Meeting (CSG) and Long Term Storgage

| | Comments (0) | TrackBacks (0)

I just attended my first Common Solutions Group (CSG) meeting and found it to be very informative and interactive. You can find a bit more about the meeting and organization at http://www.stonesoup.org And, I met Jack from UW Madison who shoots skeet!!

I'll take the first topic on the agenda which was right on target with what we're looking at in the realm of repositories and how to aggregate data in the years to come. The title was: Shared Media & Data Repositories, and What's New in Scholarly Systems

The first presentation was by Jim DeRoest from The University of Washington. They have been working on storage of video data and it has morphed into adding work flow for images and other media. Their repository called DigitalWell (link to slides) is billed as an asset management system but by my definition it is much more. I have to note one of my ITLP colleages (Jim Kerkhoff) from UT Austin has been part of a fine arts repository and also presented some great information (UT Austin Slides). Both are home grown systems and are gaining traction at their respective institutions.

The question that I just can't get answered by anyone that has anything to do with development of an archive, media storage service, repository (whatever you want to call it), is: How do you plan on storing the data and who is going to pay for it long term?? I'm astonished that nobody seems to be thinking or worrying about it? It's going to cost lots of money to store this stuff over time. I can't figure out if everyone is too scared to answer or they think that storage is or will be such a commodity over time that it's not going to be an issue. I don't think that flies in the near term (5 - 7 years).

We are now seeing institutional and historical data being created in pure digital formats. What happens to the data over time and how do we assure that this data is preserved? Please remember I'm not an advocate of keeping everything. Actually I'd prefer keeping very little, but I believe there is truly data that will be lost because few are thinking about services around this long term preservation of born digital data.

Now, what if we do create a service around this archive, repository, media service...... Oh, and I have to remind you that it has to be easy to use and scalable or nobody's going to touch it. Then what's the best platform to assure we preserve it for the long haul?

DLT is currently working on questions regarding data that must be kept for the life of a student, not to mention publications that most likely will never be deleted. That's a long time in computing terms. We're working to answer the questions around best practices, platforms and how we life cycle the actual hardware and software over time (can we trust migration tools?). One must assure what's put in is what someone can get out, or a representation thereof 70+ years later? Will it be disk, tape, Content Addressable Storage (CAS) or a combination of them all?

Feedback will be great in this area, I'd like to get as many opinions as possible!

Helpdesks of the Future or Past?

| | Comments (1) | TrackBacks (0)

I was just looking at an entry that Jeff Kuhns posted on his Strategic Planning Page regarding how we approach Help Desks in the next 5 years.

I have always thought we could do better in how we provide service at our help desks. I want to clarify this right away in saying I'm not talking about the people involved in any way, I'm talking about the work flow and how we deliver the help or services! Basically getting the right people to the right place at the right time, to intercept the person needing help and delivering quality service.

This may go a bit far for some to agree but hear me out and maybe we can glean some worthwhile approaches from a totally different perspective (or is it?).

As many of you know I'm a paramedic and had worked as one for about 20 years while still working here at PSU. What I'm going to describe is what I had a hand in developing for a 6 county Emergency Medical Services (EMS) council well over a decade ago. Computer Aided (Assisted) Dispatch (CAD).

The concept of computer aided or assisted dispatch is far from new, with most if not all 911 centers in the country applying similar procedures and practices. For those not familiar with these procedures, the use of the computer, GPS, two way radios, cell, wireless, etc... is to streamline the decision making process and dispatch the correct emergency services to the proper location in the least amount of time. An integral part of this process gives the caller or patient a person to talk to prior to the arrival of more highly trained personnel. Having the person stay on the line is not only to help ease the person's mind or offer life saving assistance, it allows for updates to the responding agency(s). And in the worst case scenarios provides instructions for life saving procedures.

Here's how it plays out, pay attention to how we could use some of this work flow. A telecommunications officer answers the 911 call by saying " 911 what is your emergency", this is done to take immediate control of the call and requiring the caller to answer questions. The caller says my significant other is having chest pain. Immediately the person answering the call knows that it's an EMS related emergency. This call can then be transfered seamlessly to the EMS Telecommunications officer, typically in the same room. This person at the console in front of him has the telephone number and address of the person calling on the screen, and can dispatch an ambulance in seconds while still talking with the caller. There is a book or screen on the multi-monitor console that the tele-communicator can type "chest pain", and get back the steps (in flowchart format) in which they can convey to a lay person the initial life saving treatment. If there are updates the Telecommunicator can update the responding units and prepare them for arrival. Only upon the arrival of the emergency responders is the call disconnected. This provides the seamless delivery of services and hopefully a very positive experience for all parties involved.

This was just a quick scenario of something that goes on each day but with a bit of a different slant. How many times have you received a call at a help desk and it was as if they should have called the 911 center?

Ok, why can't we use this model in the next several years to improve the overall service we give to our students, faculty and staff? I know it takes money and time to develop but it may be worth it. The biggest problem I see are the disconnects between help desks and the hesitation of looking at a central approach. A single point of contact for all may be much more palatable than a bunch of different numbers to call with the person needing the help trying to figure out when and where to call. You will have a (The) knowledge base available and may be able to assist the caller without passing it on to the next help desk. And if you have to pass it on you have a flow chart on the screen or in a book that shows when a college, department or other ITS help desk is open and staffed to answer the question. If it's during normal business hours the call can seamlessly be transfered to the appropriate help desk with the caller never being dropped until there are three people on the line to agree that the outcome of the call will be handled at that location.

Let me hear what you think?