PASS Migration Complete

| 3 Comments | 0 TrackBacks
We did it!

Yes, we managed to successfully copy all of the data from old PASS (DFS) into new PASS (GPFS) in less than 68 hours.  If you include the time it took us to bring servers back online to the "mostly up" state, then we're talking 23 hours to resolve the alert.

Why "mostly up"? 

Most of the work wasn't simply the copying of data. It was also the porting of programs to the new environment.  Own our tools for PASS and PASS related services took several years to build and refine to the state they were in prior to the migration, and we had a small fraction of that to adapt to the new environment.  Some parts like the Club Server Administration Console were closely fitted against the particulars of DFS and had to be recoded and retested substantially.  This change also affected many other groups, such as the various programming groups of Penn State World Campus, where we needed to be involved in helping the conversion.  While we are now officially, fully running from new PASS, there remains much to be done to get back to where we were...and beyond that to where we need to be.  But now that old PASS is behind us, we can finally move forward with new development.

Programming wasn't all the work either.  There was lots of testing, documentation changes, retesting, fixing problems, retesting, fixing new problems from old fixes.  The client software landscape was also changing right under our feet over the past 6 months since the PASS Beta was launched which provided additional challenges with our Samba and NFS gateways.  Some of this is still ongoing.  A lot of people were involved, and that took a considerable amount of coordination.

7.2 Terabytes in 23 hours?

Actually this was a lot less, both in time and data size.  Using both an incremental data rsync as well as an incremental ACL permissions copy (well "translation" may be a better term), we were able to cut down what was originally worst case 74 days to approximately 7 hours of a final pass during the 3 day weekend window.

So what changed?

Much has changed during this transition, not just the storage technology piece (GPFS replacing DFS) and the corresponding features (e.g. permissions and quotas), but connection methods (kerberized gateways), application engines (PHP switch from Solaris to Linux, apache 1 to apache 2) and the list goes on.

What I'd like to offer here is a list of the various sundry details for the more technically curious.

  • DFS replaced by GPFS
    • Multiple server access to shared disk reduces rate of failures that have plagued us in the past (most alerts on PASS were related to hardware failures that would have not been a problem for GPFS); recovery is now counted in seconds, not minutes to hours
    • Native client access to disk performance improved 10-fold; this improvement can be seen in various PASS based services
    • DCE infrastructure replaced by MIT brand Kerberos and IBM's Directory Server (LDAP); DCE to be shut down before fall semester
    • Quotas no longer fileset based, now numeric Group ID (GID) based
      • mandatory GID inheritance (setgid on folders not required)
      • quotas are now "hard"; enforcement is on write, no longer on file open
    • Access Control List (ACL) permissions are now based on the new NFSv4 standard
      • IETF standard (RFC 3530), no longer rescinded POSIX draft
      • support available from both the Samba and NFS gateways; client tools come bundled with the platform
      • 14 instead of 6 permission types
      • "deny" as well as "allow" entry types
      • arbitrary rule order replaces POSIX strict evaluation order
      • ACL inheritance set by inherit flags on each entry; no longer separate ACLs for inheritance
      • chmod has no effect on ACL; setuid, setgid and sticky still apply
      • stat() mode now approximate
      • Penn State custom "simple" ACL tools help ease usability
        • stand-alone ACL Explorer Web-based tool replaced by PASS Explorer integrated tool, with wizard workflow based on the Protected Personal Access Control Manager (ACM)
  • Gateway Authentication
    • Samba (SMB/CIFS) Gateway now accepts kerberos tickets from the client
    • Samba (SMB/CIFS) Gateway no longer accepts NTLM based authentication
    • NFS Gateway now accepts kerberos tickets from the client
    • NFS Gateway authentication mapping Web application no longer supported; sys=auth is no longer sufficient for access
  • Gateway exports
    • The "dfs" Samba share, e.g. \\win.pass.psu.edu\dfs, has been decommissioned.  Use the "pass" share instead.
    • The NFS exports nfs.pass.psu.edu:/.../dce.psu.edu/fs, nfs.pass.psu.edu:/.:/fs and nfs.pass.psu.edu:/: have been decommissioned.  Use nfs.pass.psu.edu:/pass instead.
  • Gateway server changes
    • Samba upgraded from 2 to 3
    • NFS protocol version 4 now supported; legacy version 3 still supported
    • NFS service now has load balanced backend (3 systems instead of 1 to ensure higher availability)
  • Path changes:
    • PASS is no longer mounted at /.../dce.psu.edu/fs.  It is now mounted at /pass.
    • For a limited time, the old, deprecated links /.../dce.psu.edu/fs, /.:/fs, /: will now point to /pass.
    • /:/cactus is now /pass/services/cactus
    • /:/rs_aix, /:/solaris, /:/linux, /:/dist are now, respectively, /pass/os/rs_aix, /pass/os/solaris, /pass/os/linux, /pass/os/dist
  • SSH Host key changes
    • (for current key for both sftp and rs6klab, see http://kb.its.psu.edu/psu-all/hd/passssh/)
    • sftp.pass.psu.edu, a.k.a.
      • ftp.pass.psu.edu
      • sftp.personal.psu.edu
      • ftp.personal.psu.edu
      • sftp.clubs.psu.edu
      • ftp.clubs.psu.edu
      • lutz.cac.psu.edu
    • rs6klab.aset.psu.edu, a.k.a.
      • ptp.cac.psu.edu
      • splogin.cac.psu.edu
      • krumsville.aset.psu.edu
      • slickville.aset.psu.edu
      • slickville.cac.psu.edu
      • dimsville.aset.psu.edu
      • dimsville.cac.psu.edu
      • frackville.aset.psu.edu
  • UNIX changes
    • Solaris systems (cbs.aset.psu.edu, armstrong.aset.psu.edu) unavailable for the time being due to resource constraints; expected replacements/upgrades before fall semester 2008
    • dcecp, acl_edit, dcerchacl, fts and similar DCE/DFS commands no longer available
    • account information now integrated with LDAP
    • Secure FTP service now provided by rs6klab
    • command line ACL permission tools provided by the OS vendor (aclget, aclput, acledit on AIX, etc.)
      • recursive, Penn State custom permission "simple" command line tool due out before fall
    • quota tools provided by the OS vender
    • "Native" client no longer available for use; NFS client now provides similar functionality to that of DFS native clients
  • PHP changes (php.scripts.psu.edu)
    • Platform change from Solaris to Linux
    • Vendor built RPM packages for apache, php, etc replace custom built binaries
    • Apache upgraded from 1.3 to 2.2
    • PHP still 5.1.6
    • SQLite extention discontinued; SQLite databases may continue to be accessed via PHP Data Objects (PDO)
    • register_long_arrays now disabled; deprecated $HTTP_GET_VARS and $HTTP_POST_VARS, etc. no longer available; functionality still provided by $_GET and $_POST arrays
  • WebAccess conversions
  • User Managed Group Updates on June 30
    • Paperless UMG creation
    • Improved interface
    • Single owner group may now own multiple UMGs
  • PASS Explorer changes
    • New file permissions tool available via (a) select a file/folder, (b) "Info" button, (c) "Go to permissions" button
    • "File Sharing" / share.pass.psu.edu feature decommissioned; replaced by Protected Personal
  • www.personal.psu.edu's web counter reset utility was disabled on June 27.  Later this summer:
    • The counter will be upgraded to support load balancing.
    • www.personal.psu.edu will gain a second load balanced machine for additional performance and reliability.
    • The counter reset utility will return with secure, WebAccess authentication.
  • test.scripts.psu.edu is now load balanced to 2 machines.

  • Old WebMail data left in PASS after the 2006 migration of WebMail to GPFS was cleaned up.

  • https://explorer.its.psu.edu/, the old domain for the ACL Explorer, ACL Reset, Quota Manager, Restrict Access to COLA tool, etc has been decommissioned. 
  • Services now load balanced:
    • www.psu.edu (May 15, 2007)
    • www.work.psu.edu (Oct 1, 2007)
    • www.courses.psu.edu (Oct 1, 2007)
    • www.clubs.psu.edu (May 29, 2008)
    • www.dept.psu.edu (May 29, 2008)
    • test.scripts.psu.edu (June 25, 2008)
    • php.scripts.psu.edu (June 24, 2004)
    • cifs.pass.psu.edu (a.k.a. smb.pass.psu.edu, win.pass.psu.edu)
    • nfs.pass.psu.edu (July 4, 2008)
    • sftp.pass.psu.edu (a.k.a. ftp.pass.psu.edu, sftp.personal.psu.edu, ftp.personal.psu.edu... - July 4, 2008)
  • LDAP changes
    • PSDirIDN was replaced by the UID attribute as the DN (Distinguished Name) for user account entries on February 11, 2008 (see alert-596). It remains in user entries for convenience.
    • LDAP took over the responsibility of the master group registry from DCE on March 15, 2008 (see alert-628).  The ACCESS.PSU.EDU Active Directory domain was synchronized with all User Managed Groups (UMG) at this time.
  • ITS Alerts and ITS News systems upgraded to dedicated, load balanced servers to improve reliability during infrastructure outages.

  • Web server log rotation into PASS has been suspended after the July 3 morning rotation.
    • They are expected to resume during the second week of July.
  • New account and account service (e.g. Web space) provisioning have been suspended July 2.
    • They are expected to resume during the second week of July.
And I'm sure that's not all...I'll update this list as I find changes that I missed.

No TrackBacks

TrackBack URL: https://blogs.psu.edu/mt4/mt-tb.cgi/12143

3 Comments

Awesome Job! This was one hell of an overhaul to pull off!

Hats off to everyone involved. This is a triump of planning as much as it is anything else - nice work.

Great job you guys (I'm a little late in saying this). Do you have any updates

Leave a comment

Search This Blog

Full Text  Tag

Recent Entries

pacl update
This morning, the PASS Access Control List (pacl) program, which is the permissions program used by PASS Explorer, was altered…
Configuration Change to PASS CIFS Gateways Scheduled for Maintenance Window - 07/15/2010
2 configuration changes due for this Thursday: Address issue with Windows 7 and Windows Server 2008 R2 clients mapping…
Kerberos encryption types augmented
On Wednesday, January 6, 2010, we augmented the list of encryption types supported by the Access Account realm, dce.psu.edu, to…