July 2008 Archives

Fixed issue with PHP and SQLite version 2 databases

| 0 Comments | 0 TrackBacks
Today we fixed another problem.  While the deprecated SQLite extension to PHP was expected to not be available in the new php.scripts.psu.edu servers after the PASS Migration completed on July 4, we were not expecting the SQLite version 2 driver to be missing from the PHP Data Objects (PDO) interface, which you will need to have if you haven't yet upgraded your databases to version 3 (which is an incompatible format to version 2).  Evidently, these two components are linked, you build both when you signify --with-sqlite in your build options (Redhat didn't include a binary RPM for SQLite 2, so we had to rebuild from source). 

So after a little extra effort, we now have the SQLite version 2 driver back in the php.scripts.psu.edu servers, with both interfaces available: PDO and the old SQLite extension.

Since SQLite version 2 is deprecated, we are setting the expectation that these two access methods will be removed by next summer, July 2009.  If you use them, you now have 12 months to upgrade to SQLite version 3.

More info on the changes to php.scripts.psu.edu related to the PASS Migration is available on: Phase III, Version 1 errata page.

The driver was replaced by 1:40 p.m. on July 21.

Herd of Cat Testers

| 0 Comments | 0 TrackBacks
I'm not that big into things "cute", but I am into humor.  Let's just say that the coordination of our test efforts have often times been like "herding cats".  This was just too amusing to pass up.

Fixed issue with PASS Samba service and MS Office

| 0 Comments | 0 TrackBacks
Today, we fixed a problem that emerged as a result of granting MS Office programs like Word, along with everything else that accesses the new Samba PASS gateway, access to the raw Access Control List (ACL) file permissions.

The problem was rather complicated, so I'll do my best to convey the essential bits.

Every time Word changes a document, it deletes the old file and replaces it with a new one.  When a user saves changes to a Word document that wasn't owned by him/her, this has the side effect of the ownership changing to the user making the edit.  In this case, Word will copy the ACL from the old file to the new with some modifications.  The modifications include:

  1. The owner of the previous file is added to the ACL of the new file as an explicit user entry if that owner wasn't already there.  The permissions the special:owner@ entry used to have are copied to here.
  2. If the new owner is already in the ACL with an explicit entry, it is converted to a special:owner@ type entry.
  3. If the new owner wasn't already in the ACL, then no special:owner@ entry is added.
  4. Other things...
In the case of #3, which is easy to occur when two or more people are taking turns updating a file, the UNIX mode permissions will read as if the owner of the file does not have read or write permission to the file. 

Samba had the default setting of "map read only" set to "yes", which means that in this situation, when the owner of the file in the UNIX mode does not appear to have write permission, Samba will set the "Read-Only" attribute on the file, a setting that is separate and predates the use of ACL permissions. 

This setting causes Word to open the file in a Read-Only mode, and it refuses to save to the original file name.  This occurs for all users, including the latest user to update it.

I found that by some trickery of the ACL settings could avoid this situation, but switching the "map read only" setting to an explicit "no" causes Samba to always leave the Read Only attribute not set.  This was updated on all 3 production Samba PASS Gateway servers (and a few others that may be in production later) by 11:15 a.m.

Big Summer Changes - Not PASS

| 0 Comments | 0 TrackBacks
No doubt the many changes that occurred as part or a result of the PASS Migration (see my last blog post) will cause changes in many people's computing experience at Penn State.  However, I'd like to take a moment to note the recent changes outside of PASS that may also have a substantial impact:

  1. Dan Kaminsky Discovers Fundamental issue in DNS / Massive Multivendor Patch Released - most Penn State DNS servers have already or will soon be patched.  I've heard that there are some side effects to this including slowness.
  2. Microsoft patch Tuesday changes (including above DNS patch); ACCESS.PSU.EDU domain controllers patched yesterday
There are other changes our group is planning to do with regard to the DCE/DFS phase out after the July 3-7 PASS Migration weekend.

  1. Penn State WebAccess - switch to use MIT Kerberos KDCs (dce.psu.edu realm) instead of DCE security servers (dce.psu.edu realm); date TBD (expect it shortly after the kerberos ticket expirations complete); this should only be a concern for those who run sites that need kerberos tickets, most cosign enabled sites do not.
  2. (Added 2008/07/11) - Switch MIT Kerberos KDCs to use the same, longer ticket expirations that DCE had.  The default maximum times one could request were 10 hour lifetimes with 24 hour renewal periods from a single kinit. As of this morning, they are now system wide maximums of 14 days with 28 day renewals, but they won't take effect until every principal's settings are adjusted.  Expect this to take place over the next few weeks.  Some client systems may still by default request a shorter expiration time such as 24 hours.  In these cases, the -l and -r switches to kinit would help extend them up to their max if not a local client config change.  (Update 2008/08/06) All principals were adjusted by July 28.

PASS Migration Complete

| 3 Comments | 0 TrackBacks
We did it!

Yes, we managed to successfully copy all of the data from old PASS (DFS) into new PASS (GPFS) in less than 68 hours.  If you include the time it took us to bring servers back online to the "mostly up" state, then we're talking 23 hours to resolve the alert.

Why "mostly up"? 

Most of the work wasn't simply the copying of data. It was also the porting of programs to the new environment.  Own our tools for PASS and PASS related services took several years to build and refine to the state they were in prior to the migration, and we had a small fraction of that to adapt to the new environment.  Some parts like the Club Server Administration Console were closely fitted against the particulars of DFS and had to be recoded and retested substantially.  This change also affected many other groups, such as the various programming groups of Penn State World Campus, where we needed to be involved in helping the conversion.  While we are now officially, fully running from new PASS, there remains much to be done to get back to where we were...and beyond that to where we need to be.  But now that old PASS is behind us, we can finally move forward with new development.

Programming wasn't all the work either.  There was lots of testing, documentation changes, retesting, fixing problems, retesting, fixing new problems from old fixes.  The client software landscape was also changing right under our feet over the past 6 months since the PASS Beta was launched which provided additional challenges with our Samba and NFS gateways.  Some of this is still ongoing.  A lot of people were involved, and that took a considerable amount of coordination.

7.2 Terabytes in 23 hours?

Actually this was a lot less, both in time and data size.  Using both an incremental data rsync as well as an incremental ACL permissions copy (well "translation" may be a better term), we were able to cut down what was originally worst case 74 days to approximately 7 hours of a final pass during the 3 day weekend window.

So what changed?

Much has changed during this transition, not just the storage technology piece (GPFS replacing DFS) and the corresponding features (e.g. permissions and quotas), but connection methods (kerberized gateways), application engines (PHP switch from Solaris to Linux, apache 1 to apache 2) and the list goes on.

What I'd like to offer here is a list of the various sundry details for the more technically curious.

  • DFS replaced by GPFS
    • Multiple server access to shared disk reduces rate of failures that have plagued us in the past (most alerts on PASS were related to hardware failures that would have not been a problem for GPFS); recovery is now counted in seconds, not minutes to hours
    • Native client access to disk performance improved 10-fold; this improvement can be seen in various PASS based services
    • DCE infrastructure replaced by MIT brand Kerberos and IBM's Directory Server (LDAP); DCE to be shut down before fall semester
    • Quotas no longer fileset based, now numeric Group ID (GID) based
      • mandatory GID inheritance (setgid on folders not required)
      • quotas are now "hard"; enforcement is on write, no longer on file open
    • Access Control List (ACL) permissions are now based on the new NFSv4 standard
      • IETF standard (RFC 3530), no longer rescinded POSIX draft
      • support available from both the Samba and NFS gateways; client tools come bundled with the platform
      • 14 instead of 6 permission types
      • "deny" as well as "allow" entry types
      • arbitrary rule order replaces POSIX strict evaluation order
      • ACL inheritance set by inherit flags on each entry; no longer separate ACLs for inheritance
      • chmod has no effect on ACL; setuid, setgid and sticky still apply
      • stat() mode now approximate
      • Penn State custom "simple" ACL tools help ease usability
        • stand-alone ACL Explorer Web-based tool replaced by PASS Explorer integrated tool, with wizard workflow based on the Protected Personal Access Control Manager (ACM)
  • Gateway Authentication
    • Samba (SMB/CIFS) Gateway now accepts kerberos tickets from the client
    • Samba (SMB/CIFS) Gateway no longer accepts NTLM based authentication
    • NFS Gateway now accepts kerberos tickets from the client
    • NFS Gateway authentication mapping Web application no longer supported; sys=auth is no longer sufficient for access
  • Gateway exports
    • The "dfs" Samba share, e.g. \\win.pass.psu.edu\dfs, has been decommissioned.  Use the "pass" share instead.
    • The NFS exports nfs.pass.psu.edu:/.../dce.psu.edu/fs, nfs.pass.psu.edu:/.:/fs and nfs.pass.psu.edu:/: have been decommissioned.  Use nfs.pass.psu.edu:/pass instead.
  • Gateway server changes
    • Samba upgraded from 2 to 3
    • NFS protocol version 4 now supported; legacy version 3 still supported
    • NFS service now has load balanced backend (3 systems instead of 1 to ensure higher availability)
  • Path changes:
    • PASS is no longer mounted at /.../dce.psu.edu/fs.  It is now mounted at /pass.
    • For a limited time, the old, deprecated links /.../dce.psu.edu/fs, /.:/fs, /: will now point to /pass.
    • /:/cactus is now /pass/services/cactus
    • /:/rs_aix, /:/solaris, /:/linux, /:/dist are now, respectively, /pass/os/rs_aix, /pass/os/solaris, /pass/os/linux, /pass/os/dist
  • SSH Host key changes
    • (for current key for both sftp and rs6klab, see http://kb.its.psu.edu/psu-all/hd/passssh/)
    • sftp.pass.psu.edu, a.k.a.
      • ftp.pass.psu.edu
      • sftp.personal.psu.edu
      • ftp.personal.psu.edu
      • sftp.clubs.psu.edu
      • ftp.clubs.psu.edu
      • lutz.cac.psu.edu
    • rs6klab.aset.psu.edu, a.k.a.
      • ptp.cac.psu.edu
      • splogin.cac.psu.edu
      • krumsville.aset.psu.edu
      • slickville.aset.psu.edu
      • slickville.cac.psu.edu
      • dimsville.aset.psu.edu
      • dimsville.cac.psu.edu
      • frackville.aset.psu.edu
  • UNIX changes
    • Solaris systems (cbs.aset.psu.edu, armstrong.aset.psu.edu) unavailable for the time being due to resource constraints; expected replacements/upgrades before fall semester 2008
    • dcecp, acl_edit, dcerchacl, fts and similar DCE/DFS commands no longer available
    • account information now integrated with LDAP
    • Secure FTP service now provided by rs6klab
    • command line ACL permission tools provided by the OS vendor (aclget, aclput, acledit on AIX, etc.)
      • recursive, Penn State custom permission "simple" command line tool due out before fall
    • quota tools provided by the OS vender
    • "Native" client no longer available for use; NFS client now provides similar functionality to that of DFS native clients
  • PHP changes (php.scripts.psu.edu)
    • Platform change from Solaris to Linux
    • Vendor built RPM packages for apache, php, etc replace custom built binaries
    • Apache upgraded from 1.3 to 2.2
    • PHP still 5.1.6
    • SQLite extention discontinued; SQLite databases may continue to be accessed via PHP Data Objects (PDO)
    • register_long_arrays now disabled; deprecated $HTTP_GET_VARS and $HTTP_POST_VARS, etc. no longer available; functionality still provided by $_GET and $_POST arrays
  • WebAccess conversions
  • User Managed Group Updates on June 30
    • Paperless UMG creation
    • Improved interface
    • Single owner group may now own multiple UMGs
  • PASS Explorer changes
    • New file permissions tool available via (a) select a file/folder, (b) "Info" button, (c) "Go to permissions" button
    • "File Sharing" / share.pass.psu.edu feature decommissioned; replaced by Protected Personal
  • www.personal.psu.edu's web counter reset utility was disabled on June 27.  Later this summer:
    • The counter will be upgraded to support load balancing.
    • www.personal.psu.edu will gain a second load balanced machine for additional performance and reliability.
    • The counter reset utility will return with secure, WebAccess authentication.
  • test.scripts.psu.edu is now load balanced to 2 machines.

  • Old WebMail data left in PASS after the 2006 migration of WebMail to GPFS was cleaned up.

  • https://explorer.its.psu.edu/, the old domain for the ACL Explorer, ACL Reset, Quota Manager, Restrict Access to COLA tool, etc has been decommissioned. 
  • Services now load balanced:
    • www.psu.edu (May 15, 2007)
    • www.work.psu.edu (Oct 1, 2007)
    • www.courses.psu.edu (Oct 1, 2007)
    • www.clubs.psu.edu (May 29, 2008)
    • www.dept.psu.edu (May 29, 2008)
    • test.scripts.psu.edu (June 25, 2008)
    • php.scripts.psu.edu (June 24, 2004)
    • cifs.pass.psu.edu (a.k.a. smb.pass.psu.edu, win.pass.psu.edu)
    • nfs.pass.psu.edu (July 4, 2008)
    • sftp.pass.psu.edu (a.k.a. ftp.pass.psu.edu, sftp.personal.psu.edu, ftp.personal.psu.edu... - July 4, 2008)
  • LDAP changes
    • PSDirIDN was replaced by the UID attribute as the DN (Distinguished Name) for user account entries on February 11, 2008 (see alert-596). It remains in user entries for convenience.
    • LDAP took over the responsibility of the master group registry from DCE on March 15, 2008 (see alert-628).  The ACCESS.PSU.EDU Active Directory domain was synchronized with all User Managed Groups (UMG) at this time.
  • ITS Alerts and ITS News systems upgraded to dedicated, load balanced servers to improve reliability during infrastructure outages.

  • Web server log rotation into PASS has been suspended after the July 3 morning rotation.
    • They are expected to resume during the second week of July.
  • New account and account service (e.g. Web space) provisioning have been suspended July 2.
    • They are expected to resume during the second week of July.
And I'm sure that's not all...I'll update this list as I find changes that I missed.

Search This Blog

Full Text  Tag