Recently in Software Engineering Category

As of late on the IAM effort, we have been involved in an evaluation of application servers.  Actually this is our second round of doing so.  Last year, we did an evaluation of Mule, ServiceMix and Geronimo.  At the time Geronimo, being fully Java 5 EE compliant, turned out to be a good choice for development.  We liked it because it had a great plug-in for Eclipse and it met all of our needs at the time.  As part of our project's functionality we needed to use Java Messaging Services (JMS).  The problem we ran into with Geronimo was it did not support the STOMP protocol.  So we re-grouped and switched to IBM's WebSphere Community Edition.  The message problem was solved, we thought.  Then we tired to use SSL for messaging and found the support in WebSphere CE to be lacking.

So, we commenced round #2, which included: Tomcat, WebSphere, JBoss and WebLogic.  Since all of our Web Services were SOAP-based we needed a really good servlet container, which Tomcat is.  However we needed other features like the messaging, which Tomcat does not provide, but the other products did.  We ended up doing a lot of timing tests and evaluations and our winner is?  WebLogic.  It is a very fast Application Server, and it met all of our requirements.  Since our development is going to be using Oracle, the tight integration between the database and the application server also made it an attractive choice.  And the runner up was Tomcat BTW.  Probably the best servlet container out there.  Why did we not select it?  Mainly because of having to add other tools either to it or standalone to meet the requirements that we have come up with.  Timing results are below.

Types of Code Story (Waterfall):

Back in the day, when I was at Raytheon I was involved with writing a number of proposals for a variety of things.  Sometimes the work was interesting other times it was real boring.  One part that was interesting were the line of code estimates.  For a paragraph of text that describe a particular function, you would have to come up with a line of code estimate for it along with the percentage breakdown based on type.  Then the financial folks would take that number and determine a cost.  Sometimes the estimate would give us a win, others times not so much as my friend Mark Miller would say.

The types of code that we would use had three categories: hard, medium and easy.  The productivity of each one was 10 LOC/day, 13 LOC/day and 15 LOC/day respectively.  So you are probably wondering what the heck?  A person is only going to develop 10 lines of code in a single day if the code was deemed hard.  Well, in a word yes.  Those numbers included more than just writing 10 lines of code.  It included all of the up-front design, reviews, coding and testing.  So in the end you were getting fully tested code.  Back then we were a waterfall shop and now from what I hear things have more on into other methodologies.  Here are some examples of each of the types of code:

• Hard - device driver development.  Yes, back then we did that since we built and engineered our own hardware.
• Medium - interface development.  Writing X11 programs at the time was not very easy.
• Easy - simple utilities.

WebMail (Agile):
Now in present day, we are doing some great strides with Agile techniques for WebMail.  Our development cycles are typically 2-3 weeks.  Within that time, the planned features are discussed, code is developed and full regression testing is performed.  I am very pleased with the product that our team has put together.  On May 12th, we will be doing another launch of WebMail.  This one which has been in beta since Spring Break, will enable us to turn off WebMail2.  The new version of WebMail will have all of the existing functionality of WebMail2 and loads more.

My final story (SCRUM):
And finally my story, one of the tenants of the Agile project management process known as SCRUM has to do a lot with how the team functions and communicates.  We (Applied Information Technologies, Emerging Technologies and others) participated in a SCRUM effort a while ago and probably none of us knew it.  The task was to take Shibboleth, and use that as the basis of authenticating with Napster.  Shibboleth enabled our Penn State students to authenticate at Web site and have only that they were a student passed onto Napster.  Napster never knew who the student was, just that they were a student. There are many parts of shib and I really do not want to go into them this morning.  Back to the SCRUM portion of the post, we (AIT) never worked with Shibboleth, heck for the longest time, I could not even spell it.  ET had a pilot already going, but now we had to ramp up for large scale production and we had less than 30 days to do it.  There were so many unknowns with performance, like how many machines do you need?  At the time, Kevin Morooney was the Sr. Director for ASET, he of course realized how important the project was, like all of us did.  However he did something novel which is part of SCRUM.  We would meet for 10-20 minutes every morning go over the day's objectives and previous day's results.  Then at the end of the day we would meet again to see how we did and plan the next day's objectives.  Let me tell you, this method worked out great.  How often do you find yourself in a situation where you have a major effort going on and you are only meeting weekly, bi-monthly or even worse monthly?  If the effort is critical, you need to take a page out of SCRUM's play book and do the daily meetings.  It worked out great for us.  After all of the planning and testing we did, when it came time for that first student to log in, we were watching the machines and things worked.

So there you have it, the past, present and the future all wrapped up in three little stories.   

People use that expression a lot when they talk about a time when things were just perfect for them.  There are times when in my IT career, I miss the good old days.  A time very long ago, when my title was Senior Engineer.  Engineer, of what you may ask.  Well Software, of course.  I was at Raytheon and I was engineering new software following tried and true practices and procedures.  Ah, the good old days, such a great time.  I was a Lead Software Engineer (LSWE as we called them), on a number of projects.  That was a good thing too, I had resources available to me, but I did not have to worry about their performance reviews or personal issues.  Unlike what I do here as an IT Manager.  Even though my real title says, Manager, Research Programmer.  You really cannot escape that Manager part, trust me I have tried.

Wait a minute, people are probably thinking being a manager is better, you have people and you can use them to get things done.  Well, folks hate to burst your bubble but it doesn't work that way.  As a manager, you have to deal with the human side of things.  All of the talks, and training courses really cannot prepare you for a lot of these things.  I remember when I became a manager, I was dealing with so many issues my head was spinning.  Then I took courses like Mastering Supervision which only made situations worse because I tried to use some of my new found knowledge.  Some things worked, others failed miserably.  Things that I learned from the IT Leader's Program (ITLP) were actually more useful.  Having a coach like Brian McDonald (MOR Associates) really was another big win for me.  So know you are probably concluding that being a manager now is great.  Well not really...

I truly do miss the good old days.  I am finding that most of the software that I write these days is of the maintenance variety.  Trust me folks that is not really Software Engineering.  About the only new software I have written lately was helping my kid write some Java for her class.  The reality is, I'm not the only one feeling this way.  I have friends outside of Penn State that are still doing maintenance on software they wrote 20 years ago.  I did that maintenance type of work at Raytheon for about a year and then resigned.  I'm not saying that type of work is not important.  Maintenance work keeps machines moving and airplanes flying.  So what I am saying?

Well, I personally missing the days of being a technical lead and having the ability to write new exciting software.  Do I see opportunities for something like that in the future in my current job?  I am not sure, that's the truth.  I do enjoy having the ability to blog about issues like this and I'm guessing there are other "IT Managers" out there that feel the same way.  Will I leave because of this? (dead silence)

I know a good friend of mine, told me I was nuts for going doing this route of IT Manager.  At that time, I really did not believe him, because the glory and the prestige of the title of manager was in front of me.  Now if he would make that comment again, I would probably agree with him.  Oh to yearn for the good 'ole days.

So you have this great application that you have written and you want to make a tiny change to a source file.  What do you do?  What happens if you change does not work?  Valid questions all of them.  When I was at Raytheon, I used to watch seasoned developers do a practice that was not very good.  Anytime they wanted to make a change, they would copy the current file to a new file and append the date onto the end of it.  After a long time they would have a large number of files in their directories with all of these dates on them.  Problem is they had no idea what they did or what bugs they had in them.  What a mess!  Sad to say, I still see things like this going on.

So what do you do?  Well use Revision Control of course, hey that's the topic of today's posting.  Revision control is a software application that allows you to maintain multiple versions of a source code file (at its most basic level).  There are a number of tools out there to accomplish this task and which one you select will depend on the platforms you are using and the size of your project.  Some of the major players include SCCS (Source Code Control System), RCS (Revision Control System), CVS (Concurrent Version System), and SVN (Subversion).  All of them have their good points and weaknesses.

Stay away from SCCS, it is extremely cryptic in its command use.  We used it for a couple of projects back at Raytheon and it was painful to use.  RCS is much nicer, and the command set is pretty brain dead simple.  You want to check in a file you use ci, to check it out use co.  The downside with RCS is your are dealing with single files, not entire projects.  So if you wanted to check a project tree in for example (under UNIX), you would have to do something like:

for i in `ls *.c`

do

    ci -u $i

done

Ugh right?  But hey at least your are using something.  Subversion (SVN) is a much better solution for teams, since it supports the concept of checking in/checking out entire project trees.  Again, you can find more information about it by doing a simple google search.  Right now we are using SVN for most of our projects and RCS for our configuration files (like httpd.conf for example).  CVS has been around for years like SCCS.  It has the same type of support for project trees.  Again, do a google to find more information about it.

The key thing to remember with regards to revision control is to actually do it.  Do not get into the habit of creating copies of files and appending dates on the end.  Oh yeah, another point to remember is all of these tools support the concept of a log message.  This is some descriptive text about what this new version of code represents, new features, changes and so on.  I have seen people embrace RCS/SVN into their environments and when it comes time to do the log message, they enter a single word, UPDATE.  Now, that's not really helpful is it?  How is the next developer who has to look at that code going to figure it out?  Well, probably spending a lot of time with a diff program.

Key Points:

1. Revision control is extremely important to a solid software development process.
2. Select a tool that will work with your environment.
3. Popular systems include: SCCS, RCS, CVS, and SVN
4. Be descriptive when you enter your log messages.

So what is software? That is the topic of this blog posting.  Now many of my loyal readers are probably saying, "Jimmy, we already know what software is" or "Jimmy, why didn't you make this your first posting?".  Well they are all good questions indeed.  So the time has come.  A basic question as what is software, needs to be answered by someone who has been doing software for over 20 years now.  The general trend that I hear in my daily exchanges with people is that software is just code.  "Its all about the code".  I hear that one a lot.  Well, that's part of it.  Code those magical little bits of ones and zeros really make the hardware of a computer function.  One of my nicknames that I received back in the Raytheon days was CodeMan.  I was the CodeMan, because I could pump out large quantities of well tested code in a short period of time.  Props goes out to my friend, Dr. James Wise for giving me that name.  Back in the day, coding was my life.  Notice the emphasis on the word was in my previous statement, today software engineering is my life.

So code, code, code, its software, right?  Well no, it isn't.  Software is more than just those magic bits of ones and zero.  Software (my definition) is... The complete package to include supporting documentation, design, testing, maintenance and so on.  It is more than just code.  People need to stop being in the mindset that I'm going to write some code and call it a software-based service.  Wrong!  It is more than the code, its all of the things to support the software from the life-cycle perspective (birth to maintenance).  Now my loyal readers are wondering if am talking about process?  Hum...  YES I AM!  Software is a process, it is not coding.  Now do not get me wrong, prototyping something could be a very important part of the process.  I do it myself.  When I was writing some search engine code for WebMail.  I knew I needed a data store for the inverted index, so I tried a number of things until I settled on SQLite.  NOTE: the code that I wrote during the prototype was throw-away code.  You need to do that.  That code was not pretty and not very testable.  And when I am talking process, I am not saying you need something rigid as what I used at Raytheon (MIL-STD-2145).  We need a happy middle ground where requirements are gathered, design is done, code is written, and tested.  At that point, we are well on our way to developing software.  However we need to make sure we cover the other parts, the support, the maintenance and the documentation.  All of these parts are key to developing software.  Read my previous postings about teams, you need a good one of them too.

And my final suggestion as it relates to process is something that I see as problem around here where I work.  We as an organization put out services.  Services help our users do things like Email and so on.  However how they are delivered is not very consistent.  What is needed is a checklist of sorts, that has all of the items on it necessary for a service to go from idea all of the way to production.  Gone should be the days of just putting stuff up because we can.  Just think how more efficient things will be if we all do them the same way. 

Other things that we should do as an organization is set up code repositories.  How many times do we need to have a developer write code to connect to a database or do a query?  With a software repository, they can drawn on the experience of others by using code that already has been developed and...  TESTED!  The code in the repository must have its own test plans and procedures.  Reuse is something that is so important in a mature software organization.  CMU used to have the Capability Maturity Model that ranked an organization's process from chaotic to refined.  Back at Raytheon after time we reached the refined level.  Now again, I'm not saying we need to go out and do everything related to that.  A software repository can only help us in the long run develop software in a much more quicker and efficient manner.

I consider this post to be my manifesto as it relates to software and process.  I hope you have enjoyed reading it as much as I did writing it.  There will be no weekly links tomorrow since I will be out of town.  Have a great day! 

Still striving to find the solutions to our software woes, I continue to spend my mornings and evenings (like tonight) reading the trade rags, and Internet to see what's hot.  Some of things that I'm reading about include:

• Practices vs. Processes (this one is very hot!)
  
• Model-driven development
• Test-driven development
• ESSWork (Ivar Jacobson's work)
  
• XP (not Windows XP)
• and many others

The world is changing rapidly and what I am noticing is that Waterfall has its place, but other methodologies should be considered mainly because of quick schedules and current staffing. 

My Little Story...

Along those lines, I'm still having another group within ITS gather requirements for a follow-on to WebRAT.  From what I am seeing from EMails, I see that people are following into the trap of taking the requirements one step further into design.  For example one of the requirements for the new system, is if a person is going to be in all roles for the system, then they should be in one master role.  At that point the people gathering the requirements should stop, that statement is the requirement.  They should not engage the user about how that role is named.  That is getting into design not requirements gathering.  The user should not care what that name is, only that if they are in all roles for the system, then that will be represented by a single role.  With experience will come this knowledge.  Requirements of course will change throughout the project, that is just the nature of the beast.  Its those initial requirements that form the foundation for the development of the system/service.

Well like always, keep posted for future developments. 

Its interesting trend that I am seeing these days where programmers (who think they are engineers), write something, proclaim it works and then make statements like its "time to write the documentation".  Quotes like that really scare me.  Why you ask?  Well, when the code is completely written that point in the project is not the time to starting writing documentation.  The documentation (specs, test plans and so one), should have been written well before one line of code is written.  Too often, I see situations like this where people try to force-fit the documentation based on what the program does.  Studies by many of the real pundits in Software Engineering show that this methodology will fail for a variety of reasons.  The biggest of which is not addressing the customer's requirements.  Remember the customer is the one who is paying for the software, they expect results.  Keep that in mind when you are in your office just writing code...

The specification or the spec how its called these days, is the document that details your design. The spec is directly resultant from your requirements gathering. Typically the spec contains many sections of content and in most cases your customer will dictate what they are. At a most basic level, the spec should contain:

1) An Introduction (purpose, terms and abbreviations, ...)
*) Why are you writing this document?
*) Every project these days has jargon, like AJAX for example. The customer may or may not what that means, so a description in this section is helpful.

2) Requirements by type (general, database, security, ...)
*) Remember that a requirement is a shall statement. For example, our project shall process user data from forms.

3) Projected hardware and software requirements (test and production)
*) Only a few lucky ones will be able to use the same environments for both test and production. In this section you detail the differences. Of course, maybe there are not any, which is great.

4) Database and File Layout
*) Now days there are not many projects out there that do not store user data of some sort, whether its in a database or a plain text file. This section will go into some what detail what tables look like.

5) Overall Design
*) This section will describe how you take the requirements and build something with them.

6) Concept of Operations
*) This section will detail how the final project will operate. This section should be extremely narrative

7) Other sections
*) If you are not doing a separate test plan, you should address your testing methodology somewhere in this document.
*) Same thing applies for the integration plan.

So when you are done with the specification, the goal to have something that expresses your response to your customer's needs. Typically, you will have to "sign off" on the document before you can start your detailed design. Sign off means that you and your customer read and sign a statement that is attached to the document. That statement can say pretty much anything that is related to the project. But in most cases, it will say something like we agree that the said specification addresses the requirements. When its comes down to it, the spec is your recipe for doing something for a customer.

Ah, back to the essays on Software Design and Development, what a way to start a Friday morning! Today I am going to chat about the most important part of the software life-cycle, requirements gathering. When it comes down to it, requirements are the items that tell you as a designer, what the software product should do. How do you obtain these requirements? It depends on the project. In some cases, the customer will supply you with the requirements as they know exactly what they want. When I was working for Raytheon, in most cases the government always supplied the high-level requirements. In other cases, you as an engineer need to meet with the customer and hopefully obtain the requirements as a result of the meeting. How you do this, depends on many factors, too numerous to mention here. What has worked for me over the years is something like this. You have somewhat of an idea of what the customer wants as an end-product. Knowing that, you can come with a presentation of possible solution(s). You want to make this presentation brief and then ask the customer questions (this is the gathering part). Your questions should cover the gambit, things like technology, form-factor, operating system, security and so on. After a series of meetings like this, you should be able to come up with your initial list of requirements. Keep in mind once you have the requirements, you are going to turn them into a specification, which the customer must agree to (sign off on).

The life-cycle, now that's an interesting term isn't it? Dictionary.com Defines life-cycle as the phases a software product goes through between when it is conceived and when it is no longer available for use. That is a pretty interesting definition and its pretty accurate too. Basically the software development life-cycle takes an idea/concept/problem from the beginning all of the way through the development of a usable product that has been tested. There are loads of different software design methodologies out there these days, waterfall, spiral, RAD, and so on. But all of them have a common goal, that is to develop a usable software product at its completion. In the following paragraphs, I will explore some of the "stages" of a software life-cycle. Keep in mind along the way during each stage, reviews are held to ensure that the product is on track with regards to the original design. That's not to say that the design cannot change, that happens all of the time due to things like customer changes, hardware availability and so on.

• Preliminary Design

During this stage, the engineers interview the customers to determine the requirements necessary to solve their needs. Requirements are typically expressed as shall statements, within the specification. For example, if a system is to use authentication, a requirement would be: The blah system shall use userid/password authentication.. Also during the stage, initial questions about the desired platform will be discussed. And also how the customer expects the product how to function. This is typically written up in a section called the Concept of Operations (CONOPS). The CONOPS describes in English how the system will function. For example, if the system is a data entry system it will cover things like how the data is entered and so on.

• Detailed Design

At this stage the design is refined further. This is where the data store requirements are further designed. Initial database table layouts will be designed typically using an Entity/Relationship tool. Also at this stage, psuedocode (English like) statements are "coded" for the routines that will be used to solve the customer's problem. And finally, one of the most important step is the development of the test plan. This document will be used in conjunction with the specification to help test the software product. Its very important that the test plan satisfies all of the requirements set forth in the specification.

• Implementation
  

In this stage, the design is implemented, meaning design is transformed into software using one or more programming languages. The programmer will use the detailed design to implement the code. Typically at this stage all that's looked for clean compiles of the code. But depending on the size of the team, software module testing using the test plan will also be done.

• Integration/Sell Off
  

Again, depending on the size of the team, integration may happen at the implementation stage or it may be separate. This is where the modules are placed together and tested according to the overall integration test plan. Also at this stage the customer will participate in what's called sell off. That's basically where they review the requirements and testing results to ensure what was developed is what they received. Of course at any phase, the customer can review things and request things to be re-designed.