We’ve all seen the surveys that talk about how miserable the industry is in general at doing software projects. Failure rates of up to 85% are commonly sited. Have you ever stopped to ask “What is the definition of failure?”. Scott Ambler did, and once you define failure (or success), you get a much different picture of the industry in general.
Think about it for a minute – how would YOU define success for your projects?
When Scott asked that question, here is the result based on 600 respondents:
In short, we appear to value delivering high-quality working software that meets the needs of our stakeholders in a cost effective manner while respecting the needs of the people involved with the project. This is more important to us than building something to specification on schedule and on budget, regardless of the toll it takes on the people involved.
This is a very different picture of success than the definition used by Standish Group’s Chaos Report … They define success as “on time, on budget, meeting the spec”, but that definition doesn’t seem to hold when we ask people what they actually value.
So specifically, how did the IT industry define project success? From Scott’s survey data:
- Schedule: 61.3 percent of respondents said that it is more important to deliver a system when it is ready to be shipped than to deliver it on time.
- Scope: 87.3 percent said that meeting the actual needs of stakeholders is more important than building the system to specification.
- Money: 79.6 percent said that providing the best return on investment (ROI) is more important than delivering a system under budget.
- Quality: 87.3 percent said that delivering high quality is more important than delivering on time and on budget.
- Staff: 75.8 percent said that having a healthy, both mentally and physically, workplace is more important than delivering on time and on budget.
So using actual success criteria, what is the success rate of IT projects? Is it really only 15%? Well, it is actually much better than that. Again from Scott’s survey results:
Respondents reported that Agile software development projects have a 71.5% success rate, traditional projects a 62.8% success rate, and offshored software development projects a 42.7% success rate.
Notice, these are SUCCESS rates, not failure rates. Quite a different picture of the industry in general.
I think it is important to look at the data yourself. Some of the really interesting information to me is the differences in success definition between different kinds of project participants. For example, the success criteria as stated by project managers is very different than that expressed by the other project participants. It is also interesting to see that the business stakeholders place a very low value on the actual working environment, while that is of great concern to the development staff . What this tells me is that different members of the project team have different goals, which are often in conflict with each other. And what I find in my work, is that those different goals are generally UNSTATED assumptions being made by the different team members. So everyone talks about having a successful project, but no one talks about what that actually means.
I think Scott’s article and survey data are important to open up discussions within your organization about what success means to the various members of IT. You cannot move a boat forward if everyone is rowing in different directions, and you cannot effectively steer a project team to success if there is not agreement on what success means.
===============================================
Now it is your turn:
I encourage you to look over the report and the survey data and to start discussions in your organization.
For more information, see Scott’s article in Dr. Dobb’s Journal Defining Success and the actual survey data at IT Project Success Rates Survey: August 2007
================================================
You are invited to re-publish articles from this blog, in your publication or website, as long as the article is intact and you include the following Byline paragraph (with live links) after each article you use…
START BYLINE
* Article used with permission from Wyyzzk, Inc.’s Resources for Business Analysts site at http://www.writingusecases.com This website of reports and tips contains information to help you succeed as a Business Analyst in IT.
END BYLINE
It’s worth keeping in mind that Scott’s sample was drawn from readers of Dr. Dobb’s Magazine in general and subscribers to his Agile newsletter in particular. That will introduce a bias to the survey, because the respondents are likely to be overwhelmingly people who have found Agile methods to either be interesting to them or successful–in other words, the audience already thinks that Agile techniques work, so it’s no surprise that the survey showed that.
Now, there’s no particular reason to think that Scott’s survey is any worse in this regard than other surveys. The Standish Group, for instance, overwhelmingly targets senior executives in its survey population–which is probably one of the reasons they consistently show lower results, because that’s a group with a different definition of success. It’s just something to keep in mind.
Absolutely. And another great example of why the players need to keep communicating their expectations regarding the success of the project.
Thanks for the response!
This survey is being discussed in other forums, where I always ask why it seems to be assumed that a spec does not define the actual needs of the users, and I get no answer. I do know that Mr. Ambler equates specs with the ‘horrors’ of BRUF – Big Requirements Up Front. Frankly, BRUF describes a situation that rarely happens anymore, as i can’t recall the last time I was given multiple months to do requirements, i.e. spec… and I am not ranting against Agile, if it works for you, do it. All I want is an answer to my question.
Hi David -
Thanks for the post!
I can only answer from my own experience, but here are some reasons why I think the spec may not meet the user needs:
1. In a large number of projects, the spec is written in English text. Many, many people find visualizing the finished product from a text description to be impossible. So while they think the spec describes what they want, when they see the product, they say it is not what they want. This is a large reason for approaches based on prototyping the user interface first, and I think also a large reason for the popularity of Agile processes, where you develop a little, demonstrate, make changes, develop more, demonstrate, and so on.
2. The spec may not be kept up-to-date as the project progresses. So as the team gets a better idea of what the stakeholders want, the software changes to match the actual needs, but the spec may not be updated. I know of a very large, very successful company that uses the spec to write test cases, then throws the requirements away, because their experience is that no one keeps the requirements specs up-to-date, but they do keep the test cases up-to-date.
Geri
Yes, pure textual specs and no change control will produce the situation where a spec does not match up to (often unstated or uncaptured) user expectations. Given that, a spec that includes graphics and is managed properly over the life of the project can very easily match to user expectations; the problem with the survey questions was it was assumed that a spec can never represent user needs.
So, if you ask someone if they prefer to get what they want, or some other thing that is stated as being not what they want, how would you answer the question?
Its about this time where one can use the quote “there’s lies, damned lies, and statistics’ or, as David Letterman says “Satistics’.
Whatever the respondent base maybe, the truth is that estimates should be close to deliverables. Other findings of the survey fall in line with what you generally expect from project teams.
Hi,
A member of the Orange County IIBA directed our BSA’s to this site, but it does not appear to be updated (most recent article from 2007?).
Is this site still current?
The blog is definitely active! This particular article was from 2007. You can find the most recent posts on the main page: http://www.writingusecases.com
Thanks for visiting!
Geri