Adjusting RPF Processes for Agile Development

This post was previously on the Pathfinder Software site. Pathfinder Software changed its name to Orthogonal in 2016. Read more.

Agile development, when done right, results in better software at a lower cost. More and more companies are coming to this conclusion, but most are struggling with how to adjust their RFP and Governance processes to adapt.

The typical RFP process for custom software development is looking for a fixed bid, thinking this will provide budget protection and guarantee ROI. History clearly shows this is not true. Issuers expect to come within 10-20% of the budget. Studies show that custom software costs can range from 50% under budget to 300% over budget. Why?

• Requirements are usually inadequate (though there is a very wide range of inadequacies.)
• The requirements produced generally have little to do with usability.
• Things change – requirements change, technology changes, the competition changes, etc.
• Too many features are built that aren’t used – studies show that 45% of features build under waterfall are never used.

Because all vendors know the above, they all take different approaches to responding to RFPs. Some try to provide a relatively accurate estimate by using historical patterns. This can be fairly accurate if they’re building an application they have done many times before using the same technologies. If you’re trying to use application development to create or extend a competitive advantage, you’re likely building something new that no one else has built, rather than customizing something off the shelf. In that case, all of your RFP responses will be WAGs. Rather than pretending this is not true, focus on how you’re going to differentiate vendors and how you and your chosen vendor will work together to develop good requirements, adapt to change and maximize your ROI.

If you’re trying to use application development to create or extend a competitive advantage, you’re likely building something new that no one else has built, rather than customizing something off the shelf. In that case, all of your RFP responses will be WAGs. Rather than pretending this is not true, focus on how you’re going to differentiate vendors and how you and your chosen vendor will work together to develop good requirements, adapt to change and maximize your ROI.

Think about your past choices based on the RFP process and whether you would have selected the same vendor (or any vendor) if you had known what the true cost was going to be. Was the highest vendor the one that was most knowledgeable? Was the lowest vendor the one with the least understanding of the complexity? Or was he intentionally providing an unrealistically low estimate, thinking he would make it up with change orders?

If total estimated cost isn’t a reliable measure, what can be?

Reliability is of course relative. But here’s one approach. Look at team composition and iteration costs.

Team Composition: To develop efficiently you need a core team with the right skill sets that stay together on the project and can jell, thus increasing development velocity. Does the team include project management, business analysis, information architecture, visual design, technical architecture, development and infrastructure resources? All are needed to create commercial quality, high performance, scalable, secure application the meet users needs (generally you need at least a 3-4 person team to cover these skills). You also need an approach to fill in non-core but necessary specialized skill. Does the vendor have internal resources for these areas or a plan to cover these deficiencies? Rate the vendors based on skill breadth and depth.

Iteration Costs: Calculate the iteration costs (generally 2 weeks) for the core team with an allowance for specialized resources. Compare the cost of the first 5-6 iterations for each vendor rather than the total estimated cost they provide (there’s nothing wrong for asking for expected total cost, just don’t assume it will be accurate).

Experience and Skill Match: Rate the vendors based on experience and skill match as it relates to those needed for your project. Don’t expect, or care, whether they have done something exactly like it in the past if you expect to create or retain a competitive advantage.

Compatibility: Agile development needs to be a collaborative process. Visualize how you would work with the vendor. Do they really follow an Agile process (daily scrums, test driven development, continuous integration, weekly demos, pair programming, etc)? Do they provide transparency to the development process? Do they require you to participate actively in the development process on a weekly and daily basis? The biggest mistake companies make when they use the RFP process is not getting to know the potential vendors. The responses get put is some evaluation spreadsheet, and final selection is often made by committees that have not participated in the requirements gathering or RFP process and often have no development experience. That works for some types of projects, but application development for competitive advantage is a creative, collaborative process. This is much harder to evaluate. The easy way out is to rely on quantitative metrics – but that often doesn’t translate to successful projects. Some companies use quantitative measures to get down to the 2-3 finalists and then use other criteria, including some of the above, to make their selection. This is not perfect but seems to work better.

References: Find out if they are really using a collaborative process. Ask their customers how they are to work with, whether they are open a fair, and how the internal stakeholders and real users like the end result.

I’d love your feedback and suggestions.