Ask The Tough Questions

Most PPM software evaluation guides recommend that you ask questions that play to the vendors’ strengths: they encourage you to discriminate various features and capabilities that support good project management. But, PPM is about selecting the right projects, not just doing projects right. Here are some important, but often neglected, questions that can help you assess whether a vendor’s product will actually help you achieve the primary goal of PPM; namely, to identify and select the project portfolio that will deliver the greatest possible value to your organization.

1. Does the vendor provide a product that is specifically designed for your industry and the types of projects you conduct?
   The value of a project is the worth, to the organization, of the consequences that would occur if the project is conducted. Obviously, the intended consequences that motivate projects differ greatly depending on the industry and type of project (e.g., the consequences desired from a pharmaceutical project to create a new drug are very different than the consequences desired from a utility project to upgrade a transformer). If a vendor’s tool is not specifically designed for your industry, you can be sure that it will not evaluate projects based on their consequences, and, therefore, will be incapable of identifying projects that create the consequences that you most want. Many PPM tools evaluate projects based on some sort of point scoring system that the vendor has selected as a lowest-common-denominator approach applicable to the widest possible customer set. Such tools will not help you make value-maximizing project decisions.
2. Does the software quantify and optimize the value of the project portfolio, with project and portfolio value measured in dollars?
   The ultimate goal of PPM is to enable the organization to select and manage projects so as to derive maximum value. Many tools do not measure project value in dollars (because they lack the necessary methods and analytic rigor for translating “soft benefits” to equivalent dollar values). If non-financial project benefits are not expressed in dollar terms, how can a tool combine the financial benefits expected from projects (e.g., decreases in costs, increases in revenue) with non-financial benefits (e.g., improved corporate image, client service, or learning)? How can it determine whether the benefits to be derived from a project justify its costs if benefits are not measured in dollars? Be sure the system incorporates appropriate methodology for quantifying project value in dollar units.
3. Does the software contain a “configurable model” or a fully-flexible platform for constructing project valuation models?
   If the project value model is hard-wired in source code, the vendor will typically refer to it as a “configurable model.” Configurable models are characterized by a limited number of parameters that can be set to help “fit” the model to the needs of individual customers. For example, the weights that are assigned to represent the relative importance of various criteria are common model parameters that can be chosen by the user. Although parameters are easy to set, changing a configurable model beyond the limits of its parameters is difficult or impossible, as it requires reprogramming source code. A few PPM tools take a different approach. They contain an internal platform upon which virtually any project value model can be constructed. The platform is similar to Excel in that it allows equations or algorithms to be defined or modified without the need to change source code. If you choose a tool with an internal modeling platform you can be assured that the software will provide flexibility to refine your model as needed to incorporate new understanding or to address additional types of projects.
4. Is the software able to address and optimize real project choices?
   To some tools, project choices are all-or-nothing decisions: projects are ranked, and, if the project falls above the funding cutoff, it’s a “go,” if not, it is a “no go.” In the real world, decisions are more complex. Sometimes, the choice is among alternative versions of a project (e.g., a minimum cost, minimum scope version versus one or more expanded or enhanced project solutions). Sometimes it is not meaningful to evaluate a single year of spending in isolation of what happens in subsequent years (e.g., one year of asset maintenance in isolation of the level of maintenance planned for future years). Sometimes, there are interdependencies among projects: the costs and benefits of doing a project depend on the other projects that are in the project portfolio. Find out whether the software merely ranks projects or if it can truly optimize the project portfolio.
5. Does the software provide features that help ensure the quality of data inputs?
   Garbage in means garbage out. Experience shows that features that help promote accurate inputs and quality assurance are essential to obtaining accuracy in project recommendations. One useful (but not always easy to support) feature is the capability to provide users immediate feedback on the implications of project inputs in terms that are meaningful and easily understood. Tools that require the user to toggle between data entry and analysis mode in order to see how project inputs affect project evaluations can be cumbersome and more prone to data entry errors. Evaluating projects based on the consequences of doing versus not doing the project helps promote data quality, since users can rely on professional experience to assess whether consequence estimates for a specified project are reasonable. Likewise, a tool that values a project in terms of dollars provides an objective basis for assessing the results, “Would the organization really be willing to pay the indicated amount to obtain the project benefits?” Another useful feature is allowing users to analyze project concepts prior to “publishing” results for others to see. Check whether the system provides a place for colleagues to explore possibilities and share draft materials without committing inputs to the central database. A tool that allows project proponents to investigate the attractiveness of project proposals can help them to design better project alternatives. However, ask the vendor how the tool and application process counter “gaming” in the specification of inputs.
6. Are you communicating with someone with real expertise and experience?
   PPM is a hot topic, and consultants and vendors are reinventing themselves in their rush to offer tools for the job. Also, some large software vendors with established project management tools have created PPM tools by merely adding cross-project data rollup to their older tools. Such tools do not provide true assistance for optimizing project portfolios. The real opportunity for benefiting from PPM comes from enhanced ability to select the right projects, not from managing your existing projects more effectively. Make sure your candidate supplier provides evidence that they are truly leaders in the field of the “portfolio part” of project portfolio management.