More than 80 providers supply software tools for project portfolio management (PPM). Chances are, only a handful of the offerings will be suitable candidates for your application. How can you tell which vendors and tools deserve your attention?

Ask The Tough Questions

Most PPM software evaluation guides recommend that you ask questions that play to the vendors’ strengths: they encourage you to discriminate various features and capabilities that support good project management. But, PPM is about selecting the right projects, not just doing projects right. Here are some important, but often neglected, questions that can help you assess whether a vendor’s product will actually help you achieve the primary goal of PPM; namely, to identify and select the project portfolio that will deliver the greatest possible value to your organization.

1. Does the vendor provide a product that is specifically designed for your industry and the types of projects you conduct?
   The value of a project is the worth, to the organization, of the consequences that would occur if the project is conducted. Obviously, the intended consequences that motivate projects differ greatly depending on the industry and type of project (e.g., the consequences desired from a pharmaceutical project to create a new drug are very different than the consequences desired from a utility project to upgrade a transformer). If a vendor’s tool is not specifically designed for your industry, you can be sure that it will not evaluate projects based on their consequences, and, therefore, will be incapable of identifying projects that create the consequences that you most want. Many PPM tools evaluate projects based on some sort of point scoring system that the vendor has selected as a lowest-common-denominator approach applicable to the widest possible customer set. Such tools will not help you make value-maximizing project decisions.
2. Does the software quantify and optimize the value of the project portfolio, with project and portfolio value measured in dollars?
   The ultimate goal of PPM is to enable the organization to select and manage projects so as to derive maximum value. Many tools do not measure project value in dollars (because they lack the necessary methods and analytic rigor for translating “soft benefits” to equivalent dollar values). If non-financial project benefits are not expressed in dollar terms, how can a tool combine the financial benefits expected from projects (e.g., decreases in costs, increases in revenue) with non-financial benefits (e.g., improved corporate image, client service, or learning)? How can it determine whether the benefits to be derived from a project justify its costs if benefits are not measured in dollars? Be sure the system incorporates appropriate methodology for quantifying project value in dollar units.
3. Does the software contain a “configurable model” or a fully-flexible platform for constructing project valuation models?
   If the project value model is hard-wired in source code, the vendor will typically refer to it as a “configurable model.” Configurable models are characterized by a limited number of parameters that can be set to help “fit” the model to the needs of individual customers. For example, the weights that are assigned to represent the relative importance of various criteria are common model parameters that can be chosen by the user. Although parameters are easy to set, changing a configurable model beyond the limits of its parameters is difficult or impossible, as it requires reprogramming source code. A few PPM tools take a different approach. They contain an internal platform upon which virtually any project value model can be constructed. The platform is similar to Excel in that it allows equations or algorithms to be defined or modified without the need to change source code. If you choose a tool with an internal modeling platform you can be assured that the software will provide flexibility to refine your model as needed to incorporate new understanding or to address additional types of projects.
4. Is the software able to address and optimize real project choices?
   To some tools, project choices are all-or-nothing decisions: projects are ranked, and, if the project falls above the funding cutoff, it’s a “go,” if not, it is a “no go.” In the real world, decisions are more complex. Sometimes, the choice is among alternative versions of a project (e.g., a minimum cost, minimum scope version versus one or more expanded or enhanced project solutions). Sometimes it is not meaningful to evaluate a single year of spending in isolation of what happens in subsequent years (e.g., one year of asset maintenance in isolation of the level of maintenance planned for future years). Sometimes, there are interdependencies among projects: the costs and benefits of doing a project depend on the other projects that are in the project portfolio. Find out whether the software merely ranks projects or if it can truly optimize the project portfolio.
5. Does the software provide features that help ensure the quality of data inputs?
   Garbage in means garbage out. Experience shows that features that help promote accurate inputs and quality assurance are essential to obtaining accuracy in project recommendations. One useful (but not always easy to support) feature is the capability to provide users immediate feedback on the implications of project inputs in terms that are meaningful and easily understood. Tools that require the user to toggle between data entry and analysis mode in order to see how project inputs affect project evaluations can be cumbersome and more prone to data entry errors. Evaluating projects based on the consequences of doing versus not doing the project helps promote data quality, since users can rely on professional experience to assess whether consequence estimates for a specified project are reasonable. Likewise, a tool that values a project in terms of dollars provides an objective basis for assessing the results, “Would the organization really be willing to pay the indicated amount to obtain the project benefits?” Another useful feature is allowing users to analyze project concepts prior to “publishing” results for others to see. Check whether the system provides a place for colleagues to explore possibilities and share draft materials without committing inputs to the central database. A tool that allows project proponents to investigate the attractiveness of project proposals can help them to design better project alternatives. However, ask the vendor how the tool and application process counter “gaming” in the specification of inputs.
6. Are you communicating with someone with real expertise and experience?
   PPM is a hot topic, and consultants and vendors are reinventing themselves in their rush to offer tools for the job. Also, some large software vendors with established project management tools have created PPM tools by merely adding cross-project data rollup to their older tools. Such tools do not provide true assistance for optimizing project portfolios. The real opportunity for benefiting from PPM comes from enhanced ability to select the right projects, not from managing your existing projects more effectively. Make sure your candidate supplier provides evidence that they are truly leaders in the field of the “portfolio part” of project portfolio management.

Here at Folio, efficient frontiers are our daily bread: we talk about them in our marketing materials, we graph them, test them, tweak them, and we love showing our clients where their hand-crafted portfolio choices stand against the efficient frontier.

However, newcomers to portfolio optimization sometimes misunderstand what we mean by “efficient frontier.” We do not blame them: the expression is used in different ways in different disciplines. In this article, we contrast efficient frontiers in finance theory and in project portfolio management.

Efficient Frontiers in Finance

In Markowitz‘s modern portfolio theory, the efficient frontier is obtained by plotting, for each feasible portfolio of financial instruments, its expected return (i.e., its probability-weighted average return over all risk scenarios) against its risk exposure (i.e., the standard deviation or variance of this return).

It is beyond the scope of this short post to explain the underlying math. We will simply point out that it is the existence of correlation between individual financial assets (e.g., stocks) that allows us to combine them into portfolios yielding a higher return for the same amount of risk.

Ultimately, the math acrobatics are meant to address this question: how can I allocate 100% of my capital amongst various financial instruments so as to achieve the highest possible return for a given risk level I am willing to accept?

Any such allocation is deemed to be an efficient portfolio, in the sense that it is impossible to achieve a greater return without taking on more risk. The collection of all these efficient portfolios then constitute the efficient frontier (the upper part of the blue curve below).

Return vs. risk: the efficient frontier as defined in finance theory. Source: Bob Taylor.

A number of results, most notably Sharpe and Lintner’s Capital Asset Pricing Model (CAPM), further build upon this concept by making equilibrium assumptions about the market players.

For those wishing to drill deeper, here is a very concise technical introduction to modern portfolio theory.

Efficient Frontiers in Project Portfolio Management (PPM)

PPM also makes heavy use of the concept of efficient frontier, but it has a different meaning than in finance theory.

First of all, it addresses a different question, namely: to what extent does a higher investment level enable me to create more value by funding more (or different) projects?

By plotting the value created by a portfolio against the investment necessary to fund it, or, as it is often summarized, its total benefit against its total cost, we answer a question similar to, albeit different from, the one posed in finance. For a given cost level what is the maximum total value I can create by picking the right combination of projects?

Portfolio benefit vs. cost: the efficient frontier as defined in PPM. Generated by Folio Priority System.

From here, cost benefit analysis tells us how we can generate (an approximation of) the efficient frontier by ranking candidate projects by decreasing benefit-to-cost ratio — a method we can intuitively link to the decreasing slope of the red curve.

For a more rigorous treatment, we know that ranking alone is insufficient in the presence of complicating issues such as multiple funding levels, disparate project sizes, or project interdependencies, many of which occur in real-life applications.

Does This Mean PPM Ignores Risk?

No, it certainly doesn’t.

Managing portfolio risk is a critical component of success for organizations seeking to implement portfolio optimization. It raises a number of questions: What is the nature of the portfolio risks? Can they be diversified away? How do systemic risks affect the risk of the portfolio? How much risk can my organization tolerate?

One way to address at least project-specific risk is to penalize risky projects by considering their risk-adjusted benefit rather than their expected benefit when creating the efficient frontier.

Portfolio-level risks such as energy or commodity prices present another level of complexity which must be addressed separately from individual project adjustments.

In this post, we just wanted to lay out the possibly-confusing dual terminology. We will address risk more extensively in a future post.

At its most basic level, prioritizing projects is often accomplished by ranking them by decreasing benefit-to-cost (B/C) ratio. The projects featuring the biggest “bang for the buck” are then at the top of the list, and all that is left to do is determine how many top projects can be funded given the available budget.

The following table illustrates where to draw the funding line, assuming a $10 million budget.

Notice in passing that we would not want to go below a B/C ratio of 1, since funding such a project would yield less benefit than it would cost.

What are the problems with ranking?

Ranking furnishes a fine first cut at an optimal portfolio, but fails to fully address the following situations — all encountered by our clients.

• Ranking cannot handle interdependencies. If project X is required before project Y can truly yield benefits, or even be undertaken at all, clearly looking at the B/C ratio of X alone is inadequate. A mesh of interdependencies is common with complex infrastructure or IT projects, for examples. In that case, a comprehensive optimization of this network of projects is indispensable.
• More commonly, ranking fails to consider alternative funding levels. Imagine that each of the projects on the list, instead of an all-or-nothing choice, could be funded in various cheaper alternatives than the all-out version, featuring a spectrum of lower costs and lower benefits. Conventional ranking would simply pick, for each project, the one alternative with the highest B/C ratio. However, it can sometimes be optimal to try to cut costs on one project to enable another project to be funded in a more expensive alternative.
• Finally, ranking ignores fine-tuning, i.e., the so-called knapsack problem. The knapsack problem is often perceived to be an academic and unnecessary complication for real-life portfolio optimization. In the remainder of this article, we go through a real-life case study to evaluate the validity of this perception.

What is the knapsack problem?

Bear with us as we indulge in one short theoretical paragraph. The knapsack problem goes as follows: imagine you have a bunch of objects of various values and weights, from which you have to select any number to fit into a knapsack. Your goal is to create the most valuable knapsack possible, without of course exceeding the allowable weight capacity.

Source: Dake under Creative CommonsAttribution-Share Alike 2.5 Generic license.

The parallel with portfolio optimization is obvious: value is the benefit, weight is the cost, and the weight capacity of the knapsack is your budget constraint.

A ranking approach to this problem, therefore, would order the objects by decreasing density: the yellow object right underneath the pack has the highest “value density” — i.e., B/C ratio — (2.5$/kg) and will go in first: it adds a lot of value relative to a small weight consumed. The green box at the top left will go last, if at all: its value density is the lowest (.33$/kg).

Now imagine we start fitting these objects into the knapsack by picking the highest-density objects first, very much like we ranked projects by B/C ratio in the table above and started funding them, until we hit our limit.

As many of us know from experience, we might then decide to make some changes “on the margin”: even though object 26 was the last one to go in, it leaves a lot of unused room in the knapsack. It turns out if I removed object 23 and placed object 27 instead, I would fill the entire sack, even though I have sacrificed a higher B/C object for a lower one. The degree of “shuffling on the margin” is difficult to build a good intuition for. Many practitioners dismiss this phenomenon as, precisely, marginal.

We wanted to put this question to the test: is ranking really insufficient? Do we really need to optimize and deal with the nagging knapsack problem, or can we be content with the simpler ranking method?

Does optimization really matter in real life?

Optimization really does better than ranking in “real life.” The following example is taken from a portfolio of 901 real client projects (the data has been disguised by linear rescaling, but the shape of the curves and the conclusions are strictly identical).

The red curve represents the efficient frontier of the simple ranking method. The blue curve represents the efficient frontier of a full-blown optimization.

Notice how an optimization is able to extract more value for certain budget levels than a straightforward ranking. Why is that? There are two main reasons why optimization fares better:

• Optimization switches alternative funding levels in and out as appropriate. Even though a given project may have one clear winner funding level, featuring a higher B/C ratio than other funding level alternatives for this project, it is sometimes appropriate to be less “greedy” and fund a less costly version so as to enable another high-yielding project to be funded. This is the dominant effect in area C of the curve, where optimization systematically beats ranking by about $8 million.There are actually very few projects featuring multiple, alternative funding levels in this database.
• Optimization can handle big discrepancies in project spending. Of course, the most dramatic benefit of optimization over ranking can be seen at A. A single, very expensive project “holds up” other projects from being funded in the ranking method because the expensive project still comes first due to its high B/C ratio. In the optimization solution, rather than holding up the budget gap, other, lower B/C ratios are “filled into the knapsack” until such time as the big project can be funded. At that point, the red curve abruptly catches up with the blue one. The same phenomenon happens at B with another project, and, to a smaller extent, in other places along the curve where kinks are visible. The more heterogeneous the project sizes, the more value a straight ranking method is likely to leave on the table. At its worst discrepancy ca. a $46 million budget, 19% of the value is missed by the ranking method.

This case study should not be interpreted as an indictment of B/C-ranking. But it clearly illustrates one of the advantages of optimization in the presence of multiple funding levels and heterogeneously sized projects.