An Insider’s View on Product Management

June 20, 2009

Measure this, measure that

Filed under: Product Management — Gregory @ 11:02 pm
Tags: , ,

Making decisions is arguably the most important responsibility for managers and executives. Their decisions are expected to be rational, methodical and are often backed up by a flurry of statistics and other numbers.

After all, statistics and numbers are an important part of the decision making process. They are hard facts that help to quickly identify issues, spot opportunities and measure progress. They are quantifiable value that can be acted on. In contrast, abstract ideas and strategy are much more difficult to assess and to debate: they are less tangible.  Thus, managers tend to put a lot emphasis on numbers and metrics. But they often forget that it’s only a small part of the story and that those numbers cannot always be trusted.

Statistics can be manufactured

The first problem with statistics, is that they are easy to manipulate. In the matter the US government is a master con artist. Their favorite statistics: unemployment and inflation – also called CPI (core price index). It seems they always come up in the “acceptable range” or as expected month after month, no matter if reality seems different. In fact, Uncle Sam has a clear vested interest in understating those numbers. But curiously media and experts typically accept government readings at face value and rarely challenge the methodologies applied.

Indeed many government payments like social security, and bonds interests are directly indexed to the CPI – lower CPI translated into lower payments. Furthermore CPI is also used to calculate GDP, so lower CPI also means higher GDP, which makes the economy look better than it is.

How do they do that? There are plenty of websites explaining the mechanics behind CPI and all the changes over the years. But suffice it to say that CPI exclude energy costs and food costs because those are judged too volatile – no matter how high gas and milk prices can go. It also exclude housing price because housing is considered an investment. But most pernicious is the concept of hedonic adjustments: if the “quality of goods” changes, their price is accordingly adjusted. For example, let say you bought a standard computer for $1000 last year, and you buy a new standard computer for the same price this year. You would think the price has stayed the same. No so fast… Since the computer you got this year is faster, has more memory, etc… than last year, the CPI price will reflect this increase in quality. In that case it actually means the inflation price for the computer went down… (I wonder if they take into account that last year computer had Windows XP and this year it comes up with Vista. How would that compute for a change in quality? ).

Joke aside, the point is numbers can and are tweaked to support somebody point – sometime it’s not even conscious. The governments is doing it all the time, but they are not the only one – ask Jeffrey Skilling, CEO of now defunct Enron or check how the banks got us into the current recession.

Imaginary correlations

Even if we could work with reliable and accurate numbers, the trouble is they are often meaningless. People like to point out to imaginary correlation and justify why those number matters with… you guess what, other numbers. There is no better example than the stock market that is highly subjected to random fluctuations. Traders, desperate to find some explanations to the chaos, dutifully analyze stock data, chart prices and other metrics in the hope of finding the magic formula behind stock moves. However their feverish researches sometimes get misplaced and correlations are found in unexpected places. A famous example, is the Super Bowl indicator that according to investopedia is: “An indicator based on the belief that a Super Bowl win for a team from the old AFL (AFC division) foretells a decline in the stock market for the coming year, and that a win for a team from the old NFL (NFC division) means the stock market will be up for the year.” They go on by adding “Chalk it up to coincidence, but this indicator has been surprisingly accurate (around 85% correct) over the past years. Even so, you probably shouldn’t bet the farm on it.“

Well at least they invite to caution. Is the number accurate? Probably. Is that relevant? Be your own judge.

It’s not about what you know, it’s about what you don’t know

So you gathered some good statistics, verified their accuracy and feel comfortable that those numbers are relevant to your situation. Well the dilemma is: it’s not what you know that is going to hurt you, it’s what you don’t know. When presented with numbers, those are often taken out of context and even if accurate they can be the proverbial tree hiding the forest.

In the 2006 excellent film satire “Thank You for Smoking” inspired by the novel of Christopher Buckley, Nick Naylor the main character and spokesman for the Academy of Tobacco Studies, a tobacco lobby, has the following interview during the movie:

Senator Ortolan Finistirre: And what, so far, has the Academy concluded in their investigation into the effects of tobacco?

Nick Naylor: Well, many things actually. Just the other day they uncovered evidence that smoking can offset Parkinson’s disease.”

Of course in this situation everybody knows cigarettes are harmful. Even if the tobacco industry is able to prove smoking has some unexpected health benefits, people are unlikely to get convinced by a few deceptive studies. However in other situations, especially in the business world, the context is not always as clear. As a result naive managers can take decisions based on a few isolated statistics, without understanding the complete picture.

No substitute for experience and intuition

Numbers can be fascinating. They inspire comfort and confidence and contain some mystical value. However when accepted as absolute truth they can also be misleading and even plain dangerous for making decisions. A bit of caution is highly advised. Numbers are just a tool, a good starting point for reflection. They are not substitutes for experience, careful analysis and intuition.

In the famous words of Einstein: “Not everything that can be counted counts, and not everything that counts can be counted.”


Leave a Comment »

No comments yet.

RSS feed for comments on this post. TrackBack URI

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Blog at

%d bloggers like this: