Why a research report can read well and prove nothing
Open almost any market report and you will find two kinds of sentence sitting side by side, looking the same.
One is a finding. Somebody looked at something, wrote down what they saw and where they saw it, and the sentence reports that.
The other is an inference nobody checked. Somebody who knows the industry wrote down what they would expect to be true. It may well be true. But nothing was observed, and nothing about the sentence tells you so.
Both are typeset the same way. Both read with the same confidence. The second kind is not dishonest — most of it is written by people who know their subject, and much of it turns out to be right. The problem is that you cannot tell which is which, and you are about to make a decision on the basis of both.
The test: can you walk backwards from the sentence?
There is one question that does most of the work. Take any sentence in the report that would change what you do, and ask:
If I wanted to check this myself, where would I start?
A finding answers that from the page. It names what was looked at, when, and what was there. You may not want to go and check it, but you could.
An inference cannot answer it, because there is nothing to go back to. You will find yourself reaching for the author's reputation instead of the evidence, which is the moment to slow down.
Apply the test to three or four sentences, not the whole document. The ones that matter are the ones a decision rests on. If those hold up, the rest usually does too. If they do not, the rest does not save it.
What are the warning signs in a market research report?
A few patterns come up again and again. None of them proves a report is bad. Each is a place to stop and ask one more question.
- A number with no date. A figure that was true three years ago is not wrong, but it is not current, and a report that does not say when stops you working out which you are looking at.
- A source that is itself a summary. A citation to another report, which cites another report, is a chain that usually ends somewhere nobody has looked. Follow it one step. If that step is not there, the number has no floor under it.
- An estimate wearing the clothes of a measurement. "Approximately" and "we estimate" are honest words. Their absence, on a number nobody could have measured, is the thing to notice.
- Growth with no base. Something growing quickly from a small base and something growing quickly from a large one call for opposite decisions, and a percentage on its own does not tell you which you are reading.
- A score with no workings. A single number summarising readiness, risk, attractiveness or maturity is a conclusion, not evidence. Ask what went into it and how each part was weighted. If that cannot be shown, the score is the author's judgement with a decimal point attached.
- A sample you cannot see. "Industry participants told us" raises a question the report should answer: how many, and chosen how. Six suppliers who already sell to you are not a market.
- No contradicting evidence anywhere. Real research turns up things that point the other way. A report in which every piece of evidence agrees with the conclusion has either been very lucky or has stopped looking.
What does an honest research report do when the evidence is thin?
This is the part worth reading for, and it is the fastest way to judge a firm.
A report that is honest about its limits will say "we could not establish this" in plain words, and say what it would take to establish it. It will mark an estimate as an estimate. It will tell you where the evidence was thin and what it did anyway.
That is not weakness and it is not padding. It is the thing that makes the confident sentences elsewhere in the document worth anything. A report with no admitted gaps is not a report with no gaps.
What should I ask before paying for market research?
You can settle most of this before any money changes hands.
- Ask what the deliverable will show for each conclusion. Not whether there will be sources, because everybody says yes, but whether each individual claim will carry the evidence it rests on, where you can see it beside the claim.
- Ask what happens when the evidence is not there. The answer you want is a stated gap. The answer to be wary of is one that treats this as unlikely.
- Ask which questions the study will not be able to answer. Anyone who has done this work knows the answer before you ask, and it is a short list. Someone who says none has either not thought about your question or is selling.
- Ask who the sources are, at the level of kind rather than name. Registration records, customs and trade data, marketplace listings, company filings and direct conversations are different kinds of evidence with different weaknesses. Knowing which ones a study rests on tells you where it will be strong and where it will be thin, before you read a word of it.
What can market research not tell you?
Worth saying plainly, because it sets what to expect.
No study can tell you what a competitor is about to do. It can tell you what they have done, which is usually enough to narrow the range.
No study can price your product. It can tell you what comparable things sell for, where, and under what conditions. The gap between that and your price is a judgement you have to make.
No study can tell you whether to go ahead. Research changes the odds you are working with and the things you have thought about. The decision stays yours, which is the correct arrangement, because you are the one carrying it.
The short version
Pick the three sentences the decision rests on. For each, ask where you would start if you wanted to check it. Then find the part of the report that admits what it does not know, and see whether it exists at all.
That is most of it. It takes about ten minutes, it works on a report from anybody, and it guards against the two expensive mistakes: acting on a confident sentence nobody checked, and ignoring a finding because it was written plainly.