Best In Test?

Gear: we all love it, don’t we? Let’s face it, after talking about where we’ve been to it’s probably the number one topic of conversation. And every month, page after page of magazine or blog space is dedicated to discussing the finer points of this versus that.

Increasingly bloggers are contributing a wealth of information about products and how they perform, mainly through the medium of the single item test. The advantage of this is that an in-depth review can be formulated based on extended use of the item in question, usually tested in real-life situations.

Very occasionally a blogger may get sent an item for testing from an obliging source. Normally this comes with the proviso that the retailer or manufacturer concerned receives a name check and, in some cases, that the product is returned afterwards (for example if it’s a prototype). However, in spite of the apparent generosity of the suppliers, most bloggers manage to remain impartial in their findings, and will offer balanced reviews as far as their experience will allow them.

However, despite the advantages outlined above, there is a downside to such testing: it is rarely able to be comparative. I don’t know about you, but I don’t have enough spare cash to go out and buy all the latest gear with the sole intention of seeing how good one product is against another! For this, we rely on the group test.

In the main, group tests are the preserve of the big circulation magazines: those organs of the press with sufficient penetration to reach thousands of prospective buyers, and whose pronouncements are capable of making – or breaking – a product. Rarely, if ever, will an individual blogger get the opportunity to test all the latest releases against one another in a comparative test.

Obviously, with the increased number of bloggers contributing on-line individual product reviews, the position of the professional reviewer has subtly altered (by “professional” I mean those who work for the magazines in question). However, the professional reviewer is still in a position of some responsibility given that their views will be widely read and may, ultimately, influence the development of future product.

The question is: do they wield this responsibility … er … responsibly?

Around a year ago, two magazines were reviewing lightweight waterproof jackets. As I was reading I noticed something odd – the same jacket was awarded a 100% score from one magazine (and given “Best In Test”) yet scored only 60% in the other, just about the lowest mark given. How could this be? The same jacket assessed for the same purpose. What were the circumstances that threw up two such diverse summations of the same product?

Of course, this happens all the time. We all have our own favourites, whether we are judging baked beans, washing powder, cars, TV shows or whatever. It’s only natural: we are all different. But it got me thinking: surely, part of the reviewer’s job in a group test is to put personal preference to one side and determine what might be a good purchase for ALL potential users/buyers?

Now I’m not suggesting any bias here, or any intention on their behalf to deliberately mislead, simply that sometimes these reviews may not be as impartial as we are given to understand and that, even with the best of intentions, personal preferences might cloud the tester’s objectivity.

Amongst this months’ crop of outdoor mags I came across another gear test that threw up a couple of anomalies. For the sake of the magazine and reviewer in question, I won’t make too detailed a reference. What I will say, though, is that both have respectable credentials amongst the outdoor fraternity, and both are eminently more qualified to discuss the merits of a product that I am myself.

However, as far as I could see, it became quite clear, quite early on that a particular perspective were being applied to the test – and one that not every potential purchaser of the product might apply. As a result, it is my own personal opinion that one or two decent products were unfairly marked down and at least one product with an identified (and possibly major) flaw was scored second only to Best In Test. That can’t be right, can it?

Sadly, it’s quite usual to see decent products unfairly pilloried. Often, this is because it fails some reviewer’s self-selected opinion of whether it is right or not. Take weight, for example, one of the biggest and most arbitrary culprits, with one version being deemed too “heavy” and another “light” enough when, in reality, there is only a few grams between them. The stupid thing is that very few products at all are “heavy” these days – just compare them to 10 or 20 years ago!

Of course it is much harder to slag something off if you have paid good money for it, unless it’s a complete turkey, as it suggests bad judgement on behalf of the purchaser. Equally, though, it is easy to do so if you haven’t had the anguish of parting with your cash.

So what does all this tell us? Very little, really, except that human nature is what it is, and will often prevail no matter how hard we try to be impartial. But it does illustrate two things to be mindful of when assessing group tests: one; use any reviews as a guideline only and, especially on big-ticket items, try before you buy, and two; it is much better to express conclusions as “I don’t like this” (opinion) rather than “It is bad” (statement of fact) because it may well be good for someone, sometime.

This might seem perfectly obvious, but with so much more gear being bought on-line, and with an increasing preponderance of stores with only quite basic product knowledge, doing your own homework and understanding what YOU want out of a product seems even more essential than ever.

As I said before, I'm sure there is no intent to mislead. It's just that there are so many more commentators out there that buyers and users are getting a bit more clued up these days. I may have no engineering qualifications or wide knowledge of the whole outdoors product market, but I have been walking up and down the UK and abroad for 35 years or more, so can claim to have some experience of what may or may not work! I'm sure there are thousands of others with similar experience too, all of whom can add to the cache of information we can call on.

So, are gear tests providing you with the information you need?

What do you think?