BLOG 70. How Reliable are Magazine Product Tests?

BLOG 70. How Reliable are Magazine Product Tests?

by Cliff Jacobson

I am often asked this question:

We’re planning a wilderness canoe trip and want to be prepared with the best equipment. I hear there’s a great new trail stove (tent, rain parka, canoe pack etc.) on the market that is awesome. It was top rated in the last issue of “Serious Camping Magazine”.  What do you think of this hot new stove?  Should I buy one?

            Hightech Harry

MY ANSWER

Don’t take magazine product reviews too seriously.  Writers work on deadline and are usually paid by the length of copy they produce not the time they spend researching and field-testing.  Time is money, so research and product testing are kept to a minimum.  Bad reviews irritate advertisers, which are a magazine’s life blood.  For this reason, writers are encouraged to tone down criticisms.

 

Some hard packs that work well over the long haul.  L to R: 5 gallon bucket with  waterproof Gamma® lid (fits inside CCS Quad-pocket pack (yellow); CCS foam-lined food pack; EM Wanigan (no longer made); Barrel with harness; Adirondack pack basket inside #2 Duluth Pack Cruiser.

For example, many tents and garments have small zippers that won’t take serious abuse.  But you’d better not write it that way.  Ever notice how often the word “may”—as in “may fail”—appears in equipment evaluations? You will never see the words "will fail".

In the 1980’s, as a contributing editor for “Backpacker Magazine” I evaluated many products—compasses, tents, trail pads and more. In those days, we called them “evaluations”, not reviews, because that is what they were. The products were scrupulously field tested, often for weeks or months. And the resulting evaluations often consumed a dozen or more pages in the magazine.  But too often, an honest evaluation was a bad evaluation and angry manufacturers responded by pulling their ads. So we changed our procedure from honestly “evaluating” new products to just “reviewing” them—that is, we provided specifications (length, weight, packed size, color etc.) and not much more. This pleased advertisers. And most readers didn’t pick up on the editorial change. 

 

Cliff fries fish on his 30 year old Optimus 111B stove. No stove is more  trouble-free over the long haul

Frankly, the term “expedition-proven” doesn’t mean much any more because modern canoe “expeditions”  seldom last long enough to prove anything. For example, I once made a 17 day canoe trip where the only rain was a short drizzle.  Needless to say, my rain gear worked perfectly!

The best advice is to carefully examine everything before you buy.  If a zipper looks weak or too small, it probably is. If there’s a plastic knob that can burn off or break, it likely will. How will the product perform in high winds or when it’s caked with mud or soaked with rain?  Will it break if you drop it?  Can you repair it in the field with simplel tools? Does it work as well in sub-zero temperatures as in blistering heat, on a high mountain top and in Death Valley? (The best butane stoves will fail this test.)

Be aware that some of the most highly touted products which work flawlessly over the short haul, fail miserably when the weeks turn to years. So be wary of advertising claims and the testimonials of individuals whose experience is limited. Instead, seek the advice of those who travel wild places year after year. These are the real experts even though their opinions are seldom seen in print.  

 

CCS "Lean 3" -- A super-reilable, bug-free shelter for four

All this can be summarized in a word—trust!  Why change your current tent, trail stove, sleeping bag or whatever, if it has never let you down?  Conversely, if an item is dangerously worn, or you think something better has come along, try the new replacement for a time—a long time, before you commit to it for a lengthy expedition. Trust doesn’t come in two weeks!

Cliff Jacobson

www.cliff-jacobson.com