Reviewing software, especially enterprise software, intelligently is virtually impossible unless you’ve had a chance to use the software, possibly for months, in a reasonably realistic environment.

A friend of mine, a really sharp engineer, used to write a software column for a highly respected (non-IT) trade magazine. He got tons of marketing-speak from vendors. But he didn’t have the resources to determine whether any of the stuff actually did what it was supposed to do as well as it was supposed to do it. The main reason the column existed was so that ad sales people could call on software vendors and tell them, “Yes, we write about software.”

The truth is that unless you have frank information from real users, it’s nearly impossible to understand how a particular package will work in your environment. Even then, it’s a crapshoot. Especially when you’re talking about something as complex, raw, and organizationally dependent as content management software.

One reason Walt Mossberg seems so brilliant is that he actually uses the stuff he writes about and tells the truth without worrying about his or his employer’s relationship with the vendor.