HTML Logo by World Wide Web Consortium ( Click to learn more about our commitment to accessibility and standards.

Moving forward with Composr

ocPortal has been relaunched as Composr CMS. ocPortal 9 is superseded by Composr 10.

Head over to for our new site, and to our migration roadmap. Existing ocPortal member accounts have been mirrored.

Feedback methods that don't lead you down the garden path

Feedback methods that don't lead you down the garden path One hard lesson I've learnt at ocProducts is to never rely on unsolicited user feedback to gauge and drive product development. This post will explain the issues, and detail better feedback methods.

Anyone who looks at the forums will see I reply to all kinds of things (and that I've built up a reputation for it), and quite often I mention I'll make a change in the next version if someone finds a usability issue.

But this does not control our ocPortal roadmap.

Here are a few hard realities:

  • By my estimation it averages over 10 people to experience a bug that breaks functionality for 1 person to bother reporting it. Therefore you cannot assume things work just because nobody is complaining. Evidence: we often find bugs via our automated error system, not through human contact.
  • If a bug is just a little bit annoying, or a particular deep interface is ugly but usable, chances are nobody at all will report it. This means you cannot ever decide user experience improvements by relying on user feedback. The kind of user who reports small issues is usually someone highly engaged, like an early adopter, so as your success grows you probably won't get a particularly increased quantity of detailed quality feedback. Most people are just too busy, or too focused on things that deliver value directly for them. In fact, in a sense if someone has learnt the way around an issue it is in their advantage to not explain it because they throw away a bit of earned competitive advantage (this selfishness annoys me, but I think it happens).
  • Many people either give only negative feedback (people who want to drive positive change through criticism), or only positive feedback (particularly friendly people who like to thank developers). This makes it very hard to gauge satisfaction and success accurately. Personally, from a utility perspective, I actually dislike both: negative feedback is bad PR and the really glowing feedback that is often posted overlooks things doesn't help me.
  • The kind of people who give general feedback are the most engaged and thoughtful subgroup of users, a tiny minority. Therefore, these people speak for the minority, and the feedback is usually to add new features that will actually distract from more important objectives (little usability details, design issues, and so on) and possibly risk creating bloatware (see second system syndrome). Moreover though, it prioritises the concerns of a minority and it can lead to a never-ending software development process where you're not particularly meeting the essential needs of the wider market any better but spending far too much money.
  • Generally some feedback can be a bit bizarre. For example, quite a few times we have been sincerely asked to create games or game arcades for ocPortal. Bringing great business tools to the public via Open Source brings really useful features that everybody needs: but bringing games into business websites is going to actively discourage business adoption of ocPortal (and adoption by just about anyone who is not creating a gaming website).
  • If you only listen to existing users you are ignoring the potentials you never managed to convince, and what stopped them adopting. Geoffrey Moore talks about the problems of moving beyond early adopters in his famous book, Crossing the Chasm.
  • Even if you use surveying as your feedback method, you will either only get feedback from the most engaged users, or you will get lazy feedback from very selfish users (if you awarded a prize).
So you need to throw away any idea of deciding your priorities through only unsolicited feedback from existing users. Always look at the feedback and give it very serious consideration, but never use it to decide your roadmap.

So what is a better feedback method?

You need to look at what your model of your target customer is, and then seek out these kinds of people for usability testing. You'll be amazed what very different feedback you find from them if you work from this fresh external perspective. Watch them using the product, give them tasks to perform and see how they do. You will see them get stuck on things you never imagined.

For ocPortal 7.1 we did usability testing via, testing relevant people we knew, and also by hiring people on Elance and asking them to do record their performance of tasks on Jing.

It's fairly easy to find resolutions once you know the real issues that will impact adoption.

This was article 4 of 8 in my "Entrepreneur reality check" series of blog posts.

If you think it's good advice, please share this link with others. If you think I'm wrong or have something to say, please discuss below.

View all


There have been no trackbacks yet