Comments on: How long is a piece of string?

By: David Cooper

David Cooper — Wed, 04 Feb 2009 08:44:01 +0000

Mark,

To echo those below – thanks for this.

Providing the right balance of ‘free text’ or narrative comment fields throughout the survey is also important – too many boxes become time consuming and laborious for the rater; too few make it difficult to provide summary comments against a multitude of questions spread over several competency areas. Our own research indicates that the right balance is three to five free text comments boxes spread throughout the questionnaire. Thanks again, great bogg

By: scientificleader

scientificleader — Mon, 24 Nov 2008 17:40:51 +0000

I greatly appreciate your overall point. I’m a huge fan of using Computer-Adaptive Testing for 360 surveys, to reduce the time without many of the problems you note above. Similarly, I’m equally fond of “Mike” Linacre’s Facets method that is ideal for 360 surveys, and I’ve used to adjust for severity/leniency bias. I have a white paper on this on my website if you’d like more – http://www.scientificleader.com/3.html

By: Christopher Kata

Christopher Kata — Sun, 26 Oct 2008 22:48:12 +0000

Mark – the data you provide here is very useful! I’ve personally opted for more information over less. Coaching staff towards answering the questions as fully as possibl!e is the key! I’ve also preferred electronic 360 Feedbacks vs paper based because of the speed and convenience!

By: talentinnovations

talentinnovations — Tue, 30 Sep 2008 14:43:48 +0000

Brendan,

I completely agree – the really powerful feedback is often textual, and those 19 minutes were probably ‘richer’ thanks to all that text feedback.

However, I think it’s a mistake to rely too much on text feedback. Teasing out good quality ratings can be really valuable when clients are interested in aggregate statistics. And it’s all too easy for raters to not bother writing much (such as when it’s not in the company’s culture, as Vandy alludes to), in which case it’s really important for the design of the rated part of the questionnaire to be good enough to deliver insights without comments.

Mark.

By: Brendan

Brendan — Tue, 30 Sep 2008 13:35:59 +0000

Mark,

Thanks for this – the stats are really useful. We share your avoidance of “quick and dirty” solutions. I guess the really interesting thing for me is whether the 19 minutes spent answering the HR questions gave better feedback to work with than the 14 minutes people spent answering the engineering company questions. We generally find that the narrative feedback is more useful than rating scale scores and use that insight when trying to gauge “how many” questions are needed.

Brendan

By: Vandy

Vandy — Wed, 24 Sep 2008 13:21:34 +0000

Excellent post, Mark. This is a problem we also encounter when we’re reviewing question sets that have been used in some of the “cheap and dirty” solutions.

Another point to consider when thinking about the time to complete a survey is the nature of the organisation. If a large proportion of staff are scientific or highly numerate, they may be less comfortable with text responses. In your example, the engineers may well have found the rating responses easier (and therefore quicker) than the HR consultancy staff, simply because they work in absolutes a lot of the time.

Thanks for the great article.