How long is a piece of string? September 24, 2008
Posted by talentinnovations in Uncategorized.Tags: 360 feedback, psychometrics, time
6 comments
A key decision in any 360° feedback project is how long to make your survey. Here I show how smaller isn’t always better.
The obvious argument for making a 360° survey smaller is to save people time, but this can come at the expense of depth, breadth and clarity of feedback. And we often see customers try to ‘squeeze together’ behaviours to reduce the number of questions. Here’s a real example we’ve seen from the standard 360° questionnaire (a mere 24 questions long) used by one of the biggest retailers in the UK:
“Gains commitment to achieving results through simple communication, actively listening and by adapting leadership style to the needs of others”.
I spot at least 4 different behaviours in there! A manager could easily be good at some but poor at others. This complexity makes it difficult for reviewers to understand and score the question, and hard for the recipient of the feedback to interpret the results. Both practically and psychometrically it is virtually useless!
Our approach is to work with our clients to develop a set of items that are simple and easy to understand, measure observable behaviours, and have the breadth to cover all of the important competencies. This can make for a longer survey, so what’s the time impact of all those extra questions?
We’ve been tracking how long people take to do their questionnaires, and have found some interesting results. The time taken to complete different aspects are as follows:
- Each rating on a 1-to-5 scale: 7 seconds
- Each word of written feedback: 7 seconds
It’s the written feedback that makes the biggest difference. Here are two real example projects we recently implemented:
| Project | Rated Items | Total written feedback (average) | Average time taken |
|---|---|---|---|
| HR Consultancy | 44 | 177 words | 19 mins |
| Engineering Company | 104 | 42 words | 14 mins |
Despite having less than half as many items, the HR consultants took longer to complete their surveys than the engineers!
So fewer questions does not necessarily mean less time, but as seen above, fewer questions can result in unclear, incomplete and hence less insightful feedback.
We are committed to enabling truly powerful feedback. Demands for “quick and dirty” solutions are best met with the question “what do you really want to achieve?”
Mark
