Is it most effective to have 8 – 10 raters? (Part 7 of 10) June 28, 2011
Posted by talentinnovations in 360 degree feedback, Uncategorized.Tags: 360 degree feedback, Raters
add a comment
7. Is it most effective to have 8 – 10 raters?
They key debate in this area is around the question „do more raters lead to more useful and reliable information?‟
Best practice states that individuals undergoing a 360° feedback process should select between 8 to 10 raters, but why, why not select 4 – 40? The purpose a 360 process is to obtain multisource feedback. The number of rater categories and the number of raters within each category is not definitive. There are some key preferences outlined for which the benefits are clear, for example raters should only provide feedback to those they know well in a work related context; and, those they have worked with in the past 12 months. There should also be a minimum of two raters per category to preserve anonymity and avoid the need to merge ratings together; merging ratings protects anonymity but removes important distinctions between rater groups so is not preferable. This suggests a lower limit of two raters per category but does not help define an upper limit, or indeed the number of categories.
There are a broad range of benefits from having a large number of reviewers, these include: more perspectives will provide a more complete picture of personal strengths and development needs; and, more individuals per group makes it harder to attribute feedback to any one rater. It could however be argued that more information does not necessarily equate to more quality as the feedback may not be detailed enough to be useful for development.
Additionally, the more raters: the greater the cost in man hours to the organisation; the more reliance there is on an intelligent software platform to cope with the number; and the higher the risk of ratings showing a regression to the mean. Furthermore, research has shown no relationship between the number of raters and performance improvements in the focus.
It may therefore be advantageous to focus on the minimum number of raters needed to obtain useful feedback. Research by 3D Group showed that 65% of organisations had a minimum of 3 raters per rater category; they also showed the most common rater categories to be: self (92%), boss (94%), peer (96%) and direct report (98%). This suggests a minimum of 8 raters in total (3 per group except self and boss). 3D Group also showed that 46% of those questioned required all an individual‟s direct reports to provide ratings.
Overall, although there is some logical indication of the minimum number of raters that should be selected, there is no logical maximum, only a consensus view. At Talent innovations we advise between 6 and 15 raters. We also advise that raters are separated into a minimum of 4 categories (self, manager and two others e.g. peers, direct reports). This is a sufficient number to ensure anonymity and the ability to distinguish between group differences, and is not too great to risk of regression to the mean (where there are so many raters per category that when results are presented as an average it shown nothing more than the mean).
Next to come in this series is ‘who should see the feedback report?’
For the full paper, download it for free here.
Should individuals choose their own raters (Part 2 of 10) June 3, 2011
Posted by talentinnovations in 360 degree feedback, 360 feedback, Development, HR, Performance Management.Tags: 360 degree feedback, manager, Raters
add a comment
2. Should individuals choose their own raters?
The debate here should take into account the purpose of the 360° feedback. If it is to be used for development purposes only then receiving feedback from those you choose shows clear benefits; 
however if being used for performance management (link to pay and advancement) the risks associated only receiving feedback from those you select become clearer. The below also highlights key examples of why the purpose of the 360° feedback should be made clear from the on-set: the ideal way forward for development and performance management can vary and in some cases be directly opposing.
a. Using ratings for Development only
Opinion in this area is united in the view that individuals should choose their own raters. This helps individuals to feel ownership over the 360 feedback process and helps set up a foundation of trust and candour which is seen as extremely important for the success of the 360 process. More practically, it is often the individual themselves who will be able to best judge who they have worked most closely with and also importantly for the feedback to be taken on board the individual needs to want to hear the opinion of the feedback provider. Individuals tend to have a certain view about themselves and they need to trust the opinion of a rater if they are going to challenge that view.
“Individuals are only going to take on board feedback from those they are open to feedback from, those whose opinions they respect, and those whose opinions they are interested in. Therefore in some cases raters suggested by a manager may not fit this criteria and thus may not benefit an individual‟s development”
At Talent Innovations we agree with this view. Almost all of our clients allow the focus to choose their own raters. On rare occasions a client may ask for manager‟s approval to be built into the rater selection process, and it is even more rare for clients to have complete control over rater choice. We recognise that there is an argument that suggests a focus could skew their own results by selecting those they know will rate them well, but in our experience this is rarely an issue. We also agree with the view that the common themes should hold focus over outliers i.e. what are the consistent messages here.
b. Using ratings for performance management
The key debate here is about rater accuracy: given the potential consequences (e.g. pay increase/ promotion) of feedback, how fair is rater selection by the focus likely to be, and how accurate and reliable are ratings provided by these selected raters likely to be?
When selecting raters who will ultimately influence pay or promotion there is a lot more „at stake‟ for the focus. Therefore there may be a higher risk that an individual may skew their rater selection towards those that will give positive feedback. A way to avoid this could involve individuals selecting their own raters but then needing to seek management approval. A Manager could then ensure there is a fair mix and broad spread. An alternative method to avoid skewed results could be to provide training to individuals in selecting their raters. This training could also include how to select raters to gain a broad viewpoint, how to understand the feedback report and how to develop a personal development plan.
An alternative view point is that management should have complete control over selecting raters. In most cases a direct line manager should know who an individual has worked with most frequently. If there is any uncertainty the survey could be sent out to a wider group of raters with a note explaining that they should only complete the questionnaire if they have worked with the individual closely in the last 6 (possibly 12) months – this may also result in more raters so better quality output. This may also result in frustration and wasted time as individuals receive requests to complete feedback for those they are do not have a sufficiently close working relationship with.
The benefit of this method may be that the manager will be more likely to choose raters who the individual does not get on well with; in addition to those they do get on with, thus offering a wider perspective.
Our view at Talent Innovations is that when being used for performance management, there should always be management input into rater selection. We also advise that for line managers all direct reports should be included as raters.
Next post in this series will be on the topic of whether feedback should be attributable to an individual or not….
For the full paper, download it for free here.
