Is a rating scale of 1 – 5 most effective? (Part 6 of 10) June 24, 2011
Posted by talentinnovations in 360 degree feedback, HR, Performance Management.Tags: 360 degree feedback, rating scale
add a comment
6. Is a rating scale of 1 – 5 most effective?
It is important to note that research in this area tends to focus on scales used for personality questionnaires; research focusing on 360° feedback is limited.
The key debate in this area is two-fold: firstly there is the question of whether a „not applicable‟ (NA) option should be included; and secondly whether more options lead to more accurate ratings?
The first assumption to explain here is what is meant by an effective rating scale. In this context an effective rating scale is one that allows a rater to express their views in a fast, accurate, easy to analyse and consistent way. It should also be sufficient to show between rater differences. The key aim is to be able to clearly differentiate between an individual’s strengths and development areas when completing a 360 survey. For this reason the focus will be on numerical rather than Behaviourally Anchored Rating Scales (BARS) (BARS would take too long to complete and would not be simple to analyse). It is acknowledged that each numerical value could relate to a simple descriptor e.g. above average to below average, always to never etc.
a. Should NA be included as a rating option
The debate in favour of including an NA option suggests that this gives more accurate feedback as for example an individual is not forced to rate a behaviour that they may not have seen. The debate against the inclusion of an NA option focuses on the risk of it being seen as the „easy option‟ as it provides no „colour‟ or real content.
Research conducted by one 360 provider has suggested that the inclusion of an NA option has clear benefits. In one study they showed that when using a three point scale („strength,‟ „adequate,‟ and „development needed‟) raters reported feeling „heavily constrained‟; they also avoided use of „development needed‟ (only 10% of ratings). In a second study, where NA was included as an option („strength‟, „proficient‟, „development need‟ and „not applicable‟), feedback from raters was far more positive; further more inter-rater reliability has also been shown to be higher.
The consensus view does suggest that NA should be included. At Talent Innovations we include the „NA‟ option in almost all of our off the shelf and bespoke 360 offerings. As suggested above, we feel this results in more accurate results as raters do not feel unduly „forced‟ to make a rating decision.
b. Do more rating options lead to greater rating accuracy
There is a continuum suggested in research: too few rating options and raters feel forced and restricted and it becomes difficult to distinguish between competency ratings; too many options and you get large between rater differences and it again becomes difficult to clearly identify strengths and development areas. So what is the ideal number, here‟s what the research says…
Some state that scales of five-points or less are too small to provide a clear delineation between core strengths and behavioural challenges. Raters should have access to at least seven rating options, and a ten-point scale provides for an even greater spread of responses. Further, raters should be encouraged to utilize the entire range. It could however be argued that having any more than a five point scale could lead to large between rater differences due to the way the scale has been used – this could be misleading; with a five point scale this risk is reduced.
As well as risks associated with too many rating options, too few may lead to increased central tendency as raters typically avoid rating at either end of the scale. This leads to uninformative responses. This reluctance to provide very high or very low ratings will mean that any scale that is used will effectively be reduced by two, for example a five point scale only really offers 3 options, with the midpoint being neutral, hence a smaller scale than this would not be helpful.
Looking at common practice may provide additional insight. From a survey of 360-degree feedback program managers from 50 companies including 3M and General Motors, 76% stated they use a 5 point rating scale; 4% have a 4 point scale, 4% have a 10 point scale, 16% have a 7 point scale.
The consensus view therefore suggests that a 5 point scale is the way to go. At Talent Innovations, after 10 years of tool development experience and 20 years of industry experience we have settled on a 6 point ratings scale (a typical 5 point scale with the addition of an NA option). Feedback we receive on this shows this scale to be very well received.
Keep watching for part 7 on how many raters is most effective in 360° feedback.
For the full paper, download it for free here.
Should individuals choose their own raters (Part 2 of 10) June 3, 2011
Posted by talentinnovations in 360 degree feedback, 360 feedback, Development, HR, Performance Management.Tags: 360 degree feedback, manager, Raters
add a comment
2. Should individuals choose their own raters?
The debate here should take into account the purpose of the 360° feedback. If it is to be used for development purposes only then receiving feedback from those you choose shows clear benefits; 
however if being used for performance management (link to pay and advancement) the risks associated only receiving feedback from those you select become clearer. The below also highlights key examples of why the purpose of the 360° feedback should be made clear from the on-set: the ideal way forward for development and performance management can vary and in some cases be directly opposing.
a. Using ratings for Development only
Opinion in this area is united in the view that individuals should choose their own raters. This helps individuals to feel ownership over the 360 feedback process and helps set up a foundation of trust and candour which is seen as extremely important for the success of the 360 process. More practically, it is often the individual themselves who will be able to best judge who they have worked most closely with and also importantly for the feedback to be taken on board the individual needs to want to hear the opinion of the feedback provider. Individuals tend to have a certain view about themselves and they need to trust the opinion of a rater if they are going to challenge that view.
“Individuals are only going to take on board feedback from those they are open to feedback from, those whose opinions they respect, and those whose opinions they are interested in. Therefore in some cases raters suggested by a manager may not fit this criteria and thus may not benefit an individual‟s development”
At Talent Innovations we agree with this view. Almost all of our clients allow the focus to choose their own raters. On rare occasions a client may ask for manager‟s approval to be built into the rater selection process, and it is even more rare for clients to have complete control over rater choice. We recognise that there is an argument that suggests a focus could skew their own results by selecting those they know will rate them well, but in our experience this is rarely an issue. We also agree with the view that the common themes should hold focus over outliers i.e. what are the consistent messages here.
b. Using ratings for performance management
The key debate here is about rater accuracy: given the potential consequences (e.g. pay increase/ promotion) of feedback, how fair is rater selection by the focus likely to be, and how accurate and reliable are ratings provided by these selected raters likely to be?
When selecting raters who will ultimately influence pay or promotion there is a lot more „at stake‟ for the focus. Therefore there may be a higher risk that an individual may skew their rater selection towards those that will give positive feedback. A way to avoid this could involve individuals selecting their own raters but then needing to seek management approval. A Manager could then ensure there is a fair mix and broad spread. An alternative method to avoid skewed results could be to provide training to individuals in selecting their raters. This training could also include how to select raters to gain a broad viewpoint, how to understand the feedback report and how to develop a personal development plan.
An alternative view point is that management should have complete control over selecting raters. In most cases a direct line manager should know who an individual has worked with most frequently. If there is any uncertainty the survey could be sent out to a wider group of raters with a note explaining that they should only complete the questionnaire if they have worked with the individual closely in the last 6 (possibly 12) months – this may also result in more raters so better quality output. This may also result in frustration and wasted time as individuals receive requests to complete feedback for those they are do not have a sufficiently close working relationship with.
The benefit of this method may be that the manager will be more likely to choose raters who the individual does not get on well with; in addition to those they do get on with, thus offering a wider perspective.
Our view at Talent Innovations is that when being used for performance management, there should always be management input into rater selection. We also advise that for line managers all direct reports should be included as raters.
Next post in this series will be on the topic of whether feedback should be attributable to an individual or not….
For the full paper, download it for free here.
Should 360° feedback be used for personnel and pay decisions as well as for development? (Part 1 of 10) May 26, 2011
Posted by talentinnovations in 360 degree feedback, 360 feedback, Development, HR.Tags: 360 feedback, Development, feedback, Salary
add a comment
Here at Talent Innovations our Client Manager, Suraiya, has written a paper challenging the 10 most common best practice topics in 360° feedback. Over the next coming months we will be discussing each topic and would love to hear your thoughts!! For the full paper, download it for free here.
Should 360 feedback be used for personnel and pay decisions as well as for development?
The key debate is two-fold: firstly is 360° feedback suitable for both development and personnel and pay decisions; and secondly if the feedback has been obtained for one purpose can it then be used for the other – the distinction is in the original intended use of the data.
a. Is the 360 approach suitable for development as well as for informing personnel and pay decisions?
Advice by one 360 provider suggests that in order for 360° feedback to be used as part of performance management, there first needs to exist an effective performance management system and culture. This will ensure that input from the 360 process is better understood and best utilised; but how accurate are current performance management processes? A meta-analysis of 50 studies involving a total of 8,341 individuals has shown a merely moderate relationship between objective ratings of performance and individual indicators of performance24. This suggests current processes tend not to be as effective as they could be. There may be many reasons for this, such as rater candour. For example, studies have shown that ratings made for performance appraisal purposes, may be increased if the rater wants a positive influence over reward decisions for the appraisee, or may be decreased if the opposite is desired. Ratings may also be inflated due to a reluctance to highlight development needs, either in the self or others.
“…where this a clear or indeed indirect connection to pay and reward this can have a huge impact on the content, accuracy and emphasis of the feedback and consequently the usefulness of it for the individual.”
Whatever the cause of discrepancy, performance assessment and reward decisions should be based on accurate data gathered using the most appropriate tool/ method. In this light it could be argued that the 360° feedback process was designed for development and, development and assessment are two very different concepts. The distinction between development and assessment lies in what is being measured, competence or results? It could be argued that 360° feedback is better at gathering data related to competence, where it should be results and potential that are rewarded. Furthermore it could be argued that feedback received from 360° ratings make it difficult to make the distinction between individual contribution versus team effort as in the 360 process there is an emphasis on the individual.
An alternative view, which supports the use of 360 for performance development, states that rewarding results without consideration for the methods used to get those results is not sufficient. Managers recognise that results are obtained through working with people and 360 offers a way to ensure results are not gained by compromising on this.
“We know they [managers] can’t achieve them [results] without working through people, so we don’t want them to improve the bottom line and collect bonuses for doing so while discouraging, misusing or burning out the talented people who produced those results. We don’t want them to kill the goose that lays the golden eggs.”
One very different view was presented by a law firm during a round table discussion hosted by one 360 provider. They were clear that they did not want to use 360° feedback to influence pay decisions:
“The business imperative is to win business, and it‟s understood that some of the most successful partners are not very nice in the process. Under no circumstances would we penalise them because they are not very nice and don‟t get on with their teams. We‟ll allow that behaviour because they bring in millions.”
It therefore seems that it may not always be appropriate for certain organisations to base pay decisions on the aspects measured in a 360.
Despite the above, research has shown that 22% of private sector organisations surveyed use 360 to support appraisals. Some feel that 360° feedback is a more accurate way to assess performance. Gaining a broader perspective from multiple raters leads to reduced bias; this may increase trust and engagement in the appraisal process as it is seen as fairer. The open comments element of many 360 tools may also help provide concrete examples of behaviour and impact and thus provide more reliable input into reward decisions. Some simply feel that having already invested in the process, using it for assessment as well as development will increase cost effectiveness.
At Talent Innovations we regularly use the 360° approach for performance management purposes. We strongly believe that the 360° approach provides the rounded approach required to make effective personnel and pay related decisions.
b. Can feedback gained for one purpose, then be used for the other?
Although research recognises the differing focus of 360° feedback for development and 360° feedback for personnel and pay decisions, there is limited advice on using data gathered for one purpose and then using it for the other.
What is published focuses on the ethical issues of participants not being fully aware of how their data will be used. Our thoughts at Talent Innovations however, focus on the fundamentally differing decisions that are required for each type of 360 process. For example if used for development we advise that the focus has complete choice over raters, whereas if being used for personnel decision we suggest at least some management input into rater choice.
If however it is decided that the same feedback data will be used for both there are certain criteria that should be met:
• The intended use of the feedback should be made clear from the onset.
• Feedback from the 360 report should not be used in its raw form. It may be more suitable to use the development plan that results from the 360 process. This will help put all feedback into context.
• The 360° feedback should form part of the performance appraisal and should not be the sole source of feedback.
• The manager feedback must be identifiable.
• The appraiser must be clear in how they should use the 360 data and not depend on anonymous feedback to provide the final performance rating.
Keep an eye out for the next part discussing whether individuals should choose their own raters…….
Now What? May 20, 2011
Posted by talentinnovations in 360 degree feedback, 360 feedback, Career planning, Coaching, Development, HR, Performance Management.Tags: 360 degree feedback, benchmark, coaching, Development, SMART, strengths, training
add a comment
A key question on our clients’ lips is, now what? I have a 360° process that works; it shows me the behaviours within my organisation both on a holistic and individual level. I can see where the strengths, weaknesses, and gaps are but how do I move forward and use this information to facilitate transformation?
If this is a question you have, here are a few points to consider:
1. Be clear where you are and where you want to be. For example there may be clear areas that are not as strong as others but are they required for success in your organisation/ in your career? A way to gain a greater understanding of this may be to benchmark yourself against other similar organisations/ individuals – what are their strengths, do you need them to be your own? From an organisational level this may also involve conversations with the board – where do they see the gaps that will impact future success?
2. Create a plan and set SMART milestones for achievement of the ‘to be’ state. E.g. Sales capability is low across the organisation, your 6 month goal is to have a ‘sheep dip’ approach and get basic skills up to date through online training for 75% of the organisation. Your 24 month plan may be to identify a core sales team who’s role is at least 50% sales focused, that will receive more focused training from and external sales consultant and on-going mentoring and coaching from internal regional sales leads.
3. Get clear on the training, mentoring, coaching and other personal development facilities and individuals available to you. Do you have the resources to fill the identified gaps? If not start to investigate what external resource you may need.
4. Up-skill line management to coach and mentor their staff and have open, ‘safe’ development conversations.
5. Ensure employees take responsibility for their own personal development and understand their contribution to achieving overall corporate goals and development needs.
6. Ensure senior management walk the talk and are visibly taking personal development seriously. This may involve up-skilling them to do this.
7. Make the process ‘easy’ and transparent, with clarity of expectations at all levels.
8. Keep the momentum going, lots of promises and then no action will have a detrimental effect.
9. Ensure individuals see the way forward and know that their dedication to personal development will be recognised and rewarded.
10. Make it self-driven, you cannot force an individual to develop and change at a rate that does not suit them.
Suraiya Rasheed
Should feedback ever be attributable to an individual? (Part 3 of 10) June 10, 2011
Posted by talentinnovations in 360 degree feedback, 360 feedback, Development, HR.Tags: 360 degree feedback, 360 feedback comments, anonymous, attributed data
add a comment
3. Should feedback ever be attributable to an individual?
For the full paper, download it here for free!
The debate here is that on one hand feedback should be attributable as this created the opportunity for the focus to approach their raters for clarification and encourages an open honest culture; but on the other hand, some argue that it takes a very mature culture to cope with attributed feedback and in fact this may distract from the content of the comments, as individual become pre-occupied with who said what about them. Furthermore raters are reluctant to highlight development areas at the best of times and may be even less inclined to do so if they felt they could be identified and held to account for their views.
The outcome of an online forum discussion suggested that most tend towards anonymous feedback, especially when: the 360 process is new, trust levels within the organisation are low, and the culture of the organisation does not support open feedback. This is consistent with best practice guidelines which state that all feedback should be anonymous other than that from the self and manager.
There is a perception that anonymity encourages openness and reduces the risk of reprisal from outlining development points. This last point is illustrated by results to a survey conducted by 3D Group where 96% of 360-degree Feedback program managers surveyed had promised anonymity to raters with the assumption that this would help obtain more honest feedback and less skewed ratings. The point is addressed again in studies that have shown that compared to anonymous raters, known raters tend to score significantly higher (around two thirds higher) and their scores tend to be less correlated with performance. The reason for this inflation of scores is that it is difficult for individuals to have open discussions about weaknesses.
It may however be argued that anonymous ratings do not help to harbour an open and honest working environment. Furthermore in order for 360° feedback to be trusted it has to be valued; anonymous feedback may reduce the quality of open comments as raters know they will not be individually called to question, and individuals are unable to approach raters for clarification, this may impact the value of the whole process.
“Not understanding what behaviour the respondent has seen will lead to a general ignoring or distorting of the data. The individual therefore needs sufficient understanding of the context of comments to gain full value.”
At Talent Innovations we insist on attributed comments from managers but then ensure anonymity for all other raters. We do however ensure that comments can be attributed if needed, although this information is rarely passed onto the client or focus. The reason we ensure we can at least attribute comments is simple, we want to ensure that comments provided are constructive. We also want to ensure that feedback providers are suitable prepared, so we may flag up particularly difficult feedback to them before sending over a feedback report. In a recent example this service helped to flag up an existing management issue that was already being managed by our client; with this advance support our client was able to ensure feedback was put into context and delivered by the most appropriate internal coach. This resulted in the focus, who could have reacted in quite a negative way, being a great advocate of the process, in our clients own words „she loved it!‟.
Next topic discussed in part 4 will be, do qualitative comments relate to improved performance?
Share this:
Like this: