Is a rating scale of 1 – 5 most effective? (Part 6 of 10) June 24, 2011
Posted by talentinnovations in 360 degree feedback, HR, Performance Management.Tags: 360 degree feedback, rating scale
add a comment
6. Is a rating scale of 1 – 5 most effective?
It is important to note that research in this area tends to focus on scales used for personality questionnaires; research focusing on 360° feedback is limited.
The key debate in this area is two-fold: firstly there is the question of whether a „not applicable‟ (NA) option should be included; and secondly whether more options lead to more accurate ratings?
The first assumption to explain here is what is meant by an effective rating scale. In this context an effective rating scale is one that allows a rater to express their views in a fast, accurate, easy to analyse and consistent way. It should also be sufficient to show between rater differences. The key aim is to be able to clearly differentiate between an individual’s strengths and development areas when completing a 360 survey. For this reason the focus will be on numerical rather than Behaviourally Anchored Rating Scales (BARS) (BARS would take too long to complete and would not be simple to analyse). It is acknowledged that each numerical value could relate to a simple descriptor e.g. above average to below average, always to never etc.
a. Should NA be included as a rating option
The debate in favour of including an NA option suggests that this gives more accurate feedback as for example an individual is not forced to rate a behaviour that they may not have seen. The debate against the inclusion of an NA option focuses on the risk of it being seen as the „easy option‟ as it provides no „colour‟ or real content.
Research conducted by one 360 provider has suggested that the inclusion of an NA option has clear benefits. In one study they showed that when using a three point scale („strength,‟ „adequate,‟ and „development needed‟) raters reported feeling „heavily constrained‟; they also avoided use of „development needed‟ (only 10% of ratings). In a second study, where NA was included as an option („strength‟, „proficient‟, „development need‟ and „not applicable‟), feedback from raters was far more positive; further more inter-rater reliability has also been shown to be higher.
The consensus view does suggest that NA should be included. At Talent Innovations we include the „NA‟ option in almost all of our off the shelf and bespoke 360 offerings. As suggested above, we feel this results in more accurate results as raters do not feel unduly „forced‟ to make a rating decision.
b. Do more rating options lead to greater rating accuracy
There is a continuum suggested in research: too few rating options and raters feel forced and restricted and it becomes difficult to distinguish between competency ratings; too many options and you get large between rater differences and it again becomes difficult to clearly identify strengths and development areas. So what is the ideal number, here‟s what the research says…
Some state that scales of five-points or less are too small to provide a clear delineation between core strengths and behavioural challenges. Raters should have access to at least seven rating options, and a ten-point scale provides for an even greater spread of responses. Further, raters should be encouraged to utilize the entire range. It could however be argued that having any more than a five point scale could lead to large between rater differences due to the way the scale has been used – this could be misleading; with a five point scale this risk is reduced.
As well as risks associated with too many rating options, too few may lead to increased central tendency as raters typically avoid rating at either end of the scale. This leads to uninformative responses. This reluctance to provide very high or very low ratings will mean that any scale that is used will effectively be reduced by two, for example a five point scale only really offers 3 options, with the midpoint being neutral, hence a smaller scale than this would not be helpful.
Looking at common practice may provide additional insight. From a survey of 360-degree feedback program managers from 50 companies including 3M and General Motors, 76% stated they use a 5 point rating scale; 4% have a 4 point scale, 4% have a 10 point scale, 16% have a 7 point scale.
The consensus view therefore suggests that a 5 point scale is the way to go. At Talent Innovations, after 10 years of tool development experience and 20 years of industry experience we have settled on a 6 point ratings scale (a typical 5 point scale with the addition of an NA option). Feedback we receive on this shows this scale to be very well received.
Keep watching for part 7 on how many raters is most effective in 360° feedback.
For the full paper, download it for free here.
Should individuals choose their own raters (Part 2 of 10) June 3, 2011
Posted by talentinnovations in 360 degree feedback, 360 feedback, Development, HR, Performance Management.Tags: 360 degree feedback, manager, Raters
add a comment
2. Should individuals choose their own raters?
The debate here should take into account the purpose of the 360° feedback. If it is to be used for development purposes only then receiving feedback from those you choose shows clear benefits; 
however if being used for performance management (link to pay and advancement) the risks associated only receiving feedback from those you select become clearer. The below also highlights key examples of why the purpose of the 360° feedback should be made clear from the on-set: the ideal way forward for development and performance management can vary and in some cases be directly opposing.
a. Using ratings for Development only
Opinion in this area is united in the view that individuals should choose their own raters. This helps individuals to feel ownership over the 360 feedback process and helps set up a foundation of trust and candour which is seen as extremely important for the success of the 360 process. More practically, it is often the individual themselves who will be able to best judge who they have worked most closely with and also importantly for the feedback to be taken on board the individual needs to want to hear the opinion of the feedback provider. Individuals tend to have a certain view about themselves and they need to trust the opinion of a rater if they are going to challenge that view.
“Individuals are only going to take on board feedback from those they are open to feedback from, those whose opinions they respect, and those whose opinions they are interested in. Therefore in some cases raters suggested by a manager may not fit this criteria and thus may not benefit an individual‟s development”
At Talent Innovations we agree with this view. Almost all of our clients allow the focus to choose their own raters. On rare occasions a client may ask for manager‟s approval to be built into the rater selection process, and it is even more rare for clients to have complete control over rater choice. We recognise that there is an argument that suggests a focus could skew their own results by selecting those they know will rate them well, but in our experience this is rarely an issue. We also agree with the view that the common themes should hold focus over outliers i.e. what are the consistent messages here.
b. Using ratings for performance management
The key debate here is about rater accuracy: given the potential consequences (e.g. pay increase/ promotion) of feedback, how fair is rater selection by the focus likely to be, and how accurate and reliable are ratings provided by these selected raters likely to be?
When selecting raters who will ultimately influence pay or promotion there is a lot more „at stake‟ for the focus. Therefore there may be a higher risk that an individual may skew their rater selection towards those that will give positive feedback. A way to avoid this could involve individuals selecting their own raters but then needing to seek management approval. A Manager could then ensure there is a fair mix and broad spread. An alternative method to avoid skewed results could be to provide training to individuals in selecting their raters. This training could also include how to select raters to gain a broad viewpoint, how to understand the feedback report and how to develop a personal development plan.
An alternative view point is that management should have complete control over selecting raters. In most cases a direct line manager should know who an individual has worked with most frequently. If there is any uncertainty the survey could be sent out to a wider group of raters with a note explaining that they should only complete the questionnaire if they have worked with the individual closely in the last 6 (possibly 12) months – this may also result in more raters so better quality output. This may also result in frustration and wasted time as individuals receive requests to complete feedback for those they are do not have a sufficiently close working relationship with.
The benefit of this method may be that the manager will be more likely to choose raters who the individual does not get on well with; in addition to those they do get on with, thus offering a wider perspective.
Our view at Talent Innovations is that when being used for performance management, there should always be management input into rater selection. We also advise that for line managers all direct reports should be included as raters.
Next post in this series will be on the topic of whether feedback should be attributable to an individual or not….
For the full paper, download it for free here.
Now What? May 20, 2011
Posted by talentinnovations in 360 degree feedback, 360 feedback, Career planning, Coaching, Development, HR, Performance Management.Tags: 360 degree feedback, benchmark, coaching, Development, SMART, strengths, training
add a comment
A key question on our clients’ lips is, now what? I have a 360° process that works; it shows me the behaviours within my organisation both on a holistic and individual level. I can see where the strengths, weaknesses, and gaps are but how do I move forward and use this information to facilitate transformation?
If this is a question you have, here are a few points to consider:
1. Be clear where you are and where you want to be. For example there may be clear areas that are not as strong as others but are they required for success in your organisation/ in your career? A way to gain a greater understanding of this may be to benchmark yourself against other similar organisations/ individuals – what are their strengths, do you need them to be your own? From an organisational level this may also involve conversations with the board – where do they see the gaps that will impact future success?
2. Create a plan and set SMART milestones for achievement of the ‘to be’ state. E.g. Sales capability is low across the organisation, your 6 month goal is to have a ‘sheep dip’ approach and get basic skills up to date through online training for 75% of the organisation. Your 24 month plan may be to identify a core sales team who’s role is at least 50% sales focused, that will receive more focused training from and external sales consultant and on-going mentoring and coaching from internal regional sales leads.
3. Get clear on the training, mentoring, coaching and other personal development facilities and individuals available to you. Do you have the resources to fill the identified gaps? If not start to investigate what external resource you may need.
4. Up-skill line management to coach and mentor their staff and have open, ‘safe’ development conversations.
5. Ensure employees take responsibility for their own personal development and understand their contribution to achieving overall corporate goals and development needs.
6. Ensure senior management walk the talk and are visibly taking personal development seriously. This may involve up-skilling them to do this.
7. Make the process ‘easy’ and transparent, with clarity of expectations at all levels.
8. Keep the momentum going, lots of promises and then no action will have a detrimental effect.
9. Ensure individuals see the way forward and know that their dedication to personal development will be recognised and rewarded.
10. Make it self-driven, you cannot force an individual to develop and change at a rate that does not suit them.
Suraiya Rasheed
Performance Management Tension April 14, 2008
Posted by talentinnovations in Performance Management.Tags: Annual Review, Development, Performance Management
1 comment so far
For a while now we’ve had a performance management module in our software. We think it’s pretty good – it links the person’s annual review with their 360 scores by competency, and allows HR managers to monitor everything easily. We developed it for one particular client, and they’ve recently had a bit of trouble with their ‘annual review politics’.
The basic problem is this: there is an inherent tension in doing performance reviews between consistency and openness. There are basically two ways of doing it:
- Have the review reflect an honest conversation between the manager and the individual about the individual’s performance over the year
- Have the review reflect a ‘calibrated’ measure of performance that fits correctly with everyone else’s performance reviews (and which can therefore be used to feed into bonuses / raises / etc.)
If you go for option 1 then the obvious thing to do is for the manager to only decide on the review score at the end of the review meeting. Whereas if you go for option 2 then it is important that the review score is agreed with other people (e.g. among the department leadership team) before the review meeting even happens.
Our client’s managers were using option 1, but then they all got together and realised that some managers had been far too generous, and so they had to ‘recalibrate’ everyone’s reviews, which was obviously very disappointing for some individuals who thought they were in line for fantastic bonuses! In other words, the reality (that performance reviews have to be consistent and agreed among many stakeholders) undermined the ideal (that performance reviews reflect the conversation between the manager and the individual).
So anyway, we’ve been thinking people could potentially redesign their performance management tools more closely around a new sort of process that would try to overcome this problem. You could actually have 2 separate review scores – one which is ‘what the manager thinks’, and another for ‘what the company thinks’. To avoid confusion, the two scores would deliberately be on different scales (e.g. the first on a 4-point scale and the second on a 6-point scale) in order to avoid one devaluing the other. You’d be making explicit something which is usually hidden – that a person’s final review score arises from their manager making their case to the other people who control the finances.
We think it might be a great idea, but can’t help wondering whether people really want to be made so aware of the truth?
Mark.
Do qualitative comments relate to improved performance? (Part 4 of 10) June 17, 2011
Posted by talentinnovations in 360 degree feedback, Performance Management.Tags: 360 degree feedback, coaching, Development, qualitative comments
add a comment
4. Do qualitative comments relate to improved performance?
Best practice advises that qualitative comments are provided by raters to expand on their numerical rating scores but how accurate and beneficial are they, do they relate to greater improvements in performance?
Qualitative comments fill in the gaps left by numerical ratings alone and can be particularly beneficial when creating a development plan. They can provide specific examples of where an individual‟s current behaviour has worked well, as well as showing the consequences of when it has not.
“The real colour of a 360, the brilliant insights, are obtained from the qualitative comments.”
The ability of qualitative comments to provide these benefits will of course depend upon their quality; and it could be argued that really helpful qualitative feedback is hard to come by. There may be many reasons for this, such as inadequate guidance or lack of time but organisational culture may also be a key factor. If an individual is supported in, and rewarded for providing in depth feedback they are more likely to take the time to do so. “High quality and constructive feedback takes significant time and effort from the individual providing it – including moral courage and resilience. In today’s KPI driven and short term numbers focused environments, little time, support and recognition is given to those who make the effort to give vital feedback. There seems to be greater emphasis in reducing cost than there is on focusing on staff, and many business leaders find it difficult to relate time/effort spent on people development (especially performance management) to the ‘bottom line’ or indeed other more pressing financial targets.”
Even if constructive, specific comments are made the focus may reject them if they feel they are inaccurate or from a source they do not trust. In addition to rejection of feedback, there are alternative negative reactions that could follow from qualitative comments. An individual may become pre-occupied with identifying who the comments came from by analysing perceived „give aways‟ of a raters identity such as typical wording or spelling errors. An individual may also become pre-occupied by negative feedback. Each of these negative reactions will cause a hindrance in moving the individual through to acceptance. The negative impact of these reactions could be reduced by a number of measures discussed in this report, such as use of a professional coach, removal of qualitative comments for particular groups e.g. when showing the report to managers, provision of the feedback report prior to coaching session, or checking for overly personal or inappropriate comments prior to release of the report.
The consensus view seems to state that qualitative comments are beneficial as long as they are of a high quality and are constructive. Our view on qualitative comments at Talent Innovations is that more is better. They provide the context and the real learning points that help an individual understand their perceived behaviour. However we do feel it is important for an independent party to review comments prior to the feedback report being released. This both ensures that comments are constructive but also allows time to plan for particularly difficult feedback sessions – which in our experience are few and far between.
Regardless of the feedback method chosen by our client we offer the support of a trained career coach when needed and this offer is typically accepted for the most difficult of cases. Remember it is essential that all those who take part in a 360 benefit from the experience and follow through into acceptance of the data. Although infinitely valuable when approached correctly, if a 360 is not managed well the negative impact can be detrimental and long lasting.
Part 5 discusses whether a 360 feedback should provide surprises….
For the full paper, download it for free here.
Share this:
Like this: