Posts Tagged ‘measurement’

Purposeful Objectives

Wednesday, September 26th, 2012

For effective learning to take place, what is the most important part of the process? The accurate specification, the quality of the performance improver (aka trainer, coach), the attitude of the learner, the quality of the evaluation, or something else?

I’m not sure it’s possible to make a robust argument for one particular aspect being more important or influential than any other – they are all integral to bringing about effective performance improvement.

Having said that, I do think there is one part of the process that links all the parts together. And it is an area that is often not given the priority or importance it warrants. It is the ‘learning objectives’ or ‘learning outcomes’.

When I used to train trainers, and then assess the lessons they delivered, it was the single most problematic area. Where the trainers weren’t clear on their objectives, or their methodology didn’t meet the objective (usually because they had committed the cardinal sin of choosing a methodology before arriving at their objective) they often came unstuck. Where they had a well structured objective together with a matching methodology, they had clarity of purpose – and the sessions generally went so much more smoothly for all concerned.

Well-formed objectives assure the client that the specification of their needs is clearly understood, they inform the trainer what is to be covered and to what level, they let the delegates know what they are there for and they let the evaluator know what they need to measure.

An well written learning objective needs to achieve two key criteria. Firstly, it needs to say what is to be achieved by the delegates, and secondly it needs to say to what level.

This is where we need to thank Benjamin Bloom for his Cognitive Domain Taxonomy of Learning. He developed his theory in the 1950s, together with work on the Psychomotor and Affective Domains. A Taxonomy is an “ordered list” and the Cognitive Domain describes the various stages and levels a person passes through as they develop greater knowledge and learning around a specific area.

The left hand column of the table below describes the six levels of Bloom’s Taxonomy relating to the Cognitive Domain.

LevelPossible measurable
KnowledgeLabel; Identify; Recite; List; Name
ComprehensionExplain; Summarise; Illustrate; Give examples; Distinguish between
ApplicationOrganise; Apply; Produce; Show
AnalysisDifferentiate; Compare; Prioritise; Categorise; Classify
SynthesisCompose; Construct; Hypothesise; Design; Combine
EvaluationEvaluate; Make a judgement; Recommend

The right hand column gives some of the words that can be used in writing objectives to ensure that they are both measureable and pitched at the appropriate level. Let me give an example.

An organisation wants its staff to learn about its Performance Management processes. Some staff who are perhaps not included in the process may need only to have ‘Knowledge’ of it. Those who are appraised probably require the “Comprehension” level, and managers will need to have the “Application” level in order to carry out performance reviews. Other staff in HR may require
the higher levels within Bloom’s Taxonomy.

These first three levels should generate three different learning objectives which could be, for example, that by the end of the session the delegates are able to:

  1. Identify the organisation’s Performance Management processes (Knowledge)
  2. Summarise the organisation’s Performance Management processes (Comprehension)
  3. Apply the organisation’s Performance Management processes (Application)

These different objectives dictate the levels to be achieved, and therefore the methodologies to be used and consequently the duration of any learning intervention. To achieve the first objective could take 10 minutes, to achieve the third could take 2 days – a significant time and cost differential for a one word difference in the objective.

Moving finally to our evaluator. He or she can then ask delegates, either verbally or in writing, to “Summarise the organisation’s Performance Management processes” – and the answer will tell us whether the intervention has achieved its stated objective. When reported back, the client has clarity as to whether their requirement has been met, as does the trainer.


Creating a Strategy (2) – the Engine Room

Tuesday, May 1st, 2012

Having firmly laid the Foundation Stones for the Training for Performance Improvement (TPI) Strategy within sections 1 – 3, sections 4 and 5 deal with Standards of Performance and Responsibilities in delivering the Strategy. This is the engine room of the Strategy.

Section 4 – Standards of Performance (Success Criteria)

This section deals with what you are going to deliver, and how your unit’s performance will be assessed. You need to make the Standards both specific and measureable, yet also relevant for the 5-year lifespan of the Strategy.

This is where you really show whether you are a “Training” or a “Performance Improvement” unit or department. Are your standards of performance going to be “Output” focussed or “Outcome” focussed? Your ‘Scotch Corner’ moment, as I have previously described it.

Here are some examples of Standards of Performance that you could consider using:

  1. Number of days training and development invested in staff per month (i.e. a 5 day course for 12 people equates to 60 days investment)
  2. The % of managers undertaking at least one Leadership or Management Development Course each year
  3. Rating of internal interventions by delegates – % of delegates indicating that the learning event will be of direct benefit to them in their current role
  4. Rating of all interventions, 2 – 4 months after a learning event – % of staff indicating that they have improved their performance as a result of the intervention
  5. The % of managers stating that (your unit) has delivered an improved level of performance, as measured against the previous year

For some of these standards (such as 1 & 2) it may be that your target is the same throughout the 5 years, assuming you are not anticipating any significant changes in your organisation’s staffing levels. For others (such as 3 & 4) it may be that you want to increase the performance percentage for each year of the strategy’s lifecycle.

What would these Performance Standards indicate about the particular unit? If I saw statements 1 and 2 above as the style of Standards of Performance being used in a 5 year Strategy, my initial impression would be that this unit is very “Output” focussed and has little focus on improving performance. The unit is being measured in a relatively simplistic manner. It appears to be a “Training” function.

If, however, I saw statements such as 3, 4 and 5, I would get a very different first impression. These Standards are very “Outcome” focussed, seeking to demonstrate the value of the function to the rest of the organisation. The Standards are more difficult to measure and are more testing to achieve – in order to achieve them, they require more detailed communication and more effective relationships with other parts of the organisation. They indicate that this is a “Performance Improvement” function and that across the organisation there is a more mature culture.

Remember that what is measured generally happens. The measurements you select will become important. If you measure relatively unimportant activity, it will become important – and that will be detrimental to both you and your organisation.

Section 5 – Responsibilities

It is helpful if everyone is aware of their responsibilities in relation to the Strategy. The responsibilities need to be discussed, negotiated and agreed from the outset. Getting clarity and sign off at this stage will save you time in the future.

These are the roles I suggest that you have responsibilities listed for:

  • Staff
  • Line managers
  • Senior Operations managers
  • Members of (your unit)
  • Head of (your unit)
  • Head of (overall part of the organisation within which your unit sits)
  • Senior Leadership members
  • Chief Executive / Head of the Organisation
  • Board members (where appropriate)

Here are a couple of examples of what you might list under two of the above groups.

All members of Staff are expected to:

  • Take responsibility for their own learning and development
  • Learn from their workplace experiences
  • Identify, through the PDR process, and address, with the assistance of line managers and (your unit), their particular learning and
    development needs
  • Make use of self-help facilities
  • Keep up to date with the organisation’s policy, practices and procedures

Senior Leadership members are expected to:

  • Set and review the strategic direction of the organisation
  • Set an example by being effective in managing staff
  • Value and action their own training and development needs, and those who they line manage
  • Monitor and quantify the improved performance delivered as a result of staff undertaking learning and development activities

If you would like examples of the responsibilities that you might list under the other suggested roles, just let me know – I will be happy to send you some.

The third and final part of the TPI Strategy template will be the focus for my next post.


Misleading Matters

Saturday, April 2nd, 2011

I was having a lovely email exchange with a person yesterday who has recently moved into learning and development management. She was explaining how she now needed to be more strategic, she has real plans for her company and she was so passionate about what she wanted to achieve. It was an inspiring exchange!

It started me thinking about if I was in such a position again. If I wanted to develop a culture that was as performance focused as possible and able to demonstrate how it added quantifiable value to the organisation, what are the words or phrases that would most hinder that culture shift, and what would I seek to replace them with? How could I demonstrate clarity of direction? Here are my top five:

Training Manager – that would be the first one to go. For me, the title signifies that the post-holder is purely focussed on what happens whilst a delegate is attending a course or undertaking an e-learning package. I’d want to be the Learning and Development Manager – or even the Performance Improvement Manager – thus signifying that I have a proactive interest in engaging with the managers and delegates pre and post event, and ascertaining the value of our contribution.

Abstraction – So often I used to be told at meetings that training was an abstraction, and the person making the judgment argued they couldn’t afford such an abstraction (in some companies the word ‘overhead’ is also used). I would immediately retort – sometimes interrupting the person, especially if they didn’t get the message the first time – that the learning and development that I managed and delivered was an investment (not an abstraction). If people get used to referring to your activities as an abstraction, the activities have the potential to become so. Take the opportunity to change the culture and make people think differently about what you are doing.  Eventually others will call it an investment, and then you know the culture can change and you will be viewed very differently.

Time Management Course – Why advertise that you are going to do something that is clearly impossible? Sailors don’t claim to manage the tides, and weather reporters don’t claim to manage the weather – we may wonder what planet they were coming from if they did. So what reasons do trainers and training managers have for claiming they can help people manage time?  We know what the tides will do, and can see and feel (and predict to some degree) the weather, so we manage ourselves accordingly. The same goes for time. Call such events Task Management or Task Prioritisation Courses – you will be surprised how differently people think about the event from the outset, because you are being clear about what will be achieved. Have a look at all your offerings – do they really do what they say on the tin?

Problem Person – I have often been faced with someone seeking assistance as they have a ‘Problem Person’ to deal with.  I bet Carl Rogers turns in his grave every time he hears this. If people are viewed in this way, they will potentially always be a problem. Such an individual is a person. A person who has a problem – a problem which you may well be able to assist them with. Where the manager thinks they have a ‘Problem Person’ they will more than likely become one.

Can you organise a (whatever) skills course, please? – The answer to this is ‘No’ – well not on this information, anyway. Operational managers are busy people and also may not be aware of all the ways that development needs can be met. You need to find out more – a lot more … how was the need identified, how many people does it apply to, how do we know it applies to all of them, why do all of them need it, what opportunities will there be for all these people to use the new skill, and so on. Get to the heart of the matter. You will receive so many ill-defined needs – and asking these questions (and others) will help generate a performance improvement partnership between your function and the rest of the organisation. And if you don’t ask the questions, and the learning and development intervention doesn’t work, the operational manager will make it public as to whose fault they think it is – and that will not do your culture shift (or reputation) much good!

So those are my top five – are there any that you would add?


Who’s giving training a bad name?

Wednesday, March 9th, 2011

I sometimes wonder how people are appointed to positions as trainers. Do some organisations see it as a position that people are moved if they might be good at it, and are then left to get on with it – without any training or support? Do businesses think training is such a waste of time that they just employ anyone in the position? Whatever the reason, there appear to be too many companies or trainers giving training a bad name.

I see requests for assistance from trainers on various forums, but I’m not sure how they expect people to be able to respond with targeted and focussed suggestions. I’m meaning where people are asking others to suggest methodologies. Here are a couple I have recently seen:

“I have a four-hour workshop to give on presentations skills. I see the same participants the following week and they have to give a five-minute using the information they’ve learned in just four hours. Any recommendations? No technology available for them except for one laptop. They do not have laptops themselves.”

“I work in a manufacturing facility and I am about to conduct management development training for supervisors/managers to teach them more about ‘HR in their jobs’.   Does anyone have any good icebreakers that will compliment this training?”

Do these same people (or organisations) say, “I’m going to buy a car, which one would you recommend?”, or “I’m going on holiday and need a coat – which do you think I should buy?” I think not. So why do it for training?

Trainers asking such questions cannot expect to receive any sort of meaningful advice or assistance. And if they do use what is offered, does the organisation realise how much money it is potentially wasting?  

Let’s say we have 12 people in our group, and each receives a salary of £100 per day. That means with salary on-costs they probably cost their organisation £120 per day. That’s £1,440 in wages for attending a day’s training. Add another £200 for the trainer’s delivery and preparation time, £100 for the administration of getting the people on to the course, £50 for room hire and £10 for photocopying costs – that gives us a total cost of £1,800 for the day. There may also be travel and meal costs, but we’ll leave it at this for the purpose of the exercise.

If we have 6.5 hours contact time, it means that each hour of delivery is costing our organisation approximately £277.00. Or £4.62 per minute.  So a half hour opening exercise comes in at £138.

And if it costs £138 it needs to be focussed – not focussed on the ideas of others who have no knowledge of the organisation, but focussed on operational effectiveness, focussed or organisational goals, focussed on delegate needs and focussed on extracting a return on the investment. Effective evaluation.

As effective trainers know, evaluation starts at the planning stage. An evaluation of the needs of the attendees and the organisation. Once a Training Needs Analysis of some sort has been undertaken, planning, validation, delivery and further evaluation follow. This evaluation can show the organisational and operational improvements that have been achieved as a result of the training. Very satisfying, and likely to lead to the increased credibility of the trainers and L&D function.

Training people effectively is costly. Training people ineffectively is even more costly as we reduce (or fail to get any) the return on the investment.

Or have I got it all wrong?


Sell your crunch, not your apples

Saturday, December 4th, 2010

Where does the L & D function fit within your organisation? The importance of this cultural fit cannot be underestimated when leading a L & D function. In my experience, too many L & D functions have existed to deliver training – that is what they have been seen to do, and that is what they have done – and this approach gives them little credence with operational managers.

I once had concerns about the impressions that my unit was giving to the remainder of the business. At a managers’ meeting I took the opportunity to ask a few (what I thought to be) fundamental questions to see whether they were espousing my stated aims for the unit. They all managed trainers, so firstly I asked what they saw to be the overall role of the trainer. The first answer was, “To deliver training”. This was closely followed by answers indicating that they needed to be able to write lesson plans and that they should be able to explain points clearly to delegates. Two more responses followed in a similar vein. The sixth response was, “They need to get the delegates to be more effective”. That’s what I was looking for – a link to improving performance.

After a few other responses, I then asked, “What are trainer shoes for?” The first response was, “to help people run faster”. Other verbal offerings built on this performance related theme.

I then asked what they thought was the overall role of a horse trainer. The first respondent indicated that her view was that they were there to enable horses to win races. A second added that perhaps it was to enable both horse and rider to win races.

“So”, I asked, in a John Humphreys sort of way, “Why is it that you view trainer shoes and horse trainers as being there to improve performance, yet you see that the primary purpose of our trainers is to deliver training?”. This was followed by a silence.

This short exchange had confirmed my concerns and highlighted a fundamental issue in leading a successful Learning and Development Unit. If its own staff see the focus as being delivering training, this will be apparent to others in everything they do – most of all to the already potentially sceptical operational managers.

I personally would consider changing the job title of “Trainer” to “Performance Improver”, however, – at least initially – I think this would cause problems for others to understand what the role is about. It is fundamental, however, that the trainer views their role in this way. They should not see themselves as people with training skills who use these skills to improve performance; they need to see themselves as performance improvers who achieve this by using sound training skills. This may appear to be a play on words, but I don’t believe this to be the case – it is a way of the person constantly being reminded of the primary purpose of their role.

When I used to train trainers, we engaged in some lengthy discussions which felt very important and fundamental to the role – in retrospect, I am now disappointed to admit that they were akin to navel gazing – they were all about the minutiae of training delivery which, on reflection, had little impact on delivering greater operational performance.

It is only be demonstrating this focus on improving performance that you will earn a credible and respect place around the table when the all important annual budget is being allocated – and then the more improvements you can show to have been involved in, the better your chance will be of improving your share of the budget.

All this can be achieved by a number of key activities. These include:

  • Engaging your internal customers in deciding learning priorities
  • Explicitly linking all learning to business objectives
  • Explicitly linking all learning to improving performance
  • Measuring your team’s performance in terms of operational performance improvements rather than / in addition to the quality of the training delivered
  • Clearly communicating your successes to the business
  • Where appropriate, measuring your impact by undertaking Return on Investment (ROI) evaluation activity to demonstrate the financial return for the business’s investment.

The overall culture of a unit is probably the most difficult aspect to change – but ultimately the most fundamental and rewarding. By implementing these processes, we can start to change the mindset of our team. We find out who wants to make the journey with us, and who perhaps does not.


Deliver Outcomes, not Outputs

Friday, November 12th, 2010

Where at all possible, I try and steer clear of discussing governmental politics. This is because I don’t really want to be perceived to have particular party allegiances. But rules, as with policies and directions from Sat-navs, are there to be broken where there is good reason – they exist to provide effective guidance for the majority of situations as opposed to the answer for all situations.

I am very interested by the Business Plans that the ruling coalition has published this week (I feel like a very sad person as I write that, but please bear with me …).  It’s the approach behind some of the Plans that I find particularly interesting.

When the coalition first came to power, they talked about financially rewarding private contractors for ‘housing’ prisoners and stopping them reoffending – whereas currently they are paid for merely ‘housing’ them for the duration of their sentences.  As someone who is very keen on the use of outcomes rather than outputs, I found this very refreshing.

More detail on this approach has now been published – it is within the Ministry of Justice Business Plan which was published last Monday. The first two objectives within that particular Business Plan are:

1.1 Develop an overall strategy for the ‘rehabilitation revolution’ for adults and youths, including paying local private and voluntary organisations by results.

1.2 Introduce payment by results schemes, working with local, voluntary and private sector organisations that specialise in the rehabilitation of offenders.

I am interested in this approach as I believe that society as a whole should move more towards an outcome focussed approach to business. This is particularly true of learning and development providers – and clients.

If a client seeks and purchases a course, event or programme to address a particular area or aspect of business, it is likely that it will be delivered and there may well be a short end of event questionnaire. It is rare, in my experience, that there is much further evaluation of the benefits of the event.

So, how about the client publicises the issues or skills they require addressing on the event and these are then quantified into measurable outcomes (not outputs).  The provider who then gains the contract only gets paid once these targets have been achieved – perhaps 6 months after the event. The probable consequences would include:

  • A greater focus on outcomes as opposed to outputs.
  • Only providers who have confidence in their abilities bidding for the contract.
  • Clients ensuring and demonstrating value for their money.
  • Far greater interest by clients, providers and delegates’ line managers as to how the learning is put into practice in the workplace.

Who wouldn’t want these consequences?

So why does this approach generally not happen at the moment?


Organisational maturity

Sunday, October 3rd, 2010

As we leave September and the nights grow longer here in the UK, many organisations are awaiting the Spending Review due to be published on 20 October. I was at a Business Link event last week and this was one of the main topics of conversations. There is understandably much concern and trepidation around due to the potential impact of the spending cuts.

As you will have seen from the news, many public sector organisations are already starting to make cuts in spending and people resources due to their reduction in finances. Many private sector organisations have been undertaking this for some months already.

So what are you doing to start working more efficiently and effectively? We will need to do this in order to soften the blow of any cuts. I also believe that we all have a responsibility to do this. And don’t underestimate how much you can save. A 2007 survey by the Proudfoot Consulting (Guardian 22 October 2007) covering 2,500 businesses over four years and across 38 countries, demonstrated that the cost of wasted time to UK businesses is £80bn per year, equivalent to 7% of its Gross Domestic Product. The ‘worst offender’ causes of inefficiency were found to be:

  • Inadequate workforce supervision (31%)
  • Poor management planning (30%)
  • Poor communication (18%)
  • IT problems, low morale, and lack or mismatch of skills (21%)

Recently I delivered a Leadership and Management Programme within an organisation. It took place on a number of days over a period of 4 months. It was great fun and I was a little sad when it ended as I had developed some lovely relationships with many of the very keen and talented delegates.

At the end of the Programme, the delegates delivered presentations as to what they had learning and, most importantly, how it was making a difference in their workplaces. Where possible, I assisted them to convert their achievements into cash or efficiency savings.  

One person explained how they had stopped staff from coming to the line managers and asking questions (unless business critical) and as an alternative the managers held ‘surgeries’ for such questions three times a day. This saved on the time of all concerned and allowed for more uninterrupted work. A by-product of this was that staff then started to sort out their own minor problems, thus saving even more time. Over the period of a year, based on the finding of the first month, they were due to make over £15,000 of efficiency savings due to the time they were saving.

One saving I have always been keen to make is through not having ‘Any Other Business’ on meeting agendas. If you have 10 people attending a meeting, and all work for £16 per hour (including on-costs), every 15 minutes of the meeting costs the organisation 2.5 hours staff time or £40. By cutting the AOB section, it means that people have to plan more effectively for what they want in the meeting, other delegates can plan for the item and the Chair can make the meeting more focussed.

And all these activities have knock-on effects that can help to develop a more mature organisational culture. The surgeries example has led to staff sorting out their own issues and therefore thinking for themselves more effectively, which will create a more Adult and self-sufficient environment. Losing the AOB means that staff start to plan more effectively, and this skill will then be transferred elsewhere. So, the increased efficiency leads to increased effectiveness.

So, as I said earlier, what are you doing to work more efficiently and effectively? Or, more pointedly, what changes will you introduce this week?

Please let me know – I’d be really interested to hear other ideas.


Is ‘e hurt, man?

Thursday, June 24th, 2010

Well, it certainly looked like he was hurting to me. He looked like he could hardly stand up, never mind play tennis. I’m referring to John Isner the American tennis player, one half of the record breaking duo (the other being France’s Nicolas Mahut) whose final set at Wimbledon yesterday went to 59-59 in games before bad light forced a break in proceedings. And it has already broken so many records – longest match ever (10 hours) most aces in a match (193) – and there are more, but I’ll stop there.  All those record breaking outputs, but they will lead to only one outcome (hopefully later today) which will be one of them winning the match. And that is what will be recorded in the competition.

As a trainer, I know that I have been too interested in outputs in the past, and I now see trainers who are too focussed on them. I understand that it can be difficult to see beyond the session or course itself as we care passionately about what we do and get very involved in it. I know it’s great to come up with an inspirational methodology, a role play that explores the affective domain or a great set of PowerPoint slides. But these are all outputs. We need to focus on the outcome. By increasing our focus on the outcome, the methodology becomes more meaningful and business orientated.

When one of the players was interviewed when leaving the court yesterday evening he said, “I’ll be interested to see the stats.” Again, as trainers, we are interested in “the stats” we usually get at the conclusion of any session or course – normally provided through some sort of evaluation questionnaire. We need to move beyond this, too.

What we should really be interested in is our delegates’ (and their managers) responses to the statement, “As a result of this learning event, I have (they have) improved my (their) performance within the workplace.” Followed by, “This is evidenced by …”. If this question is posed to delegates 3 months after the event, I would expect effective delivery to achieve at least 85% agreeing with this statement. Anything less and there are potentially issues with the event, the people being nominated for it or the post-event support delegates are receiving in the workplace. That is a true outcome, one which is meaningful to the organisation and one which enables the trainer to really demonstrate their worth. It also starts the process of examining and quantifying the return on the investment for the event (ROI).  Essential in the current economic climate.

How can you make your events or your trainers’ events more outcome focussed? What can managers do to ensure that their events are delivering meaningful outcomes? I’d be interested in your views – or the views of delegates.

Oh, and the title of this post? It’s an anagram of the two players’ names. Well, you have to find something to do when watching a 118 game final set of a tennis match …