Archive for the ‘Topic – Evaluation ’ Category

Purposeful Objectives

Wednesday, September 26th, 2012

For effective learning to take place, what is the most important part of the process? The accurate specification, the quality of the performance improver (aka trainer, coach), the attitude of the learner, the quality of the evaluation, or something else?

I’m not sure it’s possible to make a robust argument for one particular aspect being more important or influential than any other – they are all integral to bringing about effective performance improvement.

Having said that, I do think there is one part of the process that links all the parts together. And it is an area that is often not given the priority or importance it warrants. It is the ‘learning objectives’ or ‘learning outcomes’.

When I used to train trainers, and then assess the lessons they delivered, it was the single most problematic area. Where the trainers weren’t clear on their objectives, or their methodology didn’t meet the objective (usually because they had committed the cardinal sin of choosing a methodology before arriving at their objective) they often came unstuck. Where they had a well structured objective together with a matching methodology, they had clarity of purpose – and the sessions generally went so much more smoothly for all concerned.

Well-formed objectives assure the client that the specification of their needs is clearly understood, they inform the trainer what is to be covered and to what level, they let the delegates know what they are there for and they let the evaluator know what they need to measure.

An well written learning objective needs to achieve two key criteria. Firstly, it needs to say what is to be achieved by the delegates, and secondly it needs to say to what level.

This is where we need to thank Benjamin Bloom for his Cognitive Domain Taxonomy of Learning. He developed his theory in the 1950s, together with work on the Psychomotor and Affective Domains. A Taxonomy is an “ordered list” and the Cognitive Domain describes the various stages and levels a person passes through as they develop greater knowledge and learning around a specific area.

The left hand column of the table below describes the six levels of Bloom’s Taxonomy relating to the Cognitive Domain.

LevelPossible measurable
words
KnowledgeLabel; Identify; Recite; List; Name
ComprehensionExplain; Summarise; Illustrate; Give examples; Distinguish between
ApplicationOrganise; Apply; Produce; Show
AnalysisDifferentiate; Compare; Prioritise; Categorise; Classify
SynthesisCompose; Construct; Hypothesise; Design; Combine
EvaluationEvaluate; Make a judgement; Recommend

The right hand column gives some of the words that can be used in writing objectives to ensure that they are both measureable and pitched at the appropriate level. Let me give an example.

An organisation wants its staff to learn about its Performance Management processes. Some staff who are perhaps not included in the process may need only to have ‘Knowledge’ of it. Those who are appraised probably require the “Comprehension” level, and managers will need to have the “Application” level in order to carry out performance reviews. Other staff in HR may require
the higher levels within Bloom’s Taxonomy.

These first three levels should generate three different learning objectives which could be, for example, that by the end of the session the delegates are able to:

  1. Identify the organisation’s Performance Management processes (Knowledge)
  2. Summarise the organisation’s Performance Management processes (Comprehension)
  3. Apply the organisation’s Performance Management processes (Application)

These different objectives dictate the levels to be achieved, and therefore the methodologies to be used and consequently the duration of any learning intervention. To achieve the first objective could take 10 minutes, to achieve the third could take 2 days – a significant time and cost differential for a one word difference in the objective.

Moving finally to our evaluator. He or she can then ask delegates, either verbally or in writing, to “Summarise the organisation’s Performance Management processes” – and the answer will tell us whether the intervention has achieved its stated objective. When reported back, the client has clarity as to whether their requirement has been met, as does the trainer.

Paul

Management Gone Missing?

Monday, September 10th, 2012

I saw this comment on a learning and development forum recently – “I recently ran a management course for new and existing managers at similar levels, it is a course which all managers new to the organisation are told about and attend with agreement from their line manager. I had a number of managers who were negative and felt they shouldn’t have attended as they had a number of years’ experience as managers already. The other delegates were newer and did want to be there. I found I really struggled to turnaround / manage the really negative individual and the other few who weren’t happy to be there. Due to this I really feel I didn’t deliver the best course I could have done as, whilst I have experienced the odd 1 or 2 negative delegates in the past on various courses, I’ve never experienced such constant negativity which I didn’t seem to be able to have any influence over.”

Following the post were a number of suggestions from respondents, including:

  • Having individual discussions with delegates prior to their arrival
  • Encourage delegates to explain their concerns
  • If people don’t want to stay, don’t make them
  • Tailor the course more to their particular needs
  • Ask them what they want to get out of the course
  • Meet with their managers post-course

What surprised me, though, were that all the responses focused on what to do in the classroom environment, or post-course. In order to implement a long term solution, there needs to be wider and more systematic activity.

What can be done to ensure that such instances are minimised in the first place? What appear to be missing are effective learning and development management interventions.

For learning in the workplace to be effective, there needs to be a tripartite approach comprising the delegate, the delegate’s line management and the learning provider. And in terms of the learning provider, this means both the trainer and – importantly – their management.

Within the post there is no mention of 3 key activities or processes that I would expect to see.

1. Why is each delegate attending the course? They have been nominated by their manager – what are the reasons for the manager requiring them to attend? Managers having pre and post-course meetings with their attending delegates is one of the areas often examined during an Investor in People (IiP) assessment – because it makes a difference.

If this process doesn’t happen, it should. If it does happen, it should be recorded – and most     importantly be available to the trainer. This not only helps in terms of dealing with the issues    in the person’s post – by being able to explain to or remind them of why they are there – but it helps in terms of planning a more meaningful event for the delegates – and will generally mean an increased return on investment (ROI).

2. How might the use of a Student Charter or Learning Agreement help this situation? If such a process is in place, it means that the internal customer (the delegate and their line management) knows what to expect from the internal provider (the learning and development function). It should also include what the internal provider requires from its delegates – in terms of participation, input and behaviours – together with the consequences for a delegate who chooses not to meet this level of expectation.

Such an agreement should have been agreed across the business at senior management level – and so it is then easy for a trainer to use and implement without fear of having overstepped the mark. Transparency for all concerned.

And as a consequence, those who want to be there and want to learn can do so.

3. Finally, the person’s post on the forum indicates that this course has been delivered previously. What evaluation data has been generated?  And I don’t mean end of course or Level 1 data – I mean Level 3 or above, good quality data.  How is it improving the performance of those who are attending? How have previous delegates been able to use what they have learned? How is the course improving organisational performance?  What is the organisation’s return on its investment?

This information should be gathered from the delegates between 2 and 4 months after the completion of the programme. Both the quantitative and qualitative information can then be used by the trainer to illustrate what delegates can gain from attending – thus also demonstrating the overall value of the internal learning and development function to the business.

I have to say, I really feel for the trainer. I’ve been in that position and it’s not a lot of fun. Here is a trainer wanting to deliver a quality product, but some of the delegates are apparently thwarting his or her efforts.

And I get the impression that perhaps in this situation they are not been sufficiently supported by their learning and development management. The activities I have outlined above would both assist the trainer and probably make their role more enjoyable – and, above all, benefit the business.

Paul

Misleading Matters

Saturday, April 2nd, 2011

I was having a lovely email exchange with a person yesterday who has recently moved into learning and development management. She was explaining how she now needed to be more strategic, she has real plans for her company and she was so passionate about what she wanted to achieve. It was an inspiring exchange!

It started me thinking about if I was in such a position again. If I wanted to develop a culture that was as performance focused as possible and able to demonstrate how it added quantifiable value to the organisation, what are the words or phrases that would most hinder that culture shift, and what would I seek to replace them with? How could I demonstrate clarity of direction? Here are my top five:

Training Manager – that would be the first one to go. For me, the title signifies that the post-holder is purely focussed on what happens whilst a delegate is attending a course or undertaking an e-learning package. I’d want to be the Learning and Development Manager – or even the Performance Improvement Manager – thus signifying that I have a proactive interest in engaging with the managers and delegates pre and post event, and ascertaining the value of our contribution.

Abstraction – So often I used to be told at meetings that training was an abstraction, and the person making the judgment argued they couldn’t afford such an abstraction (in some companies the word ‘overhead’ is also used). I would immediately retort – sometimes interrupting the person, especially if they didn’t get the message the first time – that the learning and development that I managed and delivered was an investment (not an abstraction). If people get used to referring to your activities as an abstraction, the activities have the potential to become so. Take the opportunity to change the culture and make people think differently about what you are doing.  Eventually others will call it an investment, and then you know the culture can change and you will be viewed very differently.

Time Management Course – Why advertise that you are going to do something that is clearly impossible? Sailors don’t claim to manage the tides, and weather reporters don’t claim to manage the weather – we may wonder what planet they were coming from if they did. So what reasons do trainers and training managers have for claiming they can help people manage time?  We know what the tides will do, and can see and feel (and predict to some degree) the weather, so we manage ourselves accordingly. The same goes for time. Call such events Task Management or Task Prioritisation Courses – you will be surprised how differently people think about the event from the outset, because you are being clear about what will be achieved. Have a look at all your offerings – do they really do what they say on the tin?

Problem Person – I have often been faced with someone seeking assistance as they have a ‘Problem Person’ to deal with.  I bet Carl Rogers turns in his grave every time he hears this. If people are viewed in this way, they will potentially always be a problem. Such an individual is a person. A person who has a problem – a problem which you may well be able to assist them with. Where the manager thinks they have a ‘Problem Person’ they will more than likely become one.

Can you organise a (whatever) skills course, please? – The answer to this is ‘No’ – well not on this information, anyway. Operational managers are busy people and also may not be aware of all the ways that development needs can be met. You need to find out more – a lot more … how was the need identified, how many people does it apply to, how do we know it applies to all of them, why do all of them need it, what opportunities will there be for all these people to use the new skill, and so on. Get to the heart of the matter. You will receive so many ill-defined needs – and asking these questions (and others) will help generate a performance improvement partnership between your function and the rest of the organisation. And if you don’t ask the questions, and the learning and development intervention doesn’t work, the operational manager will make it public as to whose fault they think it is – and that will not do your culture shift (or reputation) much good!

So those are my top five – are there any that you would add?

Paul