Posts Tagged ‘evaluation’

Purposeful Objectives

Wednesday, September 26th, 2012

For effective learning to take place, what is the most important part of the process? The accurate specification, the quality of the performance improver (aka trainer, coach), the attitude of the learner, the quality of the evaluation, or something else?

I’m not sure it’s possible to make a robust argument for one particular aspect being more important or influential than any other – they are all integral to bringing about effective performance improvement.

Having said that, I do think there is one part of the process that links all the parts together. And it is an area that is often not given the priority or importance it warrants. It is the ‘learning objectives’ or ‘learning outcomes’.

When I used to train trainers, and then assess the lessons they delivered, it was the single most problematic area. Where the trainers weren’t clear on their objectives, or their methodology didn’t meet the objective (usually because they had committed the cardinal sin of choosing a methodology before arriving at their objective) they often came unstuck. Where they had a well structured objective together with a matching methodology, they had clarity of purpose – and the sessions generally went so much more smoothly for all concerned.

Well-formed objectives assure the client that the specification of their needs is clearly understood, they inform the trainer what is to be covered and to what level, they let the delegates know what they are there for and they let the evaluator know what they need to measure.

An well written learning objective needs to achieve two key criteria. Firstly, it needs to say what is to be achieved by the delegates, and secondly it needs to say to what level.

This is where we need to thank Benjamin Bloom for his Cognitive Domain Taxonomy of Learning. He developed his theory in the 1950s, together with work on the Psychomotor and Affective Domains. A Taxonomy is an “ordered list” and the Cognitive Domain describes the various stages and levels a person passes through as they develop greater knowledge and learning around a specific area.

The left hand column of the table below describes the six levels of Bloom’s Taxonomy relating to the Cognitive Domain.

LevelPossible measurable
KnowledgeLabel; Identify; Recite; List; Name
ComprehensionExplain; Summarise; Illustrate; Give examples; Distinguish between
ApplicationOrganise; Apply; Produce; Show
AnalysisDifferentiate; Compare; Prioritise; Categorise; Classify
SynthesisCompose; Construct; Hypothesise; Design; Combine
EvaluationEvaluate; Make a judgement; Recommend

The right hand column gives some of the words that can be used in writing objectives to ensure that they are both measureable and pitched at the appropriate level. Let me give an example.

An organisation wants its staff to learn about its Performance Management processes. Some staff who are perhaps not included in the process may need only to have ‘Knowledge’ of it. Those who are appraised probably require the “Comprehension” level, and managers will need to have the “Application” level in order to carry out performance reviews. Other staff in HR may require
the higher levels within Bloom’s Taxonomy.

These first three levels should generate three different learning objectives which could be, for example, that by the end of the session the delegates are able to:

  1. Identify the organisation’s Performance Management processes (Knowledge)
  2. Summarise the organisation’s Performance Management processes (Comprehension)
  3. Apply the organisation’s Performance Management processes (Application)

These different objectives dictate the levels to be achieved, and therefore the methodologies to be used and consequently the duration of any learning intervention. To achieve the first objective could take 10 minutes, to achieve the third could take 2 days – a significant time and cost differential for a one word difference in the objective.

Moving finally to our evaluator. He or she can then ask delegates, either verbally or in writing, to “Summarise the organisation’s Performance Management processes” – and the answer will tell us whether the intervention has achieved its stated objective. When reported back, the client has clarity as to whether their requirement has been met, as does the trainer.


So tell me what you need, what you really, really need

Monday, April 25th, 2011

I’m all for the phrase “Make it happen”, but we always have to be clear on the ‘it’. Sometimes organisations are so keen to make ‘it’ happen, the priority can become “Make something happen” which can often be one step forward and two steps back. It can also damage relationships and cause waves in previously tranquil waters. And it can waste serious amounts of money.

The area of learning and development is one of the best – or worst, dependent upon how you look at it – examples of this.

I can think of several occasions where training interventions have been delivered and they have not delivered the required results in the workplace. In several of those instances the claim has been that it was “Training’s fault because they didn’t train the right things”. Hmmm. Perhaps it was, or perhaps it wasn’t. It is, however, relatively simple to introduce a process that minimises this potential and maximises the value from an organisation’s investment in learning. As with so many successful projects, the primary requirement is for the relevant people to take the required responsibility and accountability.

Learning and development professionals (the contractors) need to slow down the commissioning process. The commissioning process is the stage where the internal or external client identifies what needs to change. And this will only work if the responsibility is in the right place.

The client has to take responsibility for job descriptions, core skills and identifying individuals’ learning and development needs. The contractor’s role is to analyse the client’s needs and develop appropriate and cost-effective interventions, deliver the interventions, assess the competence of delegates and check the intervention met its stated objectives. Finally, we arrive at the evaluation of the changes in the workplace as a result of the intervention – which is the client’s responsibility.

The start of this cycle used to be called the Training Needs Analysis (TNA). However, it is increasing being broken down into two activities called the Performance Needs Analysis (PNA) and the Learning Needs Analysis (LNA). Whilst it is the responsibility of the client, the contractor needs to assist them in this process. When we are closely involved in matters, we often lose our objectivity and sometimes make assumptions – this is what can happen for a client and so we need to help them avoid this by taking them through a structure process.

This process can be as detailed as everyone wants to make it. Having said that, if learning and development professionals use too complex a process, they will lose credibility with the client. The complexity of the process should also be driven by the size of the audience – in other words, if 20 people need to improve their performance in a particular area, the process should be less arduous than if 2,000 people require it – but both require a process to ensure we make the right thing happen.

As a start, I use seven questions. These are:

  1. What is the identified performance need?
  2. Why is a solution required (how will it add value, and what would happen if it did not exist)?
  3. How will the solution improve the organisation’s performance against its identified goals?
  4. Which organisational competencies does the solution aim to address and to what level?
  5. What are the target staff groups /  teams / Departments for this solution?
  6. How will the effectiveness of the solution be measured in the workplace?
  7. What are the clearly stated, measurable objectives for the solution?

From experience, I know that these questions can come across as threatening to clients. This is particularly the case in immature organisations – in other words, organisations where discussions around learning and development activities are often packaged in terms of the client stating they want a course on a particular aspect, rather than rational, objective and meaningful discussions between the client and contractor on what they need to achieve together.  Consequently, my preferred approach is to let the client have the questions prior to us meeting face to face or us having a further phone conversation. This gives them the opportunity to identify what they need – what they really need.

When learning and development professionals get this process right, they are well on their way to making a meaningful impact for the organisation. They will be able to demonstrate their value in improving performance.

In my next post, I will explain a little more about each of these seven star questions – and why, when you have completed the process, you need to get your sponsor’s sign-off.  


Misleading Matters

Saturday, April 2nd, 2011

I was having a lovely email exchange with a person yesterday who has recently moved into learning and development management. She was explaining how she now needed to be more strategic, she has real plans for her company and she was so passionate about what she wanted to achieve. It was an inspiring exchange!

It started me thinking about if I was in such a position again. If I wanted to develop a culture that was as performance focused as possible and able to demonstrate how it added quantifiable value to the organisation, what are the words or phrases that would most hinder that culture shift, and what would I seek to replace them with? How could I demonstrate clarity of direction? Here are my top five:

Training Manager – that would be the first one to go. For me, the title signifies that the post-holder is purely focussed on what happens whilst a delegate is attending a course or undertaking an e-learning package. I’d want to be the Learning and Development Manager – or even the Performance Improvement Manager – thus signifying that I have a proactive interest in engaging with the managers and delegates pre and post event, and ascertaining the value of our contribution.

Abstraction – So often I used to be told at meetings that training was an abstraction, and the person making the judgment argued they couldn’t afford such an abstraction (in some companies the word ‘overhead’ is also used). I would immediately retort – sometimes interrupting the person, especially if they didn’t get the message the first time – that the learning and development that I managed and delivered was an investment (not an abstraction). If people get used to referring to your activities as an abstraction, the activities have the potential to become so. Take the opportunity to change the culture and make people think differently about what you are doing.  Eventually others will call it an investment, and then you know the culture can change and you will be viewed very differently.

Time Management Course – Why advertise that you are going to do something that is clearly impossible? Sailors don’t claim to manage the tides, and weather reporters don’t claim to manage the weather – we may wonder what planet they were coming from if they did. So what reasons do trainers and training managers have for claiming they can help people manage time?  We know what the tides will do, and can see and feel (and predict to some degree) the weather, so we manage ourselves accordingly. The same goes for time. Call such events Task Management or Task Prioritisation Courses – you will be surprised how differently people think about the event from the outset, because you are being clear about what will be achieved. Have a look at all your offerings – do they really do what they say on the tin?

Problem Person – I have often been faced with someone seeking assistance as they have a ‘Problem Person’ to deal with.  I bet Carl Rogers turns in his grave every time he hears this. If people are viewed in this way, they will potentially always be a problem. Such an individual is a person. A person who has a problem – a problem which you may well be able to assist them with. Where the manager thinks they have a ‘Problem Person’ they will more than likely become one.

Can you organise a (whatever) skills course, please? – The answer to this is ‘No’ – well not on this information, anyway. Operational managers are busy people and also may not be aware of all the ways that development needs can be met. You need to find out more – a lot more … how was the need identified, how many people does it apply to, how do we know it applies to all of them, why do all of them need it, what opportunities will there be for all these people to use the new skill, and so on. Get to the heart of the matter. You will receive so many ill-defined needs – and asking these questions (and others) will help generate a performance improvement partnership between your function and the rest of the organisation. And if you don’t ask the questions, and the learning and development intervention doesn’t work, the operational manager will make it public as to whose fault they think it is – and that will not do your culture shift (or reputation) much good!

So those are my top five – are there any that you would add?


Can training ever be exceptional?

Sunday, March 27th, 2011

I saw a question on a forum yesterday that got me thinking. The question was, “I’m interested to know how folk define ‘exceptional training’ – what makes training outstanding, and sets it apart from an everyday training course?”

As a trainer, what have I delivered that has been exceptional? I can think of many sessions that I have been pleased with, and that have gone better than others that I have delivered. I can also think of ones that I have thought have been very powerful in the learning that has been generated for the delegates. But I don’t think any of that necessarily makes them ‘exceptional’.

Then I considered what training I had attended that had been exceptional. What came to mind was not training that had been exceptional, but exceptional learning that I had taken from sessions or courses – and the two, I think, are completely different.

For training to be exceptional (if, in fact, it can be) it will depend upon how the delegates respond to it – and not necessarily whilst they are experiencing it, but how they respond back in their workplaces and lives in the weeks, months and years to come.

For example, I was fortunate to be able to attend the Work Foundation’s Runge Effective Leadership Programme about 10 years ago. There was a session on how to create a Vision for a Company or Department. I cannot remember the name or much about the man who delivered the session. It wasn’t a particularly innovative or interactive session, and at the end of the one-week Programme it wasn’t in my ‘top three’ of the sessions I had attended. When I look back on it, however, I realise how valuable it was to me and how much I have used it since. It has been a foundation, a starting point, for many of my successes since then. But I really couldn’t class it as exceptional training.

Which supports my view that only delegates can really say whether such experiences are exceptional, and they can only really do that further down the line. Perhaps there can be ‘exceptional learning’?

So as a starting point in trying to answer the original question how about this – exceptional learning occurs when 100% of the full cost of the learning event is recovered within 12 months of the event – in other words, the person or company acquires a full return on its investment (ROI) within that time frame. I think that could then be classed as ‘exceptional’.

The first step in seeking to achieve this is for trainers to stop thinking of themselves as trainers, but to think of themselves as Performance Improvers – who can play their part in achieving such improvements by effective training methodologies. By reframing their role, their thought processes are likely to have a greater focus on how they create exceptional performance improvements – which then brings the realisation that it is as much about the part the delegates play as the trainer.

It includes getting the right people on the right events in the first place, and motivating and developing them once they get back to the workplace.

The trainer – or Performance Improver – plays their important part in such achievements, but others also have very important parts. And all must play their parts well to achieve exceptional learning.

Unless, of course, you think differently?


Who’s giving training a bad name?

Wednesday, March 9th, 2011

I sometimes wonder how people are appointed to positions as trainers. Do some organisations see it as a position that people are moved if they might be good at it, and are then left to get on with it – without any training or support? Do businesses think training is such a waste of time that they just employ anyone in the position? Whatever the reason, there appear to be too many companies or trainers giving training a bad name.

I see requests for assistance from trainers on various forums, but I’m not sure how they expect people to be able to respond with targeted and focussed suggestions. I’m meaning where people are asking others to suggest methodologies. Here are a couple I have recently seen:

“I have a four-hour workshop to give on presentations skills. I see the same participants the following week and they have to give a five-minute using the information they’ve learned in just four hours. Any recommendations? No technology available for them except for one laptop. They do not have laptops themselves.”

“I work in a manufacturing facility and I am about to conduct management development training for supervisors/managers to teach them more about ‘HR in their jobs’.   Does anyone have any good icebreakers that will compliment this training?”

Do these same people (or organisations) say, “I’m going to buy a car, which one would you recommend?”, or “I’m going on holiday and need a coat – which do you think I should buy?” I think not. So why do it for training?

Trainers asking such questions cannot expect to receive any sort of meaningful advice or assistance. And if they do use what is offered, does the organisation realise how much money it is potentially wasting?  

Let’s say we have 12 people in our group, and each receives a salary of £100 per day. That means with salary on-costs they probably cost their organisation £120 per day. That’s £1,440 in wages for attending a day’s training. Add another £200 for the trainer’s delivery and preparation time, £100 for the administration of getting the people on to the course, £50 for room hire and £10 for photocopying costs – that gives us a total cost of £1,800 for the day. There may also be travel and meal costs, but we’ll leave it at this for the purpose of the exercise.

If we have 6.5 hours contact time, it means that each hour of delivery is costing our organisation approximately £277.00. Or £4.62 per minute.  So a half hour opening exercise comes in at £138.

And if it costs £138 it needs to be focussed – not focussed on the ideas of others who have no knowledge of the organisation, but focussed on operational effectiveness, focussed or organisational goals, focussed on delegate needs and focussed on extracting a return on the investment. Effective evaluation.

As effective trainers know, evaluation starts at the planning stage. An evaluation of the needs of the attendees and the organisation. Once a Training Needs Analysis of some sort has been undertaken, planning, validation, delivery and further evaluation follow. This evaluation can show the organisational and operational improvements that have been achieved as a result of the training. Very satisfying, and likely to lead to the increased credibility of the trainers and L&D function.

Training people effectively is costly. Training people ineffectively is even more costly as we reduce (or fail to get any) the return on the investment.

Or have I got it all wrong?


Sell your crunch, not your apples

Saturday, December 4th, 2010

Where does the L & D function fit within your organisation? The importance of this cultural fit cannot be underestimated when leading a L & D function. In my experience, too many L & D functions have existed to deliver training – that is what they have been seen to do, and that is what they have done – and this approach gives them little credence with operational managers.

I once had concerns about the impressions that my unit was giving to the remainder of the business. At a managers’ meeting I took the opportunity to ask a few (what I thought to be) fundamental questions to see whether they were espousing my stated aims for the unit. They all managed trainers, so firstly I asked what they saw to be the overall role of the trainer. The first answer was, “To deliver training”. This was closely followed by answers indicating that they needed to be able to write lesson plans and that they should be able to explain points clearly to delegates. Two more responses followed in a similar vein. The sixth response was, “They need to get the delegates to be more effective”. That’s what I was looking for – a link to improving performance.

After a few other responses, I then asked, “What are trainer shoes for?” The first response was, “to help people run faster”. Other verbal offerings built on this performance related theme.

I then asked what they thought was the overall role of a horse trainer. The first respondent indicated that her view was that they were there to enable horses to win races. A second added that perhaps it was to enable both horse and rider to win races.

“So”, I asked, in a John Humphreys sort of way, “Why is it that you view trainer shoes and horse trainers as being there to improve performance, yet you see that the primary purpose of our trainers is to deliver training?”. This was followed by a silence.

This short exchange had confirmed my concerns and highlighted a fundamental issue in leading a successful Learning and Development Unit. If its own staff see the focus as being delivering training, this will be apparent to others in everything they do – most of all to the already potentially sceptical operational managers.

I personally would consider changing the job title of “Trainer” to “Performance Improver”, however, – at least initially – I think this would cause problems for others to understand what the role is about. It is fundamental, however, that the trainer views their role in this way. They should not see themselves as people with training skills who use these skills to improve performance; they need to see themselves as performance improvers who achieve this by using sound training skills. This may appear to be a play on words, but I don’t believe this to be the case – it is a way of the person constantly being reminded of the primary purpose of their role.

When I used to train trainers, we engaged in some lengthy discussions which felt very important and fundamental to the role – in retrospect, I am now disappointed to admit that they were akin to navel gazing – they were all about the minutiae of training delivery which, on reflection, had little impact on delivering greater operational performance.

It is only be demonstrating this focus on improving performance that you will earn a credible and respect place around the table when the all important annual budget is being allocated – and then the more improvements you can show to have been involved in, the better your chance will be of improving your share of the budget.

All this can be achieved by a number of key activities. These include:

  • Engaging your internal customers in deciding learning priorities
  • Explicitly linking all learning to business objectives
  • Explicitly linking all learning to improving performance
  • Measuring your team’s performance in terms of operational performance improvements rather than / in addition to the quality of the training delivered
  • Clearly communicating your successes to the business
  • Where appropriate, measuring your impact by undertaking Return on Investment (ROI) evaluation activity to demonstrate the financial return for the business’s investment.

The overall culture of a unit is probably the most difficult aspect to change – but ultimately the most fundamental and rewarding. By implementing these processes, we can start to change the mindset of our team. We find out who wants to make the journey with us, and who perhaps does not.


Deliver Outcomes, not Outputs

Friday, November 12th, 2010

Where at all possible, I try and steer clear of discussing governmental politics. This is because I don’t really want to be perceived to have particular party allegiances. But rules, as with policies and directions from Sat-navs, are there to be broken where there is good reason – they exist to provide effective guidance for the majority of situations as opposed to the answer for all situations.

I am very interested by the Business Plans that the ruling coalition has published this week (I feel like a very sad person as I write that, but please bear with me …).  It’s the approach behind some of the Plans that I find particularly interesting.

When the coalition first came to power, they talked about financially rewarding private contractors for ‘housing’ prisoners and stopping them reoffending – whereas currently they are paid for merely ‘housing’ them for the duration of their sentences.  As someone who is very keen on the use of outcomes rather than outputs, I found this very refreshing.

More detail on this approach has now been published – it is within the Ministry of Justice Business Plan which was published last Monday. The first two objectives within that particular Business Plan are:

1.1 Develop an overall strategy for the ‘rehabilitation revolution’ for adults and youths, including paying local private and voluntary organisations by results.

1.2 Introduce payment by results schemes, working with local, voluntary and private sector organisations that specialise in the rehabilitation of offenders.

I am interested in this approach as I believe that society as a whole should move more towards an outcome focussed approach to business. This is particularly true of learning and development providers – and clients.

If a client seeks and purchases a course, event or programme to address a particular area or aspect of business, it is likely that it will be delivered and there may well be a short end of event questionnaire. It is rare, in my experience, that there is much further evaluation of the benefits of the event.

So, how about the client publicises the issues or skills they require addressing on the event and these are then quantified into measurable outcomes (not outputs).  The provider who then gains the contract only gets paid once these targets have been achieved – perhaps 6 months after the event. The probable consequences would include:

  • A greater focus on outcomes as opposed to outputs.
  • Only providers who have confidence in their abilities bidding for the contract.
  • Clients ensuring and demonstrating value for their money.
  • Far greater interest by clients, providers and delegates’ line managers as to how the learning is put into practice in the workplace.

Who wouldn’t want these consequences?

So why does this approach generally not happen at the moment?


Ofsted – head in the sandpit?

Sunday, November 7th, 2010

As we focus on where we could save money and where we waste money in the UK in order to balance the books, I am becoming increasingly unsure of why Ofsted continues to exist – at least in its current form.

Earlier this year (15 August 2010) I blogged about my thoughts on what Chris Woodhead, the former Chief Inspector of Ofsted (Office for Standards in Education) had been saying about schooling. In The Times (25 July 2010), whilst talking about English Literature classes, he wrote, “The aim should not be to develop ‘critical thinking skills’”, and added that children, “should not be encouraged to express their opinions on the texts. Who cares what they think or feel?” I find that harder to believe every time I read it. And, incidentally, I care what they think and feel.

His more current collegues were also involved in the tragic Baby Peter case. You may recall that Ms Shoesmith’s was the Director of Children’s Services in Haringey at the time, and she appealed against unfair dismissal by the Council. During the appeal Ofsted suddenly discovered thousands of pages of undisclosed evidence which they had previously said could not be located. These contained snippets such as notes relating to Ofsted inspectors being told to delete all emails on the case, and evidence that an additional finding was popped in stating, “There is insufficient strategic leadership and management oversight …”  Ofsted had also given Haringey’s Children’s Services a glowing 3-star rating only a few weeks after Baby Peter had died. The subsequent revelations didn’t make things appear that 3-star to others.

And last week Ofsted again hit the headlines for their part in the awful Little Ted Nursery case. The Plymouth Safeguarding Children Board completed the Serious Case Review into the Nursery in March 2010, and sent it to Ofsted as is required. Apparently, Ofsted didn’t respond until October 2010, thus delaying the publishing of the Report until last week.

The purpose of a Serious Case Review is to ensure that lessons are learned and practices are changed so that incidents such as this should not recur – so that society safeguards and promotes the welfare of children. How is taking 7 months to respond to a report an effective way of protecting children? I thought that there must be a good reason for this delay, so I went to its website and had a look at the ‘News’ section. Not a mention – even though they’ve had 7 months to think about it.  

Paragraph 5.44 of the Serious Case Review states, “It has become clear from this review that whilst the Early Years Service had many concerns about the nursery there was no formal mechanism for informing Ofsted, since they did not reach the threshold of a breach of regulations. Similarly Ofsted had no means of discussing with Early Years the support need of the nursery. It is notable that whilst Early Years had the nursery identified as red or amber on its own rating system, Ofsted inspections were good or satisfactory”. Again, Ofsted making a judgement that is far removed from reality.

So why is this happening? There is a theme, and that is the ‘tick-box’ culture that Ofsted appears to support. This was noted by Deborah Orr when she wrote in the Guardian last year (26 November 2009). “Does Ofsted’s  tick-box culture deserve the criticism that has been heaped on it this week? I do believe so. I have before me an inspection report on a London primary school, rated outstanding. It scores the highest grade (out of four) in every one of 33 categories, except two. What are the two categories that this school fails to achieve perfection in? Oh, just ‘standards achieved by learners’ and ‘how well learners develop . . . skills that will contribute to their economic well-being’. Little things, of no great consequence in a school.”

What’s the common denominator in these last three examples? Ofsted have awarded ratings that are far higher than the services or institutions warrant – when looking at it from the perspective of society rather than ticked boxes.

Exactly. Because what generally happens when you create a tick-box culture? You measure things that are easy to measure – because that’s what ticking boxes is about. And that in turn makes unimportant things suddenly important – because they are generally easier to measure when using boxes to be ticked. And so people stop concentrating on the important things – because what gets measured generally happens.  And so the truly important matters are not important any longer – well, not to the inspectors and those who are being inspected and want to achieve a good rating.

These examples are three occasions where Ofsted have given ratings far higher than they deserve.

Earlier this year Zenna Atkins announced she would be standing down as Chair of Ofsted on 31 August 2010.  When she spoke of her decision to stand down, she said, “The work of Ofsted not only costs the taxpayer a third less than when undertaken by four separate inspectorates but we are also supporting improvement in a more efficient and effective way. Front-line observation and engagement with children, parents and leadership teams are now at the heart of all new inspection arrangements, sweeping away any inherited tick box culture”.

I’m not convinced, Zenna.

In my view, Ofsted is not providing value for money – even if it is costing a third less – and it needs a complete overhaul. More important than it not providing value, it is not measuring the right things, and most importantly it is not effectively protecting children or supporting their development. And it appears to be institutionalised in its processes and so I wonder whether it will be able to change. I’d scrap it.

It is great as a case study of what can go wrong when tick-box processes are created. But, sadly, in my opinion, that’s all it’s currently good for.


Avoiding evaluation pitfalls

Sunday, October 24th, 2010

In my last post I said that I would use this one to explain how to avoid common pitfalls when undertaking evaluations. So here goes …

  1. Identify your sponsor. This is the person asking for the evaluation and has the authority to implement any recommendations. If the person you are working to is not the person who can implement your findings, you haven’t found your sponsor. What’s the point in carrying out work specified by a person who cannot do anything with it?
  2. The list of ‘10 key questions to ask’ that I published in my last post have to be discussed and agreed with your sponsor. Without clarity on these answers you will not be able to create an appropriate methodology. You may want to produce lots of statistical data, but if that’s not what he or she requires, it’s pointless creating it.  This helps you get your methodology spot on.
  3. When you are asked to evaluate something, remember that there will be a reason for it – and sometimes that reason will not be shared with you. Your sponsor may want the evaluation to prove something, they might want to use it to justify something or they may have already decided what your findings should be. So, what happens when your findings are different or you come up with the ‘wrong’ answers? What can happen is that the evaluator gets blamed for problem. You have used an inappropriate methodology or interpreted the data incorrectly or just not done what the sponsor asked. The focus is turned on the evaluator. So whenever you agree something with your sponsor get it signed – it is one of the many strands of life where having an insurance policy can be useful even though you don’t expect to need one. And this one costs nothing. And if the sponsor is reluctant to acknowledge his or her agreement to it, you perhaps need to be concerned at that early stage. Why are they reluctant? Do they actually know what they want?
  4. Once the evaluator has chosen an appropriate methodology, it can be very easy to stray off the planned route. At regular intervals check the methodology still fits with the agreement you have with the sponsor. This will help you stay on track, and to appropriately amend the methodology should it be warranted.
  5. Triangulation. Often you will find what appears to be a meaningful or important piece of data. If you are to publish it, I suggest that you triangulate it. That means you corroborate it from a different source. Using non-triangulated data can be problematic and may make your findings easier to challenge.
  6. Consider letting the sponsor and key stakeholders see copies of your report prior to publication – assuming your agreement with your sponsor allows this. This is particularly useful if you have some controversial data. You may get asked to remove it – which you can then do if you think it appropriate – however, you will already have made your point to those people who matter.

By following this guidance you should avoid the common pitfalls, keep on track, protect yourself and be able to produce a very effective evaluation report.

Good luck!


How do I ask that?

Wednesday, October 13th, 2010

On the community forums where I often post comments I spend a significant proportion of the time answering or offering thoughts on learning and development evaluation. This then leads to me receiving other personal emails regarding the subject. One of the key areas that appears to be problematic for a number of people relates to questionnaire writing.

I have formed this opinion both from the questions I have been asked and from seeing some of the questionnaires that people have asked me to complete or critique. I think this is because people often think (something like) “They are only questions – I ask questions all the time”. They can therefore assume it is straightforward and so don’t put sufficient thought and time into the process. Exactly what information are you looking for? What type or style of questions are most likely to elicit the information. How should they be worded?

It is a critical area. It can have a huge impact on the quality and quantity of the primary data you generate. Many years ago I was lucky to have had the opportunity to write and deliver an 11-week course on Training Evaluation. As you might gather from the duration of the course, it was very detailed! We spent a day of the Programme looking at how to write a covering letter to a questionnaire, and another 3 days on the actual formulation and writing of the questionnaire itself.

Are you thinking of writing a questionnaire? If so, the following 10 principles will assist you to maximise the quality of data and quantity of questionnaires returned to you.

1                     Avoid general or vague questions

2                     Avoid hypothetical questions

3                     Avoid leading questions

4                     Avoid double-barrelled questions

5                     Avoid jargon

6                     Avoid potentially embarrassing questions

7                     Keep your questions short

8                     Consider your terminology

9                     Open ended or prompted questions?

10                 Apply all the principles appropriately

If you go to the Discussions area of my Company Facebook Page you will find a document with more detail on each of these principles, together with examples of each. Or you can email me if you would like a version in MS Word.

On my next post I will look at two different models of evaluation.