The unexamined life…
Like anything in life, it is not the experience, but what we do with the experience, that matters.
As a passionate learning designer, facilitator, and leadership development professional, (not to mention decades of work for the public service), I’ve consciously developed my skills in evaluation. My experience and ability to measure whether an intervention actually met the input, output, outcome, and impactful goals and objectives that were set at the time of the design of the intervention!
I invest in this, because evaluation is deeply impactful to the efficacy of the intervention.
Investment in Evaluation
In other words, if one cares about the impact you are having, you’ve got to spend some energy on bringing your awareness to whether it worked or not! You need to have thought about this before the intervention, and need to have a system for using the information collected to better inform how you can improve, maintain, sustain the good you are doing.
Over the years, I’ve developed my method, through trial and error, through classes, through mentorship, through coaching, and through on-the-job experience! And my present work, and the value I can add to my clients, benefits from this.
If you are a facilitator, learning designer, or an advocate for positive change, there are many tools now at our disposal for evaluating the success of what we do. From technology that helps to create ‘word clouds’, to easy online surveys, we’ve got the ability for instantaneous feedback. However, this comes, I’ve found, with a caution.
Cautions around the ‘easy’ evaluation
In my experience, the ‘digital solutions’ have genuine usefulness when their reason for use is to easily capture surface level indicators. They are relatively inexpensive, fast, easy to administer, but like with so many things, it is important to use these tools (transparently) for what they were intended. Quick, cheap, surface level indicators.
They are innovative tools, but do not capture innovative data.
By their very nature (character limited free-text, selection from pre-conceived ideas), they limit the amount of information ‘from left-field’ that can be captured. The type of important ideas that the designer, facilitator, evaluator did not think of, but impacted the participant.
That type of data is the important data to take action on - something you can’t do if you’ve not captured it.
How to capture data that enables understanding
Whenever I do learning design, or redesign, I definitely examine both the quantitive (Likert scale) data points, as well as the free-text data points from my evaluations.
What I’ve learned is that the Likert scale metrics are really useful in only two circumstances:
Identifying trends across multiple interventions; or
in illustrating the group’s general growth against a certain learning objective pre to post workshop
Graph of pre and post confidence against learning objectives
I know that both of these stories are very useful, not just in the quality of the product I’m evaluating, but also in the reporting back to the organisation or Delegate that I am contracted to.
However, to know ‘what was working for’ or ‘what was NOT working for’ the participant, I need to examine the free-text as they contain the ideas that a participant took the time to type or write into the evaluation.
The free text enables people to raise issues that
I may not of even thought about.
It provides the ‘lived experience’ of the participant… which, while I try to put myself into the mind of the participant during the design, there is still a ‘richness’ that is provided in free text that is not provided in pre-populated survey, or even character limited free text.
The Succesful Mindset for you as an evaluator
If you are given the chance to design evaluation, getting to appreciate the 'human understanding’ element which makes valuable the elements of evaluation - such as:
the data cleanse; and
the codifying of data collected; and
the analysis of the data (what story does it tell?).
These important steps of evaluation, when handed over to surface level ‘technological solutions’ fail to capture the nuance, the associations, the human story within the data. If we care about the performance of the product we produce, we should care enough to invest some energy into the evaluation. This is why free-text answers to questions are so essential for genuine evaluation.
Challenges and how to address them:
Many have said to me in the past, ‘yes, James, but it is hard to get anyone to fill in an evaluation, let alone write free-text!’
I’ve not found that to be the case. What has happened for my participants is that they clearly and genuinely believe that I am there ‘in service to the needs of’ the participants in the workshop. That I authentically believe in this and that I earnestly want to make sure I am improving and meeting their needs.
In short, my genuine intention is able to ’sell the benefit’ of the evaluation in a sincere, compelling way - and one in which the participant actually knows that I care about the surveys. Consequently, I get consistently high percentage response rates for evaluations (95%-100%)… and high percentage rates for free text added to the free-text questions (25%-45%) free text comments provided. (95% if I REALLY sell it!)
The take-away here is that without free-text questions, you can only test your assumptions but you can’t get new data… or, to put it more frankly, you can only test your current perspectives, you can’t get new perspectives, which is what you need if you are going to ‘look into your blind spots’….
(WARNING – CYNICAL MOMENT AHEAD)
As one who is passionate about my own development, and feedback, on cyclical days, when there is a survey that does not have ‘free text’, I know that either the company, organisation, etc… does not understand the importance of quality evaluation, or they do, and they are not really engaged in or care about the participant / user feedback.
When I am really cynical, I think that they merely want an easy, cost-effective, time-constrained method for the participant to ‘feel like’ they were given the opportunity to provide feedback, an most likely don’t even have a system for actually incorporating the feedback that they are asking us to take the time to give.
If you are passionate about not only the ‘input’, and not only the ‘output’ of the thing you are evaluating, but you want to ensure it meets the ‘outcome’ and ‘impact’ - then please, invest in good evaluation that includes free-text - and a system to use this feedback for the benefit of the product. It is, in my opinion, a level of respect that is required when assign people to take the time to evaluate your intervention!
