is in the business of moderating groups it’s not so long ago that many of us were at the vantage point behind the glass. As clients and planners we used to watch people doing what we do now, and often not envy those doing our current job. It’s from this dual perspective that we feel marginally qualified to offer some observations and thoughts about the process of watching research groups.
It seems that the hiring of a viewing facility is rather a luxury in straitened times. So such moments are, and should be treated, as fairly precious quality time with the people who matter most (by this we mean customers).
But our experience and that of our colleagues elsewhere suggests that this is not always the case and that the opportunity to see and hear customers sharing their views is sometimes ignored or misused. So we thought it worth highlighting some specific examples of less-than-ideal approaches to watching research groups, the particular issues these expose and make some suggestions as to how to get the most out of the experience.
But first let’s think about why one might watch research groups.
There are a number of reasons to watch groups.
1. To check that the process is being conducted properly and that the moderator is doing a good job, getting everything out of the research that needs to be covered. This is an understandable objective if a research company is new to the business or if there have been any areas of concern in the set up of the project but if a good relationship exists then this should not be a reason to come to watch.
2. Out of interest. Sometimes clients just want to see the whites of their consumers’ eyes, hear them speak, get a glimpse of their world. This is perhaps the best reason for watching research groups. It doesn’t require attendance at every group in the project, it doesn’t require analysis of the learnings (which is what the research is being paid to provide), it should be an interesting part of the client’s day and give them some textural examples to use in their course of their work. But attendance at one or two groups should never be seen as more than a taster of the project as a whole. It might prompt questions to ask at the debrief, based on experience of just a few of the groups, but it would be dangerous to base any conclusions on what has been seen. However vociferously and unanimously an opinion was expressed in one group, if the other 7 demonstrated a different way of thinking then the viewed response must be seen in that context.
3. To be able to report back to colleagues what has happened, perhaps to faciliate some premature decision making. This is not a reason to attend that many researchers would endorse. Unless the client viewer is going to come to all the groups, then go away and listen to/watch all the recordings again, analyse, pull apart and put together again all that was said, discuss extensively with colleagues, look for the significance of what wasn’t said as well as what was, review non-verbal cues i.e. do everything the researcher will be doing then there is a risk that the conclusions drawn from observation without analysis could be erroneous or misleading. And if the viewer is going to conduct analysis with rigour comparable to that of the researcher, it rather begs the question ‘why use a researcher?’
4. To get a topline steer on what respondents think without any subsequent analysis, to be able to ask the right questions in the debrief and to be able to make observations to be challenged by analysis. Not a bad reason at all, as long as the distinction between observations and the insight born of analysis is appreciated.
5. Because that’s what’s always done. This is fair enough, if habit has arisen for a good reason. See above…
Our conclusion from a brief survey of the reasons for viewing research groups (and we are sure to have missed some out) is that it is important to be clear why you’re viewing groups, what you plan to get out of it, what you need to do with what you see and whether this is a realistic and valid objective to have. Is viewing this group the best use of your time?
Now for some case studies that demonstrate problems that can arise when groups aren’t viewed in the right way.
Case study one: Do you want questions or answers?
A moderator is conducting some script evaluation in a viewing facility in Moscow. All the scripts are attempting to communicate that a Product A manages to work quickly but also be gentle. A particular script provokes much discussion and it is clear to the moderator, from what the respondents are saying, that this script does successfully communicate both product benefits. Respondents are playing back this message in their own words. However, the client is concerned that the moderator has not explicitly asked the question ‘does this tell you that the product works quickly and gently?’The moderator has not asked this question because respondents have provided the answer spontaneously. She would learn nothing new by asking this question outright. But the client has seen a discussion guide which indicates that this question must be answered (note: answered) and has assumed this to mean that the question needs therefore to be asked. As we all know, the thing about research groups is that they don’t always exactly follow the discussion guide to the letter. Sometimes with some gentle nudges they pretty much moderate themselves, as a discussion ranges across all required topics and resolves all proposed questions without them being asked. Sometimes it is appropriate to re-order sections because the conversation more naturally flows that way. Sometimes a later question is answered much earlier and therefore doesn’t need to be covered again.
What this illustrates: A couple of things. That discussion guides are guides not scripts. And that answers are more important than questions. If the information required is elicited or offered in another way or at another time in the group it is still just as valid. Discussion guides are not checklists to be ticked off whilst watching a group, or certainly not in the order in which they are written. It’s fair enough to object if a whole area has been neglected but as long as everything has been covered in some way then the group has been moderated properly. We would argue that discussion guides are of primary usefulness before the groups: they are the ideal opportunity to check that all objectives will be delivered, all areas will be covered, that everyone is happy with all techniques to be used, that the stimulus material can be effectively used.
Case study two: What you see is not all you get
We know a researcher who we rate exceptionally highly (we’re not telling you who he is though) but not all of his clients see him that way. They are uncertain about him because his groups sometimes seem a bit brief, not as full as they would imagine or expect. They have censured him for missing bits of the discussion guide, for finishing groups early. But his debriefs are brilliant – insightful, answering all the questions and responding to all the objectives in full and actionable ways. His groups might have appeared (wrongly) to be superficial but no-one could lay that accusation at his debriefs. And why is this? Because he’s so naturally intuitive and insightful, he can see and understand what lies behind what is said, he can get quickly to meaning without the need for slightly repetitive questions. There are a lot of researchers like him (we’d like to say we know some very close to home…) but what he doesn’t do is play to his audience, take any account of the people behind the glass and what they conventionally expect to see. He, rightly, sees qualitative research groups as being the means to answer questions but sees the answers as being of greater importance than the means. So he doesn’t perform, he does it his way.
What this illustrates: That sometimes the importance of how a group is moderated is over-estimated and the ‘performance’ of the researcher as moderator is judged separately from his/her overall performance on the project as a whole.
What can we learn from this? The way to judge is not on performance on the night but on what he/she comes back with, how well the debrief fulfils the objectives, how far it grows knowledge and understanding, how insightful and how useful it is. The end should very definitely justify the means.
Case study three: Not getting the ‘result’ desired
This is a less specific example as we have seen this so many times. We’re often called upon to research a number of ideas/concepts/routes alongside each other in order to determine which best fulfils the clients’ objectives for it. Although the term is pejorative it is essentially a beauty parade. The problem arises when there is not a clear winner, when several routes are perhaps deemed to be strong in different ways. Or something different wins in each group. Too often this is seen as a failure of research, that the researchers have not been able to identify the optimum idea. But sometimes there just isn’t a winner. This is no worse a result than the revelation of a clear direction. It might mean more work needs to be done but often the other learning and insight from the group is as directional and useful as the isolation of a lead route would be.
What this illustrates: That it is tempting to think that the answer will come from the obvious place whereas a more useful and interesting answer and direction might come from somewhere else. This also illustrates how research can be all too readily written off if it doesn’t provide an exact answer when in fact an exact answer may be elusive because none of the materials presented as stimulus contain the definitive answer.
What can we learn from this? To look at what a project as a whole reveals, to see how questions might be answered in unexpected ways and to understand that research might sometimes not be able to recognise, choose or highlight the optimum direction if the optimum direction has not yet been created. Sometimes research will point out where more work needs to be done, and good research will not shy away from doing this.
So, Divine’s top tips for getting the most out of viewed groups, in no particular order
1. Be clear why you’re going and what you want to get out of this? Is this feasible and is this fair?
2. Remember that what you see in one group is a fraction of what you’ll get at thedebrief, which is the result of a lot more analysis and insight. It’s more than the sum of the observation of every group conducted.
1. Be clear why you’re going and what you want to get out of this? Is this feasible and is this fair?
2. Remember that what you see in one group is a fraction of what you’ll get at thedebrief, which is the result of a lot more analysis and insight. It’s more than the sum of the observation of every group conducted.
3. Don’t arrange to report back to colleagues immediately after the viewed groups: you’ll be giving them a small part of the story only
4. Don’t evaluate the moderator by how many questions he asks or how closely he sticks to the discussion guide.
5. And finally always try to remember that the discussion guide is a just that, not a script - some questions don’t need to be asked to be answered.
No comments:
Post a Comment