The challenge in edtech is not simply to innovate but to do so with purpose. So how do organisations identify the opportunities most worth pursuing? In this piece, Prof. John Traxler draws on academic and consultancy perspectives to explore different approaches to identifying those opportunities, from brainstorming and horizon scanning to market research and AI. This article offers a broad overview and perspective on how the edtech sector can approach innovation more deliberately. 

Scanning, Researching and Rethinking Innovation in Edtech Development

Author: Prof. John Traxler, UNESCO Chair, Commonwealth of Learning Chair and Academic Director of the Avallain Lab

The Challenges

St. Gallen, May 28, 2026 – Where does our next edtech product come from? How do we spot good ideas, or create them, whilst avoiding the bad and the old ones? How do we avoid simply copying our competitors and do more than merely comply with our clients? How do we do more than enhance and improve the status quo? How do we fight off ‘stuckness’? We should also ask whether, in fact, the edtech sector differs much from other sectors where software systems are developed or perhaps any other kind of product? 

Each exists in its own world of stakeholders, regulations, procedures, traditions and resources. What distinguishes software development is that the raw materials are merely data and instructions, free and limitless, perhaps mistakenly suggesting that innovation is without cost. Its traditions and procedures have expanded, evolved and mutated incredibly rapidly, over less than a working lifetime. Together, these mean that the questions we raise do not have tried-and-tested answers. The ethos of developers is also a factor: are they hungry, visionary start-ups, or reliable, quality-conscious corporates, and how does each approach these questions?

Furthermore, edtech products that support education systems operate in contexts where multiple, conflicting stakeholders make achieving consensus on ‘good’ edtech very problematic. 

This blog addresses these kinds of questions, but as understood by an academic who has moved into the edtech sector, drawing on consultancy and research experiences, hoping to provoke questions and reactions and perhaps some changes. This reflects part of the Avallain Lab’s wider mission to foster productive relationships between academia, the sector and its clients.

A previous blog discussed the echo chamber/revolving door that seems evident in the people and processes of institutional IT procurement, and, in the current context, this may be a brake on change and innovation, excluding some technical voices and perspectives, fostering incremental quantitative improvement rather than radical qualitative transformation. Again, problematised by uncertainty about what constitutes ‘good’ edtech products in education systems that are very unconfident and uncertain about their own purpose.

Brainstorming

To start upstream, brainstorming is a recognised technique, widely described across the media, not just undisciplined musings or mutterings. Brainstorming is a creative technique for generating many new ideas about a problem or topic, focusing on quantity over quality initially, deferring judgement, encouraging wild ideas and building on others’ suggestions in a free-flowing, often group-based session. The basic rules are: 

• Suspend judgement during the initial phase; don’t let criticism kill the momentum.
• Encourage wild ideas; they contain the seed of a practical, breakthrough feature.
• Go for quantity; clear out the obvious ideas to reach the innovative ones underneath.
• Combine and improve participants’ ideas; transform simple ideas into better ones, building on shared contributions. 

Common sense suggests the best number of people to be involved is fewer than perhaps ten, to avoid chaos and confusion, and ‘hiding’, but more than perhaps five, to avoid stagnation. Still, clearly, composition is important, with similarity and homogeneity fostering candour and spontaneity, whilst differences in hierarchy might be inhibiting. There is, however, an argument for neurodiversity or diverse cognitive styles, but all this presupposes a large enough pool of potential participants in which to make these kinds of choices. Obviously, the physical setting is important; different settings all send different signals to different demographics and cultures, as does the timing. One possibility is the away-day format, cut off from daily pressures and obligations, and a moderator might prevent groupthink and give space to quieter, tentative voices. There is perhaps some overlap with the heuristics for effective focus groups, including tips for effective moderation that ensure a free-flowing, non-judgemental event.

Incidentally, boredom too has its uses, all the more so as phones and computers often keep it at bay, creating opportunities for creativity or originality.

These established formats and prescriptions for effective brainstorming are mostly pre-COVID and assume that working and meeting face-to-face are the norm. This is clearly no longer the case with many people, perhaps the more creative or imaginative, who are either working online from home or digitally nomadic. Their varied individual settings, disruptive external events, such as a delivery at the door, lunch burning and the changed cues, language and tacit protocols of online interaction, might not be so conducive to spontaneity or candour.

Perhaps the move of the Delphi technique events from face-to-face synchronous to asynchronous online, for all sorts of pragmatic reasons, might suggest a compromise format that reconciles individual creativity with group interactivity, with the added bonus of the latter being digitally recorded and preserved. 

Whilst these might be prescriptions for effective brainstorming, they do not address when to brainstorm in relation to any product development cycle or how to feed the outcomes into the mainstream of developments; there are presumably good ways and bad ways, and at the risk of going off at a tangent, this looks like an opportunity for ‘diffusion of innovations’ approaches to find the good ways and the factors that determine which best way.

Horizon Scanning

Horizon scanning is a way of spotting possibilities coming towards us, for example, of managing those possibilities that brainstorming has surfaced.

Some background: several years ago, I collaborated with Alison Potter from the TEL division of Health Education England (South), part of the UK NHS, to review horizon scanning and to formalise and embed it in their routines. Horizon scanning attempts to spot concepts, opportunities and technologies before they reach the market (and before they reach the competitors, hoping to catch the next Teflon or Post-it before they do), especially those not immediately and obviously relevant, the ones off in left field. 

The work examined organisations comparable in size and technology to the NHS, including the UK government’s Cabinet Office, and distilled their procedures into a set for the NHS. Our initial research question was, ‘What models exist for identifying and then prioritising which new and emerging technologies might add value to healthcare education in the UK?’ We conducted a literature review of horizon-scanning methods to identify existing models and systems. Then we conducted interviews with six experts across education, government, healthcare and the independent strategic foresight sector. The findings from the literature shaped the interview design. Interviews comprised of three parts: a short experiment to gauge how each expert horizon scans, their reaction to our proposed framework and lastly, their thoughts on the skills and tools necessary to horizon scan.

Alison’s final version of the horizon-scanning framework, the culmination of the whole research process, features a sequence of several distinct activities, and her paper goes into greater detail. 

• Identification, or scanning a defined set of sources, addressing what is out there
• Classifying, or filtering, then prioritising, addressing what is relevant
• Assessing, addressing, what is its potential impact
• Disseminating, or navigating, addressing where it needs to go
• Evaluating, or reflecting, addressing how we do it better

And then, start again, perhaps on some predetermined cycle time matched to the organisational timescales and responsiveness. 

We should, however, always bear in mind, when defining the sources to be scanned, assessing the impact of any discoveries and disseminating them, that any such discoveries need to align with various commercial, technical and organisational factors. These factors might include the headroom and skill set among staff, the alignment with the existing product portfolio and client base, and the organisation’s management of change. These factors are, in effect, among those identified in the diffusion of innovations community, a body of expertise stretching back many decades, tackling innovations from new technical products to changed farming practices to improved attitudes to smoking and drink-driving. In this context, the ‘innovation’ is the horizon-scanning discovery. Diffusion of Innovations work in its various forms over the years looks at factors such as the characteristics of the people involved, perhaps the developers inside edtech or the clients outside, whether they are naturally risk-taking or risk-averse, the development, whether it can be deployed without a tangle of interoperability issues, whether it can be easily explained and understood (and sold), the nature of any competitive advantage, so on.

It does, however, leave the sources to be scanned unanswered. Horizon scanning is one; others might exploit the expertise and experience of researchers, described briefly later, exploiting their literature searching skills, their contacts and their colleagues, and also their intuitions and ability to pick up ‘weak signals’.

Market Research

Looking now at market research as another source of innovative ideas, I am deeply indebted over the years to the work of Gordon Rugg on knowledge and its elicitation, in every kind of research that involves people, meaning clients, users, learners and the wider market. This work recognises that people know, believe and feel all sorts of different things and that finding out what they are thus requires all sorts of different techniques and tools. This work is expressed as the ACRE, ACquisition of REquirements, framework, a tabulation that goes from every type of knowledge or feeling or value to the most effective tool or technique for eliciting it, from the conventional, namely surveys, questionnaires, etc., to the ‘contrived’, such as card sorts, rep grids and laddering, to the physical, such as models and prototypes. Within this overarching framework, there is still the need for adaptation, refinement and common sense, so don’t ask compound or double-negative questions; do make sure participants are not hungry, uncomfortable or embarrassed and so on.

In my work, I have often lambasted ‘the usual suspects’ of social science (and market research), namely the focus group, the interview, the questionnaire and the survey, rounded up unthinkingly to answer every conceivable question, as ethically problematic, methodologically deeply flawed and usually inappropriate. 

Without unpacking and explaining all of the alternatives to the ‘usual suspects’, which you can unpack here, it might suffice to say that asking questions only provides the answers to those questions, even assuming the respondent is able and willing to give an adequate, honest answer, rather than finding out what is actually important to the respondent. Furthermore, asking questions about desirable futures only elicits answers based on modified presents and remembered pasts rather than any radically reimagined futures. 

These are the weaknesses in expecting clients or users – actually, users are not always asked, often their managers or IT do so on their behalf  – to guide future products or projects; merely asking them will likely elicit only requests for what they already have, but faster, easier, bigger or bug-free. So perhaps academic research can represent a more rigorous version of market research?

‘Real’ Research

Separating market research from ‘real’ research is an artificial and unnecessary distinction, since both should be activities aimed at acquiring, analysing, understanding and contextualising what people know or want or feel in ways that are trustworthy, cheap, appropriate, ethical and efficient. Both can suffer from exactly the same flaws because each, in its own sphere, is subject to very similar pressures and constraints. The distinction might in fact be between the people, the market research researchers on the one hand and academic researchers on the other, and on the expectations, timescales and resources around their different professions. The question here is, what can academic researchers contribute?  

Two things, really, namely, what might be called primary and what might be called secondary research, the former being actually doing stuff, conducting empirical studies, setting up interventions, taking measurements, listening to people, building prototypes and running workshops, the latter being connecting with the outputs and activities of the people who are doing primary research, using experience and expertise, to understand what is happening and what might be useful, an informal version of horizon scanning in practice.

It has to be said that primary research, especially in the context of commercial edtech, is probably a waste of time, since any commercial advantage is likely to be small and short-term, though it may have value as an agent of culture change within an organisation, raising awareness of methods and limitations, and this may be something of indirect commercial value. There is a far better case for secondary research since it spreads the risks and costs and is perhaps a semi-intuitive version of horizon-scanning, based on gut feelings and looking for otherwise undetected ‘weak signals’. This rationale underpins the Avallain Lab, built on expertise and experience that a search engine or chatbot can’t simulate and tapping into contacts and colleagues before their work hits the public domain. This model is still being refined.

Process Maturity

To go off at a tangent, process maturity models have recently been spotted being applied to AI development, though not yet to educational AI development, and that may be an important or provocative opportunity.  

Process maturity and its models are ways of describing how well an organisation handles bugs, mistakes and mishaps. If an organisation just deals with them as they crop up, it might be categorised as relatively immature. It may, however, document or record them, perhaps analyse and reflect on them, and have procedures for analysis and reflection, and indeed departments and specialisms for doing this, indicating a progressively more mature organisation. These stages have been formalised as process maturity models, progressing from chaotic (Level 1) to consistently effective and optimised (Level 5), using models to standardise procedures, enhance quality, boost efficiency and ensure scalability to achieve strategic goals (and, accordingly, to gain certification). This approach was adopted in large-scale software development in the 1990s; for example, the Capability Maturity Model of the 1980s. Also, later, in courseware development and now, it seems, in some AI development and perhaps next in future edtech development, why not?

The relationship between notions of process maturity and the other earlier topics is, however, oblique; the first ones talk about qualitative or strategic jumps, thinking ‘outside the box’, about breaking away from the established trajectory, whereas the last one talks about incremental quantitative or technical improvement, about moving along the established trajectory but more effectively and efficiently, ‘inside the box’. They must, however, be reconciled; otherwise, organisations risk either forever improving the past or never shaping the future. 

The way forward may be to treat brainstorming and horizon scanning as processes in their own right, ones that, on reflection, could be monitored and measured and thus improved, but also then feed into roadmaps. In essence, the way forward must reconcile the tensions between the ‘stay hungry’ of start-ups and the quality assurance expected of established organisations.

Artificial Intelligence

These are all largely pre-digital accounts, and we should now perhaps look for digital tools that capture these methods and techniques, especially for AI tools, the generative ones that answer our questions and the agentic ones that execute our processes. At the moment, however, the best advice might be to proceed with caution. Current AI, working on probabilistic mechanisms, risks emphasising the existing norms rather than breaking away from them; perhaps ‘hallucinations’ have a part to play. One under-researched area comprises scenarios depicting how society, its education systems, the economy, its labour markets and the workforce and their skill sets will evolve under the impact of artificial intelligence. In their different ways, these all form the contexts of edtech products and how they are developed.

Finally

This piece outlines how disparate techniques from disparate communities might have productive synergy. Each technique, and probably others, deserves greater attention in order to explore adaptation and integration. Taken together, they offer useful perspectives on how edtech organisations might think more deliberately about identifying meaningful opportunities, challenging established assumptions and navigating future change. The impact of AI is currently limited to answering questions and making discrete activities more efficient. Clearly, it won’t stop there.

About Avallain

For more than two decades, Avallain has enabled publishers, institutions and educators to create and deliver world-class digital education products and programmes. Our award-winning solutions include Avallain Author, an AI-powered authoring tool, Avallain Magnet, a peerless LMS with integrated AI, and TeacherMatic, a ready-to-use AI toolkit created for and refined by educators.

Our technology meets the highest standards with accessibility and human-centred design at its core. Through Avallain Intelligence, our framework for the responsible use of AI in education, we empower our clients to unlock AI’s full potential, applied ethically and safely. Avallain is ISO/IEC 27001:2022 and SOC 2 Type 2 certified and a participant in the United Nations Global Compact.

Find out more at avallain.com

_

Contact:

Daniel Seuling

VP Client Relations & Marketing

dseuling@avallain.com

Digital learning has evolved significantly over the past few decades, shaped by virtual learning environments, the rise of open, networked technologies and, more recently, the emergence of AI. In this piece, Prof. John Traxler examines the divide between the quality-assured environments of formal education and the more open and less structured world of informal digital learning. He highlights both the challenge and opportunity to develop a balanced space between the extremes, one that requires learners to develop key skills such as criticality, curation, metacognition and reflection, and raises questions about responsibility and opportunity.

Closing the Gap Between Formal and Informal Digital Learning

Author: Prof. John Traxler, UNESCO Chair, Commonwealth of Learning Chair and Academic Director of the Avallain Lab

St. Gallen, March 30, 2026 – This piece is based largely on experience, research, writing and analysis of education beyond compulsory schooling (typically from age 16), mostly in the UK, Western Europe and similar international education systems; other sectors and other countries will have their own versions, but are still driven by underlying social, financial, technical and political factors, albeit differently across contexts. The piece aims to present a different perspective on digital learning from the more conventional one and, in doing so, may, in a relatively small space, simplify and generalise; but it is the perspective that matters, and the challenge and opportunity it represents.

The Dawn of Digital Learning

Many years ago, probably for the course of the decade straddling the turn of the century, if learners wanted to access digital educational resources and opportunities, they could only do so as students of some kind of formal education from an established education provider. The World Wide Web existed and was populated by institutions, corporations and organisations, often using the emerging technology of the Virtual Learning Environment (VLE), aka Learning Management System (LMS), such as Blackboard, WebCT or the open-source Moodle. This was their only access, which meant the institution could define what was learnt, how it was to be learnt and what behaviour was acceptable; the institution controlled the hardware, software and infrastructure. 

Pedagogies, Espoused and Enacted

Interestingly, the rapid emergence of digital learning technologies and the expansion of higher education led to the professionalisation of teaching, no longer an essentially amateur activity by researchers in elite universities. It also led to the increased visibility of teaching, no longer confined to the privacy of the face-to-face seminar room. Not only was there the expectation that academics would learn to teach and be certified once they had, but the theorising of teaching should become much more explicit, no longer intuitive but conceptualised. 

In particular, the component technologies of the VLE, such as the chat and webinar functions, were portrayed as vehicles for social constructive pedagogies, in which learners would interact and engage with each other, and for constructivist pedagogies, in which individualised learning would build on individual understandings. Gone were the days of learners being given content to absorb and retain; instead, their understanding was built on and was valued. Lecturers, it was proposed, would mutate from ‘the sage on the stage’ to the ‘guide by your side’, taking on a more facilitative role rather than acting as the primary source of knowledge.

The VLE did ensure consistency and efficiency; it seemed to allow teaching to be visible and thus monitored, and different lecturers to be swapped into and out of courses according to staffing; it also encouraged the expansion of distance learning, but this in fact offered little competitive advantage since every institution had the same idea and global markets already had global players. 

Many years ago, I remarked to South African colleagues that making students use a VLE was like making them wear a school uniform. The reaction was, ‘What’s wrong with a school uniform?’ So, yes, I understood the need for consistency, equity, stability and quality assurance, but it is not adequate preparation for fashion choices, dress codes and expressing individuality through adult clothing. In other words, structured systems ensure fairness, but not independence.

Unfortunately, institutional digital learning meant that educational change and its institutional processes were now interlocked with IT change and its institutional processes. This meant that slow change became slower as decision-making became more complex. This logjam would be further exacerbated if ‘estates’, the departments managing, commissioning, renovating and remodelling teaching spaces, were also involved, requiring additional consultation and approval.

Twenty or thirty years later, inspection of many VLEs would reveal that they are still used mostly as repositories for notes and slide decks, for assessment hand-in alerts and for the digital submission of assessments, partly necessitated by the need for plagiarism detection.

In short, the pedagogies intended were not the pedagogies being enacted. 

The Industrialisation of Education

Educational institutions were, however, driven by wider societal, financial and political pressures, not just educational or technological factors or fashions.

From a political and financial perspective, there was less public money, as a consequence of the 2008 global financial crisis (‘subprime mortgages’, remember those?). There was also less commitment to a vision of publicly funded education as some vague liberal public good, and instead a shift towards viewing it as a mechanism among competing free-market providers to put more and more trained graduates straight into jobs, leading to enormous pressures to increase throughput, maintain cost-efficiency and ensure consistency. 

Faced with these kinds of pressures, digital learning could save the day, and in effect,  education became electrified and industrialised, with increased throughput now based on a production line of rooms full of networked desktop computers.

What Changed?

What changed in the first decade of the current century was the emergence and growth, and then the universality and ubiquity of personal networked digital technologies, notably the mobile (smart) phone, with all its functions for capturing context and content, as well as tablets and laptops. Alongside this was the rise of web2.0 applications and social media in all its different forms, such as Facebook and WeChat. This also includes blogs, podcasts, video and image sites such as YouTube, Flickr, Instagram, as well as question-and-answer sites such as Reddit and Quora and information and knowledge sites such as Wikipedia and its offshoots. 

All of these empowered users to upload their ideas, images, information and opinions, and to share, discuss, transform, merge, broadcast and discard them on a massive and unprecedented scale. In short, people and communities, not organisations and institutions, could manage, own and control their learning, adapted to whatever style they preferred, at whatever time and with whatever technology they preferred.

The New Knowledge Economy

In a sense, we are describing the transition from knowledge produced in a centralised top-down, centre-out web1.0 fashion from a small number of official producers, namely publishers, universities, ministries and broadcasters using a handful of broadcast technologies, to a flatter, peer-to-peer web 2.0 cottage industry of individuals and communities giving or bartering the knowledge they produce using or appropriating any technologies that are cheap, accessible and familiar. 

It would be a mistake to portray this as democratisation of digital learning, given the ownership, control and politics of these technologies, but it is perhaps demotic. Nor should we assume that this is good or useful knowledge, only that there is a lot of it, some of it faulty, some of it harmful.

Nevertheless, the ‘locus of control’ seems to shift from clearly defined professional teachers to vaguely defined amateur learners. 

The Chasm

So what we have is a chasm between the managed digital environments and contents of formal education, quality assured, professionally managed, and the anarchic and potentially dangerous chaos of informal digital learning. One critique of formal education might be the lack of preparation received by learners in making the transition from one to the other, from being taught, whatever exactly that might mean, to becoming competent, critical, lifelong learners. Another critique, given the sluggishness of education systems in recognising and responding to change in the outside world, is the threat to their credibility. 

Tradition, Nothing More?

Perhaps, this is the wrong argument. In formal education, students wear gowns, write sit-down exams with a pen and learn from a VLE. It has nothing to do with employment skills; it is traditional, and the students collect a certificate if they do it all correctly, just a ‘rite de passage’, a rubber stamp. Perhaps students in education systems do not just learn what they are taught but something else, perhaps independence, socialisation and various other attributes described as maturity? That rather depends on their experiences within fragmented and unstable education systems.

Many years ago, at the dawn of mobile learning, a conference panel were asked, given this kind of analysis, what is now the role of universities? One answer from the panel was: ‘We give degrees.’ With increasing concern about student loans and the graduate premium, if any, this may not be such a great answer. 

Why Does This Matter

It matters because education, or rather learning, matters and the chasm represents a challenge and an opportunity, one that can make or mar economic and social wellbeing for people and communities outside – or probably, inside – formal education. So now, a bigger challenge and opportunity is with us. 

Of course, while many of the players in this discussion have been emerging over the past two decades, a new player suddenly appeared about three years ago, accessible conversational AI, generative artificial intelligence, mediated a chatbot on every laptop, tablet and phone. So we are obliged to ask whether this makes a qualitative difference to the argument or merely an enormous quantitative one. That may be a bigger question than can be addressed here and now, as we see AI haphazardly deployed in formal education and amongst informal learners, presumably changing what needs to be learnt in societies permeated by AI and how it could be learnt.

So, in the meantime, is there some third space, between the risky anarchy of the web and the managed conservatism of formal digital? And if there is such a third space, how does it represent a scalable and sustainable environment?

It is axiomatic that this informal digital learning that we describe rests on familiar, accessible and cheap technologies, both hardware and software, devices and networks, that give users control and confidence, ownership and autonomy. At first sight, there is no business model here; the system is self-sustaining and self-contained. There is, however, for both the individual and collective good, a need for support to nudge users towards efficient and benign learning and away from harmful and wasteful learning. 

The Skills This Now Requires

Criticality would be a key skill, helping users tell good from bad, useful from useless, find the digital tools, content and communities that suit them, and help them question, enquire, scrutinise, and critique what they are getting, who they are getting it from and what choices they have. 

Curation is another key skill, helping users find, evaluate, select, organise and classify the digital tools, content and communities that suit them. Finally, metacognition and reflection will help users understand and improve their own digital learning, digital ethics and digital relationships. 

With some imagination, it should be possible for the edtech industry to develop and populate this third space, a permeable space between the resources of formal education and the freedoms of cyberspace, between the lucrative platforms of the one and the lucrative platforms of the other.

About Avallain

For more than two decades, Avallain has enabled publishers, institutions and educators to create and deliver world-class digital education products and programmes. Our award-winning solutions include Avallain Author, an AI-powered authoring tool, Avallain Magnet, a peerless LMS with integrated AI, and TeacherMatic, a ready-to-use AI toolkit created for and refined by educators.

Our technology meets the highest standards with accessibility and human-centred design at its core. Through Avallain Intelligence, our framework for the responsible use of AI in education, we empower our clients to unlock AI’s full potential, applied ethically and safely. Avallain is ISO/IEC 27001:2022 and SOC 2 Type 2 certified and a participant in the United Nations Global Compact.

Find out more at avallain.com

_

Contact:

Daniel Seuling

VP Client Relations & Marketing

dseuling@avallain.com

Simply asking questions is not enough to understand the people we serve. Effective research and responsible design require more thoughtful, grounded ways of getting to know users, learners and clients. In this piece, Prof. John Traxler reflects on the limitations of familiar methods and explores alternative approaches to uncover values, feelings and knowledge that are often difficult to articulate. He further examines the ethical and methodological assumptions that shape how we gather and interpret insight.

Exploring People’s Values, Feelings and Knowledge Beyond Traditional Methods

Author: Prof. John Traxler, UNESCO Chair, Commonwealth of Learning Chair and Academic Director of the Avallain Lab

The Challenge

St. Gallen, February 20, 2026 – We all want, or we all need to know about the feelings, knowledge and values of other people. So how do we get answers? We ask them questions. Is this a good idea? No, rarely, and this piece explains why and the broader scope of any findings or conclusions. If it were even possible that the people concerned gave accurate, complete and trustworthy answers, does this tell us anything at all about the views, feelings or knowledge of any other people, in any other place, at any other time?

Someone once observed that much accepted psychological theory is based on research using psychology undergraduate subjects because such subjects are the nearest, cheapest and easiest for university-based academics conducting psychological research. That is not necessarily a good basis for theories supposedly applicable to the rest of humanity. At an early age, Freud supposedly explained our inner workings, but did so based on a small number of case studies of some very ill people. Not a good evidence base.

The Usual Suspects and Their Defects

The ‘usual suspects’ are roughly interviews, semi-structured or otherwise, questionnaires, focus groups and surveys. They get routinely rounded up whenever anyone has a question that needs an answer. They do, however, have two overall problems, namely, firstly, that they will only get the answer to the question that they asked, nothing else, nothing more important, nothing more relevant, and secondly, the question or the questioning may be so flawed that they do not even really get that. 

To be more specific, the people answering the question may not know what it means; they may misunderstand or misinterpret it; they may be uncomfortable answering it, uncomfortable disclosing their ignorance of the answer or uncomfortable with its implications; they may mishear or misread the question. They may be consciously or unconsciously needing to perform a particular identity or persona, to appear as knowledgeable, affable, professional, naive, superior, cautious, flirtatious, relaxed, important, distant or rushed, depending on the context and depending on their psychological needs and preferences, and thus only provide answers in line with that performance. These may only lead to changes in emphasis or tone, but can still be highly significant.

Why Answers Cannot Be Taken at Face Value

Examples come easily. People who use pornography but won’t admit to it, people who tell me my lecture was great but tell each other it was rubbish; people who give me any answer for fear of disappointing me with no answer, people who rush to reach the end of the survey or the end of the interview; people who don’t want to be different or conversely do really want to be different, and so on. In essence, people are not machines, and neither are these methods objective or scientific.

To take a step back from questions, we need to think about the different kinds of thoughts and feelings that people have and thus try to match our methods of enquiry to those different kinds of thoughts and feelings. Finding out about people’s aspirations is not the same as finding out about their height; finding out about the future is not the same as finding out about the past; finding out about their habits is not the same as finding out about their worries. 

Furthermore, we all know things without realising we know them or without being able to clearly express them; being able to change manual gear, lace your shoes or play the guitar does not mean being able to recollect or explain them, they are intuitive, tacit or compiled; some activities and assumptions are unconscious or ‘hard-wired’ and a question will not produce a useful account or explanation. This means that questioning is not always effective, and a portfolio of alternative methods is needed, each appropriate to the type of knowledge, feeling or value being explored. We briefly mention some later, but in the context of procurement, perhaps models, role-play, simulation or prototypes are a more useful way of eliciting requirements than merely asking clients what they would like.

All of these concerns worsen as we attempt to question people more distant or different from ourselves, as do the ethical concerns, which is another reason for exploring a portfolio of alternative methods.

Methodological Limits and Ethical Concerns

The commonly used methods are not only methodologically problematic, in the sense that they are not necessarily trustworthy, but also ethically problematic. They privilege and empower the questioner, turning the people involved into passive data sources, and the greater the distance, the difference and the differential between the questioner and the people answering, the greater this ethical problem. Think only of middle-class professionals questioning working-class people, the employed questioning the jobless, men questioning women, Europeans questioning Africans, the neurotypical questioning the neurodiverse, the settled questioning the nomadic, the urban questioning the rural, the affluent questioning the poor and many other comparable dichotomies. These may be generalisations or simplifications, but the problem should be apparent even in less blatant situations.

There are a variety of common mistakes. Quantitative findings, based on statistics, usually provide precise percentage figures while overlooking small sample sizes and confounding contextual factors. Whilst qualitative findings based on interviews or focus groups can depend merely on ‘cherry-picking’ the most attractive and agreeable quotes to make their case.

There are tactical improvements, perhaps making the best of a bad job. The literature on interview structuring and questionnaire design can give enormous amounts of guidance. ‘Start with easy topics, don’t be too challenging too soon.’ ‘Don’t ask questions that are double negatives.’ ‘Don’t ask questions that have multiple clauses’, such as ‘do you like apples and oranges?’ or ‘do you not dislike pears?’. It is also important to think about changing the delivery or the setting to adapt to the barriers or challenges, and think about a proxy for the researcher nearer to the class or culture of the research participants.

The Usual Suspects and the Alternatives

OK, so if the ‘usual suspects’ are methodologically and ethically problematic, are there any alternatives? More to the point, are there any established, efficient, cheap and trustworthy alternatives? Luckily, the answer is ‘yes’, but context is the caveat and expertise and experience might be the prerequisites. By context, we mean that one-size-fits-all will not work; thus, expertise and experience are needed to make choices, allowances and adaptations that are context- and circumstance-dependent.

We can provide examples, but the underlying motivation is to provide a space and opportunity for people’s thoughts and feelings to emerge as candidly as possible. One stance that can help with this is Personal Construct Theory, PCT. This suggests that people are like scientists, creating unique mental frameworks called constructs, ways of seeing the world, to interpret and predict events in their lives, however trivial, embarrassing, superstitious, irrational, or mundane. Behaviour stemming from these personal understandings rather than from objective reality, these are ways to make a bit of sense of the worlds in which each lives. This, in turn, leads to a range of tools and techniques to elicit personal constructs and gain small insights into how each person understands the world.

Card sorts are one such tool, in which individuals repeatedly sort cards of images or ideas to identify underlying clusterings in how they perceive or apprehend them, without being asked for any rationalisation, explanation or justification. Card sorts, despite or because of their simplicity and efficiency, have an established track record in designing products and websites, accessing preferences and reactions that people cannot necessarily easily articulate. Laddering is a companion follow-up technique that repeatedly asks ‘why?’ to uncover the deeper foundations of preferences for the clusterings. Again, efficient, effective and cheap. Both are only slightly more sophisticated than these explanations suggest, but still ethically more acceptable than the ’usual suspects’ since the explanation is also not much more sophisticated, and consent really is ‘informed’.

Alternative Tools, Formats and Settings

There are others, from other academic or commercial sources, rich pictures, a way of community members or organisation members, say employees or clients, expressing alliances, affiliations, antagonisms, transactions and relationships, with just cartoonish drawings. Without them, any survey or focus group might be hopelessly naive about what is bubbling away under the surface.

Tackling the challenge from a different direction, it can be worth asking whether the formats by which communities or cultures interact and discuss might map onto a format that we as European researchers would already recognise; is the talking circle near enough to a focus group, for example, and can we meet in the middle with a little adaptation? 

This suggests, of course, that the surroundings as well as the format are important, some more naturalistic, informal and relaxing than, say, a university office, interview room or laboratory, especially when video or audio recording can add extra intimidation. Some market researchers, for example, testing television advertisements, will mock up the apparently authentic living room of the target demographic audiences. This would be complete with a TV in the corner, pictures on the walls, tatty sofas, chairs, coffee tables and hidden cameras, before recruiting families to inhabit this ersatz living room and watch television programmes and un-self-consciously, the proposed advertisements.

So What Have We Learnt?

Clearly, don’t just round up the ‘usual suspects’. More positively, think about the nature of your enquiry and the nature of the people who can help with it. Consider the findings and your claims and avoid overselling them. Be brave, be eclectic, experiment, reflect and adapt, but build on what others have done and ask why they did it. It would be unwise not to explore how the expertise and experience captured, albeit imperfectly, by AI might at least allow us to explore permutations and possibilities, nudging our imaginations.

These efforts are, in the end, about understanding our users, learners and clients more fully in order to respond to them more appropriately and responsibly.

About Avallain

For more than two decades, Avallain has enabled publishers, institutions and educators to create and deliver world-class digital education products and programmes. Our award-winning solutions include Avallain Author, an AI-powered authoring tool, Avallain Magnet, a peerless LMS with integrated AI, and TeacherMatic, a ready-to-use AI toolkit created for and refined by educators.

Our technology meets the highest standards with accessibility and human-centred design at its core. Through Avallain Intelligence, our framework for the responsible use of AI in education, we empower our clients to unlock AI’s full potential, applied ethically and safely. Avallain is ISO/IEC 27001:2022 and SOC 2 Type 2 certified and a participant in the United Nations Global Compact.

Find out more at avallain.com

_

Contact:

Daniel Seuling

VP Client Relations & Marketing

dseuling@avallain.com