Exclusionary Evaluations of Participatory Design?

This week’s paper, ‘Evaluation in Participatory Design: A Literature Survey’ by Bossen et al. starts by introducing the core values of participatory design (PD) as inherently human-centric, heralding mutual learning, empowerment and democratisation as fundamental cornerstones to this diverse field. Indeed, many of the methods entrenched in the PD field reflect these values (i.e. workshops, prototyping etc.). However, this paper questions whether our approaches to evaluating PD processes retain these core values; do we adhere to the same values in our reflections on PD or are our evaluations in fact exclusionary?

In their attempts to answer this question, the authors undertook a systematic review of past literature that deals specifically with evaluation of PD products, processes and/or outcomes. The results of this review found that, although there is an inherent level of evaluation inbuilt into PD, there is a significant dearth of papers in this field that formally or explicitly deals with evaluations. Unlike some of the more established fields of Public Health, education etc. that have followed a wider societal push for increased audit and evaluation, PD has yet to develop a consensus on how we assess the relative “success” of PD methods.

The paper makes a number of conclusions about pre-existing evaluations in PD and highlights several observed weaknesses. Firstly they suggest that a lack of methodological clarity makes formal evaluation challenging at best. Equally a lack of cross-referencing of similar evaluative methods from other studies means that many evaluations appear to exist completely independently of one another. Papers were also found to make little use of pre-existing evaluation literature and theory. These weaknesses suggest a need for improved clarity and improvements to the writing culture of evaluative papers so as to facilitate the potential development of a universally-accepted corpus of formal evaluative techniques within the PD field.

Despite an apparent lack of referral to existing evaluation theory in papers, the study found that this is not due to any absence of such work. In fact, they found several existing frameworks for evaluation that provided potential models for future evaluations. Despite this, it was observed that papers seldom utilised these models within their own work. Why this is the case is unclear but the authors speculate it may be to do with how recently these frameworks have been published. The paper suggests that by evading explicit and systematic evaluation frameworks in such a way, researchers are missing opportunities for learning and knowledge generation and instead limit reflections to the subjective and anecdotal. This is problematic when trying to establish methodological robustness and validity.

Finally, and perhaps for me most poignantly, the paper highlights how the core values that are the driving force behind the PD process (empowerment, democratisation and mutual learning) are not adopted in evaluations of these very processes. Summary statements such as “the users were happy” are frequently used, yet it is seldom explained how this conclusion was made. Were participants in the PD process also given voice in the PD evaluation or is this “happiness” the opinion of the researcher? By speaking on behalf of those involved, are researchers going against the very principles that underpin PD as a whole?

Whilst the authors acknowledge implicit evaluative efforts in previous PD evaluation, they conclude by calling for more explicit and systematic evaluations within the field so as to improve learning and subsequently mature PD methods in the future. In doing so, it may allow for the adoption of PD methods beyond the research field.

The paper I have chosen that exemplifies Participatory design is “Creating Creative Spaces for Co-Designing with Autistic Children – The Concept of a “Handlungsspielraum”” by Makhaeva, Frauenberger and Spiel.