Comparison of Paper- and Tool-based Participatory Design Approaches: A Case Study with PDotCapturer

Heintz, Matthias; Law, Effie; Andrade, Pamela

doi:10.14236/ewic/HCI2018.29

Abstract

In the current Participatory Design (PD) practice, paper-based approaches are commonly applied. However, the use of software tools can potentially enhance the PD process and outcome, given their flexibility for prototype presentation, data storage, retrieval and analysis. Driven by this assumption, we developed the specific-purpose PD online tool PDotCapturer. To evaluate the effectiveness of PDotCapturer, as compared with a paper-based approach, we conducted a within-subjects study where 33 participants were asked to provide feedback on e-learning prototypes, using either paper or PDotCapturer in one session and vice-versa in another session. The evaluation criteria were quantity and quality of the participants’ comments. For qualitative data analysis, we applied the refined coding scheme CAt+. Results show that using PDotCapturer partly led to feedback of higher quality, but the number of comments was higher for the paper-based approach. Our work contributes to the HCI community by augmenting the repertoire of the PD toolbox.

Content

Author and article information

Contributors

Matthias Heintz

Effie Law

Pamela Andrade

Conference

Publication date: July 2018

Publication date (Print): July 2018

Pages: 1-12

Affiliations

[1 ] University of Leicester

University Road, Leicester

LE1 7RH, United Kingdom

Article

DOI: 10.14236/ewic/HCI2018.29

SO-VID: c7db686a-ce22-446a-a669-194e75b84de9

License:

This work is licensed under a Creative Commons Attribution 4.0 Unported License. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/

Conference name: Proceedings of the 32nd International BCS Human Computer Interaction Conference

Conference acronym: HCI

Conference number: 32

Conference location: Belfast, UK

Conference date: 4 - 6 July 2018

Conference sponsor: Electronic Workshops in Computing (eWiC)

Conference theme: Human Computer Interaction Conference

History

Product

1477-9358 BCS Learning & Development

Self URI (article page): https://www.scienceopen.com/hosted-document?doi=10.14236/ewic/HCI2018.29

Self URI (journal page): https://ewic.bcs.org/

Categories

Series title: Electronic Workshops in Computing

ScienceOpen disciplines: Applied computer science,Computer science,Security & Cryptology,Graphics & Multimedia design,General computer science,Human-computer-interaction

Keywords: Participatory Design,PDotCapturer,paper,tool,comparison,CAt+

D. G. Altman Practical statistics for medical research Chapman and Hall, London 1991
B. P. BaileyJ. A. Konstan Are informal tools better?: comparing DEMAIS, pencil and paper, and authorware for early multimedia design In Proc. of SIGCHI conference on human factors in computing systems, ACM Press 2003 313 320
Balsamiq. Rapid, effective and fun wireframing software. | Balsamiq http://balsamiq.com/
K. Crewe The Quality of Participatory Design: The Effects of Citizen Input on the Design of the Boston Southwest Corridor Journal of the American Planning Association 67 4 2001 437 455
T. de JongS. SotiriouD. Gillet Innovations in STEM education: the Go-Lab federation of online labs Smart Learning Environments 1 1 2014 1 16
J. L. FleissJ. Cohen The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability Educational and Psychological Measurement 33 1973 613 619
J. L. FleissB. LevinM. C. Paik Statistical methods for rates and proportions 3 Hoboken John Wiley & Sons 2003
GWT Project http://www.gwtproject.org/
K. A. Hallgren Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial Tutorials in quantitative methods for psychology 8 1 2012 23 34
M. HeintzE. L. Law-C., S. GovaertsA. HolzerD. Gillet Pdot: participatory design online tool In CHI ’14 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’14), ACM Press 2014 2581 2586
M. HeintzE. L.-C. LawS. Heintz Review of Online Tools for Asynchronous Distribute Online Participatory Design. In: Dominik Renzel, Ralf Klamma (eds.), “Large-Scale Social Requirements Engineering” IEEE Computer Society Special Technical Community on Social Networking E-Letter 2 3 September 2014 2014
M. HeintzE. L.-C. LawS. Soleimani Paper or Pixel? Comparing Paper- and Tool-Based Participatory Design Approaches In Human-Computer Interaction – INTERACT 2015 Springer International Publishing 2015 501 517
C. HundhausenS. TrentA. BalkarM. Nuur The design and experimental evaluation of a tool to support the construction and wizard-of-oz testing of low fidelity prototypes In Visual Languages and Human-Centric Computing, 2008. VL/HCC 2008. IEEE Symposium on, IEEE 2008 86 90
R. J. K. JacobA. GirouardL. M. HirshfieldM. S. HornO. ShaerE. T. SoloveyJ. Zigelbaum Reality-based interaction: a framework for post-WIMP interfaces In Proc. CHI 2008, ACM Press 2008 201 210
J. KindredS. N. Mohammed “He Will Crush You Like an Academic Ninja!”: Exploring Teacher Ratings on Ratemyprofessors. com. Journal of Computer‐Mediated Communication 10, 3 2005
K. D. KöningsS. Brand-GruwelJ. J. G. van Merriënboer An approach to participatory instructional design in secondary education: an exploratory study Educational Research 52 1 2010 45 59
F. MacdonaldJ. Miller A comparison of tool-based and paper-based software inspection Empirical Software Engineering 3, 3 1998 233 253
A. MaddenI. RuthvenD. McMenemy A classification scheme for content analyses of YouTube video comments Journal of documentation 69, 5 2013 693 714
J. MaltbyL. Day Early success in statistics. Pearson Education 2002
A. M. NaghshA. Dearden GABBEH: A tool to support collaboration in electronic paper prototyping. CSCW 2004 2004
C. NeustaedterA. J. Bernheim Brush “LINC-ing” the family: the participatory design of an inkable family calendar In Proc. CHI 2006, ACM Press 2006 141 150
M. W. NewmanJ. LinJ. I. HongJ. A. Landay DENIM: An informal web site design tool inspired by observations of practice Human-Computer Interaction 18, 3 2003 259 324
J. Nielsen 1994a Heuristic evaluation Usability inspection methods 17 1 25 62
Painless Remote UX | myBalsamiq http://www.mybalsamiq.com/
E. B.-N. SandersE. BrandtT. Binder A framework for organizing the tools and techniques of participatory design In Proc. PDC 2010, ACM Press 2010 195 198
V. C. V. B. SeguraS. D. J. BarbosaF. P. Simões UISKEI: a sketch-based prototyping tool for defining and evaluating user interface behavior In Proc. of the International Working Conference on Advanced Visual Interfaces, ACM Press 2012 18 25
S. StumpfV. RajaramL. LiM. BurnettT. DietterichE. SullivanR. DrummondJ. Herlocker Toward harnessing user feedback for machine learning In Proc. of the 12th international conference on Intelligent user interfaces, ACM Press 2007 82 91
H.-H. TeoL.-B. OhC. LiuK.-K. Wei An empirical study of the effects of interactivity on web user attitude International Journal of Human-Computer Studies 58, 3 2003 281 305
G. WalshA. DruinM. L. GuhaE. BonsignoreE. FossJ. C. YipE. GolubT. CleggQ. BrownR. BrewerA. JoshiR. Brown DisCo: a co-design online tool for asynchronous distributed child and adult design partners In Proc. of the 11th International Conference on Interaction Design and Children, ACM Press 2012 11 19
G. WalshA. DruinM. L. GuhaE. FossE. GolubL. HatleyE. BonsignoreS. Franckel Layered elaboration: a new technique for co-design with children In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, ACM Press 2010 1237 1240
G. WalshE. FossJ. YipA. Druin FACIT PD: a framework for analysis and creation of intergenerational techniques for participatory design In Proc. CHI 2013, ACM Press 2013 2893 2902

Comments

[1] D. G. Altman Practical statistics for medical research Chapman and Hall, London 1991

[2] B. P. BaileyJ. A. Konstan Are informal tools better?: comparing DEMAIS, pencil and paper, and authorware for early multimedia design In Proc. of SIGCHI conference on human factors in computing systems, ACM Press 2003 313 320

[3] Balsamiq. Rapid, effective and fun wireframing software. | Balsamiq http://balsamiq.com/

[4] K. Crewe The Quality of Participatory Design: The Effects of Citizen Input on the Design of the Boston Southwest Corridor Journal of the American Planning Association 67 4 2001 437 455

[5] T. de JongS. SotiriouD. Gillet Innovations in STEM education: the Go-Lab federation of online labs Smart Learning Environments 1 1 2014 1 16

[6] J. L. FleissJ. Cohen The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability Educational and Psychological Measurement 33 1973 613 619

[7] J. L. FleissB. LevinM. C. Paik Statistical methods for rates and proportions 3 Hoboken John Wiley & Sons 2003

[8] GWT Project http://www.gwtproject.org/

[9] K. A. Hallgren Computing Inter-Rater Reliability for Observational Data: An Overview and Tutorial Tutorials in quantitative methods for psychology 8 1 2012 23 34

[10] M. HeintzE. L. Law-C., S. GovaertsA. HolzerD. Gillet Pdot: participatory design online tool In CHI ’14 Extended Abstracts on Human Factors in Computing Systems (CHI EA ’14), ACM Press 2014 2581 2586

[11] M. HeintzE. L.-C. LawS. Heintz Review of Online Tools for Asynchronous Distribute Online Participatory Design. In: Dominik Renzel, Ralf Klamma (eds.), “Large-Scale Social Requirements Engineering” IEEE Computer Society Special Technical Community on Social Networking E-Letter 2 3 September 2014 2014

[12] M. HeintzE. L.-C. LawS. Soleimani Paper or Pixel? Comparing Paper- and Tool-Based Participatory Design Approaches In Human-Computer Interaction – INTERACT 2015 Springer International Publishing 2015 501 517

[13] C. HundhausenS. TrentA. BalkarM. Nuur The design and experimental evaluation of a tool to support the construction and wizard-of-oz testing of low fidelity prototypes In Visual Languages and Human-Centric Computing, 2008. VL/HCC 2008. IEEE Symposium on, IEEE 2008 86 90

[14] R. J. K. JacobA. GirouardL. M. HirshfieldM. S. HornO. ShaerE. T. SoloveyJ. Zigelbaum Reality-based interaction: a framework for post-WIMP interfaces In Proc. CHI 2008, ACM Press 2008 201 210

[15] J. KindredS. N. Mohammed “He Will Crush You Like an Academic Ninja!”: Exploring Teacher Ratings on Ratemyprofessors. com. Journal of Computer‐Mediated Communication 10, 3 2005

[16] K. D. KöningsS. Brand-GruwelJ. J. G. van Merriënboer An approach to participatory instructional design in secondary education: an exploratory study Educational Research 52 1 2010 45 59

[17] F. MacdonaldJ. Miller A comparison of tool-based and paper-based software inspection Empirical Software Engineering 3, 3 1998 233 253

[18] A. MaddenI. RuthvenD. McMenemy A classification scheme for content analyses of YouTube video comments Journal of documentation 69, 5 2013 693 714

[19] J. MaltbyL. Day Early success in statistics. Pearson Education 2002

[20] A. M. NaghshA. Dearden GABBEH: A tool to support collaboration in electronic paper prototyping. CSCW 2004 2004

[21] C. NeustaedterA. J. Bernheim Brush “LINC-ing” the family: the participatory design of an inkable family calendar In Proc. CHI 2006, ACM Press 2006 141 150

[22] M. W. NewmanJ. LinJ. I. HongJ. A. Landay DENIM: An informal web site design tool inspired by observations of practice Human-Computer Interaction 18, 3 2003 259 324

[23] J. Nielsen 1994a Heuristic evaluation Usability inspection methods 17 1 25 62

[24] Painless Remote UX | myBalsamiq http://www.mybalsamiq.com/

[25] E. B.-N. SandersE. BrandtT. Binder A framework for organizing the tools and techniques of participatory design In Proc. PDC 2010, ACM Press 2010 195 198

[26] V. C. V. B. SeguraS. D. J. BarbosaF. P. Simões UISKEI: a sketch-based prototyping tool for defining and evaluating user interface behavior In Proc. of the International Working Conference on Advanced Visual Interfaces, ACM Press 2012 18 25

[27] S. StumpfV. RajaramL. LiM. BurnettT. DietterichE. SullivanR. DrummondJ. Herlocker Toward harnessing user feedback for machine learning In Proc. of the 12th international conference on Intelligent user interfaces, ACM Press 2007 82 91

[28] H.-H. TeoL.-B. OhC. LiuK.-K. Wei An empirical study of the effects of interactivity on web user attitude International Journal of Human-Computer Studies 58, 3 2003 281 305

[29] G. WalshA. DruinM. L. GuhaE. BonsignoreE. FossJ. C. YipE. GolubT. CleggQ. BrownR. BrewerA. JoshiR. Brown DisCo: a co-design online tool for asynchronous distributed child and adult design partners In Proc. of the 11th International Conference on Interaction Design and Children, ACM Press 2012 11 19

[30] G. WalshA. DruinM. L. GuhaE. FossE. GolubL. HatleyE. BonsignoreS. Franckel Layered elaboration: a new technique for co-design with children In Proc. of the SIGCHI Conference on Human Factors in Computing Systems, ACM Press 2010 1237 1240

[31] G. WalshE. FossJ. YipA. Druin FACIT PD: a framework for analysis and creation of intergenerational techniques for participatory design In Proc. CHI 2013, ACM Press 2013 2893 2902

UK Computing Summit 2025: Navigating change (surviving and beyond) - 29-30 April @ Sheffield Hallam University - Register here.