From Mice to Men – 24 years of Evaluation in CHI by Louise Barkhuus and Jennifer A. Rode, alt.chi 2007. The paper reports a comprehensive analysis of how practitioners and academics have employed evaluation in CHI through the past 24 years: what are the roles of evaluation? how it evolved? what type of subjects had been used? Concerning the evolution, it's good to read it in conjunction of The Evolution of Evaluation by Joseph Kaye and Phoebe Sengers: both shows how things evolve from experimental psychology then usability to more ethnographic methodologies. Some of the findings/conclusions:
"By taking a closer look at the number of subjects used in quantitative and qualitative evaluation respectively, we find an interesting trend. The median number of subjects in the quantitative empirical studies has decreased over time, and the median number of subjects in qualitative studies seems to have increased (...) Many traditional psychology experiments often use students as their main population. (...) Not surprisingly, HCI research followed suit. (...) Now that computers are more widespread and many applications are targeting a diverse set of people, students have too much computer savvy to be representative of the entire spectrum of novice to expert users. Moreover, them being in an educational setting and used to learning new things makes them unusual in terms of ability to learn. (...) many studies failed to use a gender-balanced sample. (...) The diversity of evaluation methods also comes into play with the decrease in papers presenting evaluation methods themselves. (...) the proportion of CHI work that discusses the role of evaluation is very small, which makes us question whether evaluation is in fact responding dynamically to the radical changes in technological innovation. (...) A positive trend that we observed was a recent increase in qualitative evaluation studies, studies often taking place over longer time and using multiple sets of inquiry methods. This is a trend that illustrates how evaluation is not just a validation tool. It provides us with indications of user appropriation and contextual fitting of the technology in question. (...) Ethnographies, for example, can provide insight into situated technology use and a social setting"
Why do I blog this? I like this kind of articles, they give an interesting overview of the research field, pointing at important elements (student-computer interactions, bad gender balance). Besides, given that I am doing "evaluation" research, it's important to understand the evolutions and how they are tackled or considered.