[MyResearch] ANOVA not suited for the analysis of intragroup

Very interesting point by JAN-WILLEM STRIJBOS in his paper THE EFFECT OF FUNCTIONALROLES ON GROUP EFFICIENCY: Using Multilevel Modeling and Content Analysis to Investigate Computer-Supported Collaboration in Small Groups:

it is important to note the implications of nonindependent observations with respect to the analysis of intragroup collaboration. This issue was only recently raised in CSCL and small group research. In research on cooperative learning, frequently the ANOVA procedure has been used to investigate the impact of an instructional strategy using individual level observations (see Slavin, 1995). This is no exception in some CSCL studies (Reiserer, Ertl, & Mandl, 2002). However, ANOVA appears not to be suited for this type of data. Stevens (1996) points out that the assumption of independence, between scores of members of the same small group, is violated. Students’ perceptions of group performance depends on all other members’ activities. Violation of independence increases as a function of the interdependence in a group, thus yielding a major increase of a Type 1 error. Stevens (1996) suggest either to test with a stricter level of significance (p below .01 or even p below .001) or to use the group average. Bonito (2002) discusses three alternative procedures that take nonindependence into account, with respect to the analysis of participation in small groups: the actor-partner interdependence model, the social relations model, and multilevel modeling (MLM).

Another reason why he advocates for multilevel modeling (MLM) is:

Another point is that, unlike a considerable amount of studies in social psychology, CSCL is not conducted in laboratory settings. Its naturalistic context adds to its ecological validity but simultaneously complicates analysis. Most CSCL studies suffer from a relatively small number of participants, and research designs in general do not exceed 20 participants (see Stahl, 2002). Furthermore, quantitative statistical analyses are rarely used. Analysis focuses on qualitative methodologies to explore intragroup interaction and the level of collaboration.MLMappears to be best suited to investigate questionnaire data that consists of self-report perceptions (cf. Bonito, 2002). However, MLM analyses with a small sample size (less than 50) are not often reported. Therefore, the methodological and analytical considerations will be discussed in more detail in the Method and Results section that covers the MLM analyses.