The Library

Model evaluation using grouped or individual data

Tools

Cohen, Andrew L., Sanborn, Adam N. and Shiffrin, Richard M.. (2008) Model evaluation using grouped or individual data. Psychonomic Bulletin & Review, Vol.15 (No.4). pp. 692-712. ISSN 1069-9384

Full text not available from this repository.

Abstract

Analyzing the data of individuals has several advantages over analyzing the data combined across the individuals (the latter we term group analysis): Grouping can distort the form of data, and different individuals might perform the task using different processes and parameters. These factors notwithstanding, we demonstrate conditions in which group analysis outperforms individual analysis. Such conditions include those in which there are relatively few trials per subject per condition, a situation that sometimes introduces distortions and biases when models are fit and parameters are estimated. We employed a simulation technique in which data were generated from each of two known models, each with parameter variation across simulated individuals. We examined how well the generating model and its competitor each fared in fitting (both sets of) the data, using both individual and group analysis. We examined the accuracy of model selection (the probability that the correct model would be selected by the analysis method). Trials per condition and individuals per experiment were varied systematically. Three pairs of cognitive models were compared: exponential versus power models of forgetting, generalized context versus prototype models of categorization, and the fuzzy logical model of perception versus the linear integration model of information integration. We show that there are situations in which small numbers of trials per condition cause group analysis to outperform individual analysis. Additional tables and figures may be downloaded from the Psychonomic Society Archive of Norms, Stimuli, and Data, www.psychonomic.org/archive.

Item Type: Journal Article
Subjects: B Philosophy. Psychology. Religion > BF Psychology
Divisions: Faculty of Science > Psychology
Library of Congress Subject Headings (LCSH): Cognitive psychology, Human information processing
Journal or Publication Title: Psychonomic Bulletin & Review
Publisher: Springer New York LLC
ISSN: 1069-9384
Date: 2008
Volume: Vol.15
Number: No.4
Page Range: pp. 692-712
Identification Number: 10.3758/PBR.15.4.692
Status: Peer Reviewed
Funder: National Institute of Mental Health (U.S.)
Grant number: 1 R01 MH12717 (NIMH), 1 R01 MH63993 (NIMH)
References: Akaike, H. (1973). Information theory as an extension of the maximum likelihood principle. In B. N. Petrov & F. Csaki (Eds.), Second International Symposium on Information Theory (pp. 267-281). Budapest: Akademiai Kiado. Anderson, N. H. (1981). Foundations of information integration theory. New York: Academic Press. Anderson, R. B., & Tweney, R. D. (1997). Artifactual power curves in forgetting. Memory & Cognition, 25, 724-730. Ashby, F. G., & Gott, R. E. (1988). Decision rules in the perception and categorization of multidimensional stimuli. Journal of Experimental Psychology: Learning, Memory, & Cognition, 14, 33-53. Ashby, F. G., & Maddox, W. T. (1993). Relations between prototype, exemplar, and decision bound models of categorization. Journal of Mathematical Psychology, 37, 372-400. Ashby, F. G., Maddox, W. T., & Lee, W. W. (1994). On the dangers of averaging across subjects when using multidimensional scaling or the similarity-choice model. Psychological Science, 5, 144-151. Barron, A. R., Rissanen, J., & Yu, B. (1998). The MDL principle in modeling and coding. IEEE Transactions on Information Theory, 44, 2743-2760. Berger, J. O., & Pericchi, L. R. (1996). The intrinsic Bayes factor for model selection and prediction. Journal of the American Statistical Association, 91, 109-122. Bower, G. H. (1961). Application of a model to paired-associate learning. Psychometrika, 26, 255-280. Browne, M. W. (2000). Cross-validation methods. Journal of Mathematical Psychology, 44, 108-132. Burnham, K. P., & Anderson, D. R. (1998). Model selection and inference: A practical information-theoretic approach. New York: Springer. Burnham, K. P., & Anderson, D. R. (2002). Model selection and multimodel inference: A practical information-theoretic approach (2nd ed.). New York: Springer. Busemeyer, J. R., & Wang, Y.-M. (2000). Model comparisons and model selections based on generalization criterion methodology. Journal of Mathematical Psychology, 44, 171-189. Cover, T. M., & Thomas, J. A. (1991). Elements of information theory. New York: Wiley. Estes, W. K. (1956). The problem of inference from curves based on group data. Psychological Bulletin, 53, 134-140. Estes, W. K., & Maddox, W. T. (2005). Risks of drawing inferences about cognitive processes from model fits to individual versus average performance. Psychonomic Bulletin & Review, 12, 403-408. Grünwald, P. [D.] (2000). Model selection based on minimum description length. Journal of Mathematical Psychology, 44, 133-152. Grünwald, P. D. (2005). Minimum description length tutorial. In P. D. Grünwald, I. J. Myung, & M. A. Pitt (Eds.), Advances in minimum description length: Theory and applications (pp. 23-80). Cambridge, MA: MIT Press. Grünwald, P. D. (2007). The minimum description length principle. Cambridge, MA: MIT Press. Hastie, T., Tibshirani, R., & Friedman, J. (2001). The elements of sta tistical learning: Data mining, inference, and prediction. New York: Springer. Hayes, K. J. (1953). The backward curve: A method for the study of learning. Psychological Review, 60, 269-275. Jeffreys, H. (1935). Some tests of significance, treated by the theory of probability. Proceedings of the Cambridge Philosophical Society, 31, 203-222. Jeffreys, H. (1961). Theory of probability (3rd ed.). Oxford: Oxford University Press. Karabatsos, G. (2006). Bayesian nonparametric model selection and model testing. Journal of Mathematical Psychology, 50, 123-148. Kass, R. E., & Raftery, A. E. (1995). Bayes factors. Journal of the American Statistical Association, 90, 773-795. Lagarias, J. C., Reeds, J. A., Wright, M. H., & Wright, P. E. (1998). Convergence properties of the Nelder–Mead simplex method in low dimensions. SIAM Journal of Optimization, 9, 112-147. Lee, M. D., & Webb, M. R. (2005). Modeling individual differences in cognition. Psychonomic Bulletin & Review, 12, 605-621. Mack, A., & Rock, I. (1998). Inattentional blindness. Cambridge, MA: MIT Press. Macmillan, N. A., & Creelman, C. D. (2005). Detection theory: A user’s guide (2nd ed.). Mahwah, NJ: Erlbaum. Massaro, D. W. (1998). Perceiving talking faces: From speech perception to a behavioral principle. Cambridge, MA: MIT Press. Medin, D. L., & Schaffer, M. M. (1978). Context theory of classification learning. Psychological Review, 85, 207-238. Minda, J. P., & Smith, J. D. (2002). Comparing prototype-based and exemplar-based accounts of category learning and attentional allocation. Journal of Experimental Psychology: Learning, Memory, & Cognition, 28, 275-292. Myung, I. J., Kim, C., & Pitt, M. A. (2000). Toward an explanation of the power law artifact: Insights from response surface analysis. Memory & Cognition, 28, 832-840. Navarro, D. J., Griffiths, T. L., Steyvers, M., & Lee, M. D. (2006). Modeling individual differences using Dirichlet processes. Journal of Mathematical Psychology, 50, 101-122. Navarro, D. J., Pitt, M. A., & Myung, I. J. (2004). Assessing the distinguishability of models and the informativeness of data. Cognitive Psychology, 49, 47-84. Nosofsky, R. M. (1986). Attention, similarity, and the identification– categorization relationship. Journal of Experimental Psychology: General, 115, 39-57. Nosofsky, R. M., & Zaki, S. R. (2002). Exemplar and prototype models revisited: Response strategies, selective attention, and stimulus generalization. Journal of Experimental Psychology: Learning, Memory, & Cognition, 28, 924-940. Oden, G. C., & Massaro, D. W. (1978). Integration of featural information in speech perception. Psychological Review, 85, 172-191. Reed, S. K. (1972). Pattern recognition and categorization. Cognitive Psychology, 3, 382-407. Rissanen, J. (1978). Modeling by the shortest data description. Automatica, 14, 465-471. Rissanen, J. (1987). Stochastic complexity. Journal of the Royal Statistical Society B, 49, 223-239, 252-265. Rissanen, J. (1996). Fisher information and stochastic complexity. IEEE Transactions on Information Theory, 42, 40-47. Rissanen, J. (2001). Strong optimality of the normalized ML models as universal codes and information in data. IEEE Transactions on Information Theory, 47, 1712-1717. Schwarz, G. (1978). Estimating the dimension of a model. Annals of Statistics, 6, 461-464. Shepard, R. N. (1987). Toward a universal law of generalization for psychological science. Science, 237, 1317-1323. Sidman, M. (1952). A note on functional relations obtained from group data. Psychological Bulletin, 49, 263-269. Siegler, R. S. (1987). The perils of averaging data over strategies: An example from children’s addition. Journal of Experimental Psychology: General, 116, 250-264. Wagenmakers, E.-J., Ratcliff, R., Gomez, P., & Iverson, G. J. (2004). Assessing model mimicry using the parametric bootstrap. Journal of Mathematical Psychology, 48, 28-50. Wixted, J. T., & Ebbesen, E. B. (1991). On the form of forgetting. Psychological Science, 2, 409-415.
URI: http://wrap.warwick.ac.uk/id/eprint/36013

Request changes to a record

Actions (login required)

View Item View Item