Abstract: Figural reasoning tests (as represented by Raven’s tests) are widely applied as effective measures of fluid intelligence in recruitment and personnel selection. However, several studies have revealed that those tests are not appropriate anymore due to high item exposure rates. Computerized automatic item generation (AIG) has gradually been recognized as a promising technique in handling item exposure. Understanding sources of item variation constitutes the initial stage of Computerized AIG, that is, searching for the underlying processing components and the stimuli that significantly influence those components. Some studies have explored sources of item variation, but so far there are no consistent results. This study investigated the relation between item difficulties and stimuli factors (e.g., familiarity of figures, abstraction of attributes, perceptual organization, and memory load) and determines the relative importance of those factors in predicting item difficulities.
Eight sets of figural reasoning tests (each set containing 14 items imitating items from Raven’s Advanced Progressive Matrics, APM) were constructed manipulating the familiarity of figures, the degree of abstraction of attributes, the perceptual organization as well as the types and number of rules. Using anchor-test design, these tests were administrated via the internet to 6323 participants with 10 items drawing from APAM as anchor items; thus, each participant completed 14 items from either one set and 10 anchor items within half an hour. In order to prevent participants from using response elimination strategy, we presented one item stem first, then alternatives in turn, and asked participants to determine which alternative was the best.
DIMTEST analyses were conducted on the participants’ responses on each of eight tests. Results showed that items measure a single dimension on each test. Likelihood ratio test indicated that the data fit two-parameter logistic model (2PL) best. Items were calibrated with BILOG-MG 3.0 (marginal maximum likelihood estimation and 2PL model) and displayed good item difficulties and discriminations. In order to make items from different sets comparable, item parameters were equated using the IRTEQ (Stocking and Lord’s Test Characteristic Curve approach) with the scale of set one as the reference. The 2×2×2 between group ANOVA showed two main effects for degree of abstraction of attributes and perceptual organization (p < 0.05), while the main effect for familiarity of figures as well as all interaction effects were not significant. Regression analysis indicated that memory load , abstraction of attributes, perceptual organization and familiarity of figures could significantly predict item difficulty and dominance analysis revealed that memory load (i.e. the combination of types and number of rules) was the most important predictor.
The present findings support several previous cognitive theories concerning figural reasoning problems in that working memory plays a key role in solving such kind of items. They also indicate that the degree of abstraction of attributes, the perceptual organization and the familiarity of figures also affect the processing components. Furthermore, a combination of those factors in item stems can predict item difficulties well. The findings are important for computerized AIG, because they enable us to generate new items with predicted item difficulties by manipulating those factors. However, as part of figural reasoning tests, distractors also have effects on item difficulties. The relationship between characteristics of distractors and item difficulties should be investigated to improve item generation algorithms for this kind of tests in the future.

Key words: automatic item generation, memory load, degree of abstraction of attributes, perceptual organization, types of rules