Movatterモバイル変換

autoFC

A collection of tools to automatically pair forced-choice items andexamine their measurement performance

Overview

Forced-choice (FC) tests are gaining researcher’sinterest increasingly for its faking resistance when well-designed.Well-designed FC tests should often be characterized byitems withina block measuring different latent traits, anditems within ablock having similar magnitude, or high inter-item agreement(IIA) in terms of their social desirability. Otherscoring models may also requirefactor loading differences or itemlocations within a block to be maximized or minimized.

Either way, decision on which items should be assigned to the sameblock - item pairing - is a crucial issue in building a well-designed FCtest, which is currently carried out manually. However, given that weoften need to simultaneously meet multiple objectives, manual pairingwill turn out to be impractical and even infeasible, especially when thenumber of latent traits and/or the number of items per trait becomerelatively large.

The R packageautoFC is developed to address thesedifficulties and provides a tool for facilitating automatic FC testconstruction as well as evaluating measurement performance usingsimulation data. It offers users the functionality to:

1. Include multiple criteria for pairing items into the same block,with user-specified weights and calculating functions for eachcriterion.

2. Automatically optimize the target function combined from themultiple criteria and produce near-optimal item pairings that satisfythe user-defined criteria.

3. Specify blueprints for the FC blocks (i.e., exact specificationon how the block should be, for example, in terms of measured traits andkeying) and build FC blocks that are aligned with the setups in theblueprints.

4. Produce simulated responses to FC scales, based on theThurstonian IRT model (Brown & Maydeu-Olivares, 2011), and estimatethe Thurstonian IRT model using the simulated responses.

5. Examine the empirical reliability and measurement precision ofthe resulting trait scores produced from the estimation model.

Users are allowed to create an FC test of any block size (e.g. Pairs,Triplets, Quadruplets) and they can produce simulated responses to FCscales in both MOLE (Most & Least like me) and RANK formats.

Installation

You can install autoFC from CRAN:

install.packages("autoFC")

You can install the development version of autoFC from GitHub:

devtools::install_github("tspsyched/autoFC")

Functions

Below is a brief explanation of all functions provided by the initialversion ofautoFC.

1. cal_block_energy() andcal_block_energy_with_iia() both calculate the total energyfor a single item block, or a full FC test with multiple blocks, given adata frame of item characteristics. The latter function incorporates IIAmetrics into energy calculation.

• By default, numeric item characteristics are paired by minimizingvariance within each block, and factor item characteristics are pairedsuch that it’s more preferable for items in the same block to be fromdifferent factor levels. Each characteristic has a default weight of1.
• In addition,cal_block_energy_with_iia() incorporatesfour IIA metrics in which items are paired by maximizing the IIA withineach block. Each IIA has a default weight of 1.

2. make_random_block() takes in number of items andblock size as input arguments and produces a test with blocks ofrandomly paired item numbers. Information about item characteristics isnot required.

3. get_iia() takes in item responses and a single itemblock (Or a full FC test with multiple blocks), then returns IIA metricsfor each item block.

4. sa_pairing_generalized() is the automatic pairingfunction which takes in item characteristics (and also individualresponses for all items) and an initial FC test, then optimizes theenergy of the test based on Simulated Annealing (SA) algorithm.

• SA is a probabilistic technique for approximating the globaloptimum of a given function, in which each iteration involves the cooldown of the “Temperature” until it reaches a certain value. Within eachiteration, a new solution (FC test) is produced andcompared with current solution in terms of their energy (Which iscalculated by callingcal_block_energy() orcal_block_energy_with_iia(). Acceptance or rejection of newsolution is determined as follows:

  • Better solution (An FC test with higher energy) is always acceptedand updated into the new current solution.
  • Worse solution (An FC test with lower energy) is conditionallyaccepted, determined by the current temperature and the deviation of newsolution’s energy from the current one’s. A worse solution is morelikely to be accepted when temperature is high and when the deviance isrelatively small.

• If all items in the item characteristic data frame are used toconstruct the FC test,sa_pairing_generalized() willproduce new solutions by randomly exchanging items between two blocks;Otherwise, it will randomly select between exchanging items andreplacing with unused items based on proportion of items used toconstruct the test.

In the Feb, 2024 update, we added a lot more functions, including thefollowing core ones:

1. construct_blueprint() builds up exact specificationsof the FC blocks (i.e., blueprints), which typically indicates thekeying and measured traits of each item for each block. An additionalmatching criteria can also be set, indicating how well should the itemsbe matched based on certain indicators using a pre-specifiedcutoff.

2. build_scale_with_blueprint() takes in the blueprintthat user built manually or throughconstruct_blueprint()and automatically produces the paired FC blocks consistent with thespecifications in the blueprint.

• This is an addition to the automatic item pairing module as wasalready there back in the earlier version of autoFC. Typically whenusers want to construct FC scales, they would wish at least some blocksto be exactly in certain designs. This function exactly serves as thatpurpose.

3. get_simulation_matrices() produces simulated itemand person parameters based on the Thurstonian IRT model, using thefactor analysis results extracted fromlavaan::cfa() orget_CFA_estimates().

4. convert_to_TIRT_response(),get_TIRT_long_data(),fit_TIRT_model() areextensions to the various functions in theThurstonianIRTpackage (Bürkner, 2019) which allow the simulated (usingconvert_to_TIRT_response()) or actual responses to FCscales to be processed and converted into long format (usingget_TIRT_long_data()) and fitted using lavaan, Mplus orstan methods (usingfit_TIRT_model()).

5. RMSE_range(),plot_scores(), andempirical_reliability() for diagnostic purposes, examiningthe measurement accuracy of the trait scores produced from the TIRTmodel.

Detailed descriptions of all functions and other functions that arenot listed here can be found in the manual and the help document of eachfunction.

We also recently published a paper (Li et al., 2024) discussing theissues related to the development of forced-choice scales, which alsoincludes a detailed tutorial on how to construct FC scales using theselatest functionalities of theautoFC package. Users are alsoencouraged to refer to this paper for further details.

References

Brown, A., & Maydeu-Olivares, A. (2011). Item response modelingof forced-choice questionnaires.Educational and PsychologicalMeasurement, 71(3), 460-502.https://doi.org/10.1177/0013164410375112 Bürkner, P. C. (2019).thurstonianIRT: Thurstonian IRT models in R.Journal of Open SourceSoftware, 4(42), 1662. https://doi.org/10.21105/joss.01662 Li, M.,Zhang, B., Li, L., Sun, T., & Brown, A., (2024). Mixed-Keying orDesirability-Matching in the Construction of Forced-Choice Measures? AnEmpirical Investigation and Practical Recommendations.Organizational Research Methods.https://doi.org/10.1177/10944281241229784