| Title: | The Fill-Mask Association Test |
| Version: | 2025.12 |
| Date: | 2025-12-15 |
| Maintainer: | Han Wu Shuang Bao <baohws@foxmail.com> |
| Description: | The Fill-Mask Association Test ('FMAT') <doi:10.1037/pspa0000396> is an integrative and probability-based method using Masked Language Models to measure conceptual associations (e.g., attitudes, biases, stereotypes, social norms, cultural values) as propositions in natural language. Supported language models include 'BERT' <doi:10.48550/arXiv.1810.04805> and its variants available at 'Hugging Face'https://huggingface.co/models?pipeline_tag=fill-mask. Methodological references and installation guidance are provided athttps://psychbruce.github.io/FMAT/. |
| License: | GPL-3 |
| Encoding: | UTF-8 |
| URL: | https://psychbruce.github.io/FMAT/ |
| BugReports: | https://github.com/psychbruce/FMAT/issues |
| SystemRequirements: | Python (>= 3.9.0) |
| Depends: | R (≥ 4.0.0) |
| Imports: | reticulate, data.table, stringr, forcats, rvest, psych, irr,glue, crayon, cli, purrr, plyr, dplyr, tidyr |
| Suggests: | bruceR, PsychWordVec, text, sweater, nlme |
| RoxygenNote: | 7.3.3 |
| NeedsCompilation: | no |
| Packaged: | 2025-12-15 06:11:55 UTC; baohw |
| Author: | Han Wu Shuang Bao [aut, cre] |
| Repository: | CRAN |
| Date/Publication: | 2025-12-15 06:30:02 UTC |
FMAT: The Fill-Mask Association Test
Description
The Fill-Mask Association Test ('FMAT')doi:10.1037/pspa0000396 is an integrative and probability-based method using Masked Language Models to measure conceptual associations (e.g., attitudes, biases, stereotypes, social norms, cultural values) as propositions in natural language. Supported language models include 'BERT'doi:10.48550/arXiv.1810.04805 and its variants available at 'Hugging Face'https://huggingface.co/models?pipeline_tag=fill-mask. Methodological references and installation guidance are provided athttps://psychbruce.github.io/FMAT/.
Author(s)
Maintainer: Han Wu Shuang Baobaohws@foxmail.com (ORCID)
See Also
Useful links:
A simple function equivalent tolist.
Description
A simple function equivalent tolist.
Usage
.(...)Arguments
... | Named objects (usually character vectors for this package). |
Value
A list of named objects.
Examples
.(Male=c("he", "his"), Female=c("she", "her"))Download and save BERT models to local cache folder.
Description
Download and save BERT models to local cache folder "%USERPROFILE%/.cache/huggingface".
Usage
BERT_download(models = NULL, verbose = FALSE)Arguments
models | A character vector of model names atHuggingFace. |
verbose | Alert if a model has been downloaded.Defaults to |
Value
Invisibly return a data.table of basic file information of local models.
See Also
Examples
## Not run: models = c("bert-base-uncased", "bert-base-cased")BERT_download(models)BERT_download() # check downloaded modelsBERT_info() # information of all downloaded models## End(Not run)Get basic information of BERT models.
Description
Get basic information of BERT models.
Usage
BERT_info(models = NULL)Arguments
models | A character vector of model names atHuggingFace. |
Value
A data.table:
model name
model type
number of parameters
vocabulary size (of input token embeddings)
embedding dimensions (of input token embeddings)
hidden layers
attention heads
[MASK] token
See Also
Examples
## Not run: models = c("bert-base-uncased", "bert-base-cased")BERT_info(models)BERT_info() # information of all downloaded models# speed: ~1.2s/model for first use; <1s afterwards## End(Not run)Scrape the initial commit date of BERT models.
Description
Scrape the initial commit date of BERT models.
Usage
BERT_info_date(models = NULL)Arguments
models | A character vector of model names atHuggingFace. |
Value
A data.table:
model name
initial commit date (scraped from huggingface commit history)
Examples
## Not run: model.date = BERT_info_date()# get all models from cache folderone.model.date = FMAT:::get_model_date("bert-base-uncased")# call the internal function to scrape a model# that may not have been saved in cache folder## End(Not run)Remove BERT models from local cache folder.
Description
Remove BERT models from local cache folder.
Usage
BERT_remove(models)Arguments
models | Model names. |
Value
NULL.
Check if mask words are in the model vocabulary.
Description
Check if mask words are in the model vocabulary.
Usage
BERT_vocab( models, mask.words, add.tokens = FALSE, add.method = c("mean", "sum"), add.verbose = TRUE)Arguments
models | A character vector of model names atHuggingFace. |
mask.words | Option words filling in the mask. |
add.tokens | Add new tokens (for out-of-vocabulary words or phrases) to model vocabulary? It only temporarily adds tokens for tasks but does not change the raw model file. Defaults to |
add.method | Method used to produce the token embeddings of appended tokens. Can be |
add.verbose | Print composition information of new tokens (for out-of-vocabulary words or phrases)? Defaults to |
Value
A data.table of model name, mask word, real token (replaced if out of vocabulary), and token id (0~N).
See Also
Examples
## Not run: models = c("bert-base-uncased", "bert-base-cased")BERT_info(models)BERT_vocab(models, c("bruce", "Bruce"))BERT_vocab(models, 2020:2025) # some are out-of-vocabularyBERT_vocab(models, 2020:2025, add.tokens=TRUE) # add vocabBERT_vocab(models, c("individualism", "artificial intelligence"), add.tokens=TRUE)## End(Not run)Prepare a data.table of queries and variables for the FMAT.
Description
Prepare a data.table of queries and variables for the FMAT.
Usage
FMAT_query( query = "Text with [MASK], optionally with {TARGET} and/or {ATTRIB}.", MASK = .(), TARGET = .(), ATTRIB = .())Arguments
query | Query text (should be a character string/vector with at least one |
MASK | A named list of
|
TARGET,ATTRIB | A named list of Target/Attribute words or phrases. If specified, then |
Value
A data.table of queries and variables.
See Also
Examples
FMAT_query("[MASK] is a nurse.", MASK = .(Male="He", Female="She"))FMAT_query( c("[MASK] is {TARGET}.", "[MASK] works as {TARGET}."), MASK = .(Male="He", Female="She"), TARGET = .(Occupation=c("a doctor", "a nurse", "an artist")))FMAT_query( "The [MASK] {ATTRIB}.", MASK = .(Male=c("man", "boy"), Female=c("woman", "girl")), ATTRIB = .(Masc=c("is masculine", "has a masculine personality"), Femi=c("is feminine", "has a feminine personality")))Combine multiple query data.tables and renumber query ids.
Description
Combine multiple query data.tables and renumber query ids.
Usage
FMAT_query_bind(...)Arguments
... | Query data.tables returned from |
Value
A data.table of queries and variables.
See Also
Examples
FMAT_query_bind( FMAT_query( "[MASK] is {TARGET}.", MASK = .(Male="He", Female="She"), TARGET = .(Occupation=c("a doctor", "a nurse", "an artist")) ), FMAT_query( "[MASK] occupation is {TARGET}.", MASK = .(Male="His", Female="Her"), TARGET = .(Occupation=c("doctor", "nurse", "artist")) ))Run the fill-mask pipeline on multiple models (CPU / GPU).
Description
Run the fill-mask pipeline on multiple models with CPU or GPU (faster but requires an NVIDIA GPU device).
Usage
FMAT_run( models, data, gpu, add.tokens = FALSE, add.method = c("mean", "sum"), add.verbose = TRUE, pattern.special = list(uncased = "uncased|albert|electra|muhtasham", prefix.u2581 = "albert|xlm-roberta|xlnet", prefix.u2581.excl = "chinese", prefix.u0120 = "roberta|bart|deberta|bertweet-large|ModernBERT", prefix.u0120.excl = "chinese|xlm-|kornosk/"), file = NULL, progress = TRUE, warning = TRUE, na.out = TRUE)Arguments
models | A character vector of model names atHuggingFace. |
data | A data.table returned from |
gpu | Use GPU (3x faster than CPU) to run the fill-mask pipeline? Defaults to missing value that willautomatically use available GPU (if not available, then use CPU). An NVIDIA GPU device (e.g., GeForce RTX Series) is required to use GPU. SeeGuidance for GPU Acceleration. Options passing on to the
|
add.tokens | Add new tokens (for out-of-vocabulary words or phrases) to model vocabulary? It only temporarily adds tokens for tasks but does not change the raw model file. Defaults to |
add.method | Method used to produce the token embeddings of appended tokens. Can be |
add.verbose | Print composition information of new tokens (for out-of-vocabulary words or phrases)? Defaults to |
pattern.special | Regular expression patterns (matching model names) for special model cases that are uncased or require a special prefix character in certain situations. WARNING: As the developer is not able to check all models, users are responsible for checking the models they would use and for modifying this argument if necessary.
|
file | File name of |
progress | Show a progress bar? Defaults to |
warning | Alert warning of out-of-vocabulary word(s)? Defaults to |
na.out | Replace probabilities of out-of-vocabulary word(s) with |
Details
The function automatically adjusts for the compatibility of tokens used in certain models: (1) for uncased models (e.g., ALBERT), it turns tokens to lowercase; (2) for models that use<mask> rather than[MASK], it automatically uses the corrected mask token; (3) for models that require a prefix to estimate whole words than subwords (e.g., ALBERT, RoBERTa), it adds a certain prefix (usually a white space; \u2581 for ALBERT and XLM-RoBERTa, \u0120 for RoBERTa and DistilRoBERTa).
These changes only affect thetoken variable in the returned data, but will not affect theM_word variable. Thus, users may analyze data based on the unchangedM_word rather than thetoken.
Note also that there may be extremely trivial differences (after 5~6 significant digits) in the raw probability estimates between using CPU and GPU, but these differences would have little impact on main results.
Value
A data.table (classfmat) appendingdata with these new variables:
model: model name.output: complete sentence output with unmasked token.token: actual token to be filled in the blank mask(a note "out-of-vocabulary" will be addedif the original word is not found in the model vocabulary).prob: (raw) conditional probability of the unmasked token given the provided context, estimated by the masked language model.It isNOT SUGGESTED to directly interpret the raw probabilities because thecontrast between a pair of probabilities is more interpretable. See
summary.fmat().
See Also
Examples
## Running the examples requires the models downloaded## Not run: models = c("bert-base-uncased", "bert-base-cased")query1 = FMAT_query( c("[MASK] is {TARGET}.", "[MASK] works as {TARGET}."), MASK = .(Male="He", Female="She"), TARGET = .(Occupation=c("a doctor", "a nurse", "an artist")))data1 = FMAT_run(models, query1)summary(data1, target.pair=FALSE)query2 = FMAT_query( "The [MASK] {ATTRIB}.", MASK = .(Male=c("man", "boy"), Female=c("woman", "girl")), ATTRIB = .(Masc=c("is masculine", "has a masculine personality"), Femi=c("is feminine", "has a feminine personality")))data2 = FMAT_run(models, query2)summary(data2, mask.pair=FALSE)summary(data2)## End(Not run)Intraclass correlation coefficient (ICC) of BERT models.
Description
Interrater agreement of log probabilities (treated as "ratings"/rows) among BERT language models (treated as "raters"/columns), with both row and column as ("two-way") random effects.
Usage
ICC_models(data, type = "agreement", unit = "average")Arguments
data | Raw data returned from |
type | Interrater |
unit | Reliability of |
Value
A data.table of ICC.
Reliability analysis (Cronbach's\alpha) of LPR.
Description
Reliability analysis (Cronbach's\alpha) of LPR.
Usage
LPR_reliability(fmat, item = c("query", "T_word", "A_word"), by = NULL)Arguments
fmat | A data.table returned from |
item | Reliability of multiple |
by | Variable(s) to split data by. Options can be |
Value
A data.table of Cronbach's\alpha.
Run the fill-mask pipeline and check the raw results.
Description
Normal users should useFMAT_run().This function is only for technical check.
Usage
fill_mask(query, model, targets = NULL, topn = 5, gpu)fill_mask_check(query, models, targets = NULL, topn = 5, gpu)Arguments
query | Query sentence with mask token. |
model,models | Model name(s). |
targets | Target words to fill in the mask.Defaults to |
topn | Number of the most likely predictions to return. Defaults to |
gpu | Use GPU (3x faster than CPU) to run the fill-mask pipeline? Defaults to missing value that willautomatically use available GPU (if not available, then use CPU). An NVIDIA GPU device (e.g., GeForce RTX Series) is required to use GPU. SeeGuidance for GPU Acceleration. Options passing on to the
|
Value
A data.table of raw results.
Functions
fill_mask(): Check performance of one model.fill_mask_check(): Check performance of multiple models.
Examples
## Not run: query = "Paris is the [MASK] of France."models = c("bert-base-uncased", "bert-base-cased")d.check = fill_mask_check(query, models, topn=2)## End(Not run)Set (change) HuggingFace cache folder temporarily.
Description
This function allows you to change the default cache directory (when it lacks storage space) to another path (e.g., your portable SSD)temporarily.
Usage
set_cache_folder(path = NULL)Arguments
path | Folder path to store HuggingFace models. If |
Keep in Mind
This function takes effect only for the current R sessiontemporarily, so you should run this each time BEFORE you use other FMAT functions in an R session.
Examples
## Not run: library(FMAT)set_cache_folder("D:/huggingface_cache/")# -> models would be saved to "D:/huggingface_cache/hub/"# run this function each time before using FMAT functionsBERT_download()BERT_info()## End(Not run)[S3 method] Summarize the results for the FMAT.
Description
Summarize the results ofLog Probability Ratio (LPR), which indicates therelative (vs.absolute) association between concepts.
Usage
## S3 method for class 'fmat'summary( object, mask.pair = TRUE, target.pair = TRUE, attrib.pair = TRUE, warning = TRUE, ...)Arguments
object | A data.table (class |
mask.pair,target.pair,attrib.pair | Pairwise contrast of |
warning | Alert warning of out-of-vocabulary word(s)? Defaults to |
... | Other arguments (currently not used). |
Details
The LPR of just one contrast (e.g., only between a pair of attributes) maynot be sufficient for a proper interpretation of the results, and may further require a second contrast (e.g., between a pair of targets).
Users are suggested to use linear mixed models (with the R packagesnlme orlme4/lmerTest) to perform the formal analyses and hypothesis tests based on the LPR.
Value
A data.table of the summarized results with Log Probability Ratio (LPR).
See Also
Examples
# see examples in `FMAT_run`