Bruno Nicenboim

pangoling: Access to Large Language Model Predictions

Provides access to word predictability estimates usinglarge language models (LLMs) based on 'transformer'architectures via integration with the 'Hugging Face'ecosystem. The package interfaces with pre-trained neuralnetworks and supports both causal/auto-regressive LLMs (e.g.,'GPT-2'; Radford et al., 2019) and masked/bidirectional LLMs(e.g., 'BERT'; Devlin et al., 2019,<doi:10.48550/arXiv.1810.04805>) to compute the probability ofwords, phrases, or tokens given their linguistic context. Byenabling a straightforward estimation of word predictability,the package facilitates research in psycholinguistics,computational linguistics, and natural language processing(NLP).

Authors:Bruno Nicenboim [aut, cre],Chris Emmerly [ctb],Giovanni Cassani [ctb],Lisa Levinson [rev],Utku Turk [rev]

• df_jaeger14 - Self-Paced Reading Dataset on Chinese Relative Clauses
• df_sent - Example dataset: Two word-by-word sentences