This articlemay rely excessively on sourcestoo closely associated with the subject, potentially preventing the article from beingverifiable andneutral. Please helpimprove it by replacing them with more appropriatecitations toreliable, independent sources.(August 2023) (Learn how and when to remove this message)

Generative Pre-trained Transformer 1 (GPT-1)

Original author(s)	OpenAI
Initial release	June 2018; 6 years ago (June 2018)
Repository	github.com/openai/finetune-transformer-lm
Successor	GPT-2
Type	Large language model Generative pre-trained transformer
License	MIT[1]
Website	openai.com/blog/language-unsupervised/

Generative Pre-trained Transformer 1 (GPT-1) was the first ofOpenAI'slarge language models followingGoogle's invention of thetransformer architecture in 2017.^[2] In June 2018,OpenAI released a paper entitled "Improving Language Understanding by Generative Pre-Training",^[3] in which they introduced that initial model along with the general concept of agenerative pre-trained transformer.^[4]

Up to that point, the best-performing neural NLP models primarily employedsupervised learning from large amounts of manually labeled data. This reliance on supervised learning limited their use of datasets that were not well-annotated, in addition to making it prohibitively expensive and time-consuming to train extremely large models;^[3][5] many languages (such asSwahili orHaitian Creole) are difficult to translate and interpret using such models due to a lack of available text for corpus-building.^[5] In contrast, a GPT's "semi-supervised" approach involved two stages: an unsupervisedgenerative "pre-training" stage in which a language modeling objective was used to set initial parameters, and a superviseddiscriminative "fine-tuning" stage in which these parameters were adapted to a target task.^[3]

The use of atransformer architecture, as opposed to previous techniques involving attention-augmented RNNs, providedGPT models with a more structured memory than could be achieved through recurrent mechanisms; this resulted in "robust transfer performance across diverse tasks".^[3]

BookCorpus was chosen as a training dataset partly because the long passages of continuous text helped the model learn to handle long-range information.^[6] It contained over 7,000 unpublished fiction books from various genres. The rest of the datasets available at the time, while being larger, lacked this long-range structure (being "shuffled" at a sentence level).^[3]

The BookCorpus text was cleaned by theftfy library to standardized punctuation and whitespace and thentokenized byspaCy.^[3]

The GPT-1 architecture was a twelve-layer decoder-onlytransformer, using twelvemasked self-attention heads, with 64-dimensional states each (for a total of 768). Rather than simplestochastic gradient descent, theAdam optimization algorithm was used; the learning rate was increased linearly from zero over the first 2,000 updates to a maximum of 2.5×10⁻⁴, andannealed to 0 using a cosine schedule.^[3] GPT-1 has 117 million parameters.^[4]

While the fine-tuning was adapted to specific tasks, its pre-training was not; to perform the various tasks, minimal changes were performed to its underlying task-agnostic model architecture.^[3] Despite this, GPT-1 still improved on previous benchmarks in several language processing tasks, outperforming discriminatively-trained models with task-oriented architectures on several diverse tasks.^[3]

GPT-1 achieved a 5.8% and 1.5% improvement over previous best results^[3] on natural language inference (also known astextual entailment) tasks, evaluating the ability to interpret pairs of sentences from various datasets and classify the relationship between them as "entailment", "contradiction" or "neutral".^[3] Examples of such datasets include QNLI (Wikipedia articles) and MultiNLI (transcribed speech, popular fiction, and government reports, among other sources);^[7] It similarly outperformed previous models on two tasks related to question answering andcommonsense reasoning—by 5.7% on RACE,^[8] a dataset of written question-answer pairs from middle and high school exams, and by 8.9% on the StoryCloze Test.^[9]

GPT-1 improved on previous best-performing models by 4.2% onsemantic similarity (orparaphrase detection), evaluating the ability to predict whether two sentences are paraphrases of one another, using theQuora Question Pairs (QQP) dataset.^[3]

GPT-1 achieved a score of 45.4, versus a previous best of 35.0^[3] in a text classification task using the Corpus of Linguistic Acceptability (CoLA). Finally, GPT-1 achieved an overall score of 72.8 (compared to a previous record of 68.9) on GLUE, a multi-task test.^[10]

• Large language models
• Generative pre-trained transformers
• Software using the MIT license
• OpenAI

• Articles with short description
• Short description is different from Wikidata
• Articles lacking reliable references from August 2023
• All articles lacking reliable references