OpenAI was the first to applygenerative pre-training (GP) to the transformer architecture, introducing theGPT-1 model in 2018.[6] The company has since released many bigger GPT models. The popular chatbotChatGPT, released in late 2022 (usingGPT-3.5), was followed by many competitor chatbots using their own "GPT" models to generate text, such asGemini,DeepSeek orClaude.[7]
GPTs are primarily used to generate text, but can be trained to generate other kinds of data. For example,GPT-4o can process and generate text, images and audio.[8] To improve performance on complex tasks, some GPTs, such asOpenAI o3, spend more time analyzing the problem before generating an output, and are calledreasoning models. In 2025,GPT-5 was released with a router that automatically selects whether to use a faster model or slower reasoning model based on task.
According toThe Economist, improved algorithms, more powerful computers, and an increase in the amount of digitized material fueled a revolution inmachine learning during the 2010s. New techniques in the years before theAI boom resulted in "rapid improvements in tasks", including manipulating language.[9] Modern software models are trained to learn by using millions of examples inartificial neural networks that are inspired by biological neural structures.[9]
Separately, the concept of generative pre-training (GP) was a long-established technique in machine learning. GP is a form ofself-supervised learning wherein a model is first trained on a large, unlabeled dataset (the "pre-training" step) to learn to generate data points. This pre-trained model is then adapted to a specific task using a labeled dataset (the "fine-tuning" step).[10]
The transformer architecture for deep learning is the core technology of a GPT. Developed by researchers atGoogle, it was introduced in the paper "Attention Is All You Need", which was published on June 12, 2017. The transformer architecture solved many of the performance issues that were associated with olderrecurrent neural network (RNN) designs fornatural language processing (NLP). The architecture's use of anattention mechanism allows models to process entire sequences of text at once, enabling the training of much larger and more sophisticated models. Since 2017, numerous transformer-based NLP systems have been available that are capable of processing, mining, organizing, connecting, contrasting, andsummarizing texts as well as correctlyanswering questions from textual input.[11][12]
On June 11, 2018, OpenAI researchers and engineers published a paper called "Improving Language Understanding by Generative Pre-Training", which introducedGPT-1, the first GPT model.[13] It was designed as a transformer-basedlarge language model that used generative pre-training (GP) onBookCorpus, a diversetext corpus, followed by discriminativefine-tuning to focus on specific language tasks.[14] This semi-supervised approach was seen as a breakthrough. Previously, the best-performing neural models innatural language processing (NLP) had commonly employedsupervised learning from large amounts of manually labeled data – training a large language model with this approach would have been prohibitively expensive and time-consuming.[13]
On February 14, 2019, OpenAI introducedGPT-2, a larger model that could generate coherent text. Created as a direct scale-up of its predecessor, it had both its parameter count and dataset size increased by a factor of 10. GPT-2 has 1.5 billion parameters and was trained on WebText, a 40-gigabyte dataset of 8 millionweb pages.[15][16][17] Citing risks of malicious use, OpenAI opted for a "staged release", initially publishing smaller versions of the model before releasing the full 1.5-billion-parameter model in November.[18]
On February 10, 2020,Microsoft introduced its Turing Natural Language Generation, which it claimed was the "largest language model ever published at 17 billion parameters." The model outperformed all previous language models at a variety of tasks, includingsummarizing texts andanswering questions.[19]
On May 28, 2020, OpenAI introducedGPT-3, a model with 175 billion parameters that was trained on a larger dataset compared to GPT-2. It marked a significant advancement in few-shot and zero-shot learning abilities. With few examples, it could perform various tasks that it was not explicitly trained for.[20][21]
Following the release of GPT-3, OpenAI started usingreinforcement learning from human feedback (RLHF) to align models' behavior more closely with human preferences. This led to the development ofInstructGPT, a fine-tuned version of GPT-3. OpenAI further refined InstructGPT to createChatGPT, the flagship chatbot product of OpenAI that was launched on November 30, 2022.[22] ChatGPT was initially based onGPT-3.5, but it was later transitioned to theGPT-4 model, which was released on March 14, 2023.[23][24] GPT-4 was also integrated into parts of several applications, includingMicrosoft Copilot,GitHub Copilot,Snapchat,Khan Academy, andDuolingo.[25]
The immense popularity of ChatGPT spurred widespread development of competing GPT-based systems from other organizations.EleutherAI released a series ofopen-weight models, includingGPT-J in 2021. Other major technology companies later developed their own GPT models, such asGoogle'sPaLM andGemini as well asMeta AI'sLlama.[26]
Many subsequent GPT models have been trained to bemultimodal (able to process or to generate multiple types of data). For example,GPT-4o can both process and generate text, images, and audio.[27] Additionally, GPT models likeo3 andDeepSeek R1 have been trained withreinforcement learning to generate multi-stepchain-of-thought reasoning before producing a final answer, which helps to solve complex problems in domains such as mathematics.[28]
On August 7, 2025, OpenAI releasedGPT-5, which includes a router that automatically selects whether to use a faster model or slower reasoning model based on task.[29][30]
Afoundation model is an AI model trained on broad data at scale such that it can be adapted to a wide range of downstream tasks.[31][32]
Thus far, the most notable GPT foundation models have been fromOpenAI'sGPT-n series. The most recent from that isGPT-5.[33]
Other such models includeGoogle'sPaLM, a broad foundation model that has been compared toGPT-3 and has been made available to developers via anAPI,[34][35] and Together's GPT-JT, which has been reported as the closest-performingopen-source alternative toGPT-3 (and is derived fromearlier open-source GPTs).[36]Meta AI (formerlyFacebook) also has a generative transformer-based foundational large language model, known asLLaMA.[37]
Foundational GPTs can also employmodalities other than text, for input and/or output.GPT-4 is a multi-modal LLM that is capable of processing text and image input (though its output is limited to text).[38] Regarding multimodaloutput, some generative transformer-based models are used fortext-to-image technologies such asdiffusion[39] and parallel decoding.[40] Such kinds of models can serve as visual foundation models (VFMs) for developing downstream systems that can work with images.[41]
Training workflow of original ChatGPT/InstructGPT release[42][43]
A foundational GPT model can be further adapted to produce more targeted systems directed to specific tasks and/or subject-matter domains. Methods for such adaptation can include additionalfine-tuning (beyond that done for the foundation model) as well as certain forms ofprompt engineering.[44]
An important example of this isfine-tuning models to follow instructions, which is of course a fairly broad task but more targeted than a foundation model. In January 2022,OpenAI introduced "InstructGPT" – a series of models which were fine-tuned to follow instructions using a combination ofsupervised training andreinforcement learning from human feedback (RLHF) on base GPT-3 language models.[45][46] Advantages this had over the bare foundational models included higher accuracy, less negative/toxic sentiment, and generally better alignment with user needs. Hence, OpenAI began using this as the basis for itsAPI service offerings.[47] Other instruction-tuned models have been released by others, including a fully open version.[48][49]
Another (related) kind of task-specific models arechatbots, which engage in human-like conversation. In November 2022, OpenAI launchedChatGPT – an online chat interface powered by an instruction-tuned language model trained in a similar fashion to InstructGPT.[50] They trained this model using RLHF, with human AI trainers providing conversations in which they played both the user and the AI, and mixed this new dialogue dataset with the InstructGPT dataset for a conversational format suitable for a chatbot. Other major chatbots currently includeMicrosoft'sBing Chat, which uses OpenAI'sGPT-4 (as part of a broader close collaboration between OpenAI and Microsoft),[51] andGoogle's competing chatbotGemini (initially based on theirLaMDA family of conversation-trained language models, with plans to switch toPaLM).[52]
Yet another kind of task that a GPT can be used for is themeta-task of generatingits own instructions, like developing a series of prompts for 'itself' to be able to effectuate a more general goal given by a human user.[53] This is known as an AIagent, and more specifically a recursive one because it uses results from its previous self-instructions to help it form its subsequent prompts; the first major example of this wasAuto-GPT (which uses OpenAI's GPT models), and others have since been developed as well.[54]
GPT systems can be directed toward particular fields or domains. Some reported examples of such models and apps are as follows:
EinsteinGPT – for sales and marketing domains, to aid with customer relationship management (usesGPT-3.5)[55][56]
BloombergGPT – for the financial domain, to aid with financial news and information (uses "freely available" AI methods, combined with their proprietary data)[57]
Khanmigo – described as a GPT version for tutoring, in the education domain, it aids students usingKhan Academy by guiding them through their studies without directly providing answers (powered byGPT-4)[58][59]
SlackGPT – for theSlack instant-messaging service, to aid with navigating and summarizing discussions on it (usesOpenAI'sAPI)[60]
BioGPT – for the biomedical domain, to aid with biomedical literature text generation and mining (usesGPT-2)[61]
Sometimes domain-specificity is accomplished via softwareplug-ins or add-ons. For example, several different companies have developed particular plugins that interact directly with OpenAI'sChatGPT interface,[62][63] andGoogle Workspace has available add-ons such as "GPT for Sheets and Docs" – which is reported to aid use ofspreadsheet functionality inGoogle Sheets.[64][65]
OpenAI, which created the first generative pre-trained transformer (GPT) in 2018, asserted in 2023 that "GPT" should be regarded as abrand of OpenAI.[66] In April 2023, OpenAI revised the brand guidelines in itsterms of service to indicate that other businesses using itsAPI to run their AI services would no longer be able to include "GPT" in such names or branding.[67] In May 2023, OpenAI engaged a brand management service to notify its API customers of this policy, although these notifications stopped short of making overt legal claims (such as allegations oftrademark infringement or demands tocease and desist).[66] As of November 2023, OpenAI still prohibits its API licensees from naming their own products with "GPT",[68] but it has begun enabling its ChatGPT Plus subscribers to make "custom versions of ChatGPT" calledGPTs on the OpenAI site.[69] OpenAI's terms of service says that its subscribers may use "GPT" in the names of these, although it's "discouraged".[68]
Relatedly, OpenAI has applied to theUnited States Patent and Trademark Office (USPTO) to seek domestictrademark registration for the term "GPT" in the field of AI.[66] OpenAI sought to expedite handling of its application, but the USPTO declined that request in April 2023.[70] In May 2023, the USPTO responded to the application with a determination that "GPT" was both descriptive and generic.[71] As of November 2023, OpenAI continues to pursue its argument through the available processes. Regardless, failure to obtain aregistered U.S. trademark does not preclude some level ofcommon-law trademark rights in the U.S.[72] and trademark rights in other countries.[73]
For any given type or scope of trademark protection in the U.S., OpenAI would need to establish that the term is actually "distinctive" to their specific offerings in addition to being a broader technical term for the kind of technology. Some media reports suggested in 2023 that OpenAI may be able to obtain trademark registration based indirectly on the fame of its GPT-basedchatbot product,ChatGPT,[70][74] for which OpenAI hasseparately sought protection (and which it has sought to enforce more strongly).[75] Other reports have indicated that registration for the bare term "GPT" seems unlikely to be granted,[66][76] as it is used frequently as a common term to refer simply to AI systems that involve generative pre-trained transformers.[3][77][78][79] In any event, to whatever extent exclusive rights in the term may occur the U.S., others would need to avoid using it for similar products or services in ways likely to cause confusion.[76][80] If such rights ever became broad enough to implicate other well-established uses in the field, the trademark doctrine ofdescriptive fair use could still continue non-brand-related usage.[81]
^Erhan, Dumitru; Courville, Aaron; Bengio, Yoshua; Vincent, Pascal (March 31, 2010)."Why Does Unsupervised Pre-training Help Deep Learning?".Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings:201–208.Archived from the original on January 24, 2024. RetrievedJanuary 24, 2024.
^Vaswani, Ashish;Shazeer, Noam; Parmar, Niki; Uszkoreit, Jakob; Jones, Llion;Gomez, Aidan N; Kaiser, Łukasz; Polosukhin, Illia (June 12, 2017). "Attention Is All You Need". In I. Guyon and U. Von Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R. Garnett (ed.).31stConference on Neural Information Processing Systems. Advances in Neural Information Processing Systems. Vol. 30. Curran Associates, Inc.arXiv:1706.03762.
^Ouyang, Long; Wu, Jeff; Jiang, Xu; et al. (November 4, 2022). "Training language models to follow instructions with human feedback".NeurIPS.arXiv:2203.02155.
^Luo (et-al), Renqian (April 3, 2023). "BioGPT: Generative pre-trained transformer for biomedical text generation and mining".Briefings in Bioinformatics.23 (6) bbac409.arXiv:2210.10341.doi:10.1093/bib/bbac409.PMID36156661.
.svg){kind=logo}
Data from Wikidata
