| GPT-J | |
|---|---|
 Logo | |
| Developer | EleutherAI |
| Initial release | June 9, 2021; 4 years ago (2021-06-09) |
| Type | |
| License | Apache License 2.0 |
| Website | 6b |
GPT-J orGPT-J-6B is an open-sourcelarge language model (LLM) developed byEleutherAI in 2021.[1] As the name suggests, it is agenerative pre-trained transformer model designed to produce human-like text that continues from a prompt. The optional "6B" in the name refers to the fact that it has 6 billion parameters.[2] The model is available onGitHub, but the web interface no longer communicates with the model. Development stopped in 2021.[3]
GPT-J is aGPT-3-like model with 6 billion parameters.[4] Like GPT-3, it is anautoregressive, decoder-onlytransformer model designed to solvenatural language processing (NLP) tasks by predicting how a piece of text will continue.[1]
Its architecture differs from GPT-3 in three main ways.[1]
Beyond that, the model has 28 transformer layers and 16 attention heads. Its vocabulary size is 50257tokens, the same size asGPT-2's.[2] It has acontext window size of 2048 tokens.[7]
It was trained onthe Pile dataset,[2][4] using the Mesh Transformer JAX library inJAX to handle the parallelization scheme.[2][8]
GPT-J was designed to generate English text from a prompt. It was not designed for translating or generating text in other languages or for performance without firstfine-tuning the model for a specific task.[2] Nonetheless, GPT-J performs reasonably well even without fine-tuning, even in translation (at least from English to French).[9]
When neither is fine-tuned, GPT-J-6B performs almost as well as the 6.7 billion parameter GPT-3 (Curie) on a variety of tasks.[4] It even outperforms the 175 billion parameter GPT-3 (Davinci) on code generation tasks.[10] With fine-tuning, it outperforms an untuned GPT-3 (Davinci) on a number of tasks.[1]
Like all LLMs, it is not programmed to give factually accurate information, only to generate text based on probability.[2]
The untuned GPT-J is available on EleutherAI's website,[11]NVIDIA's Triton Inference Server,[12] and NLP Cloud's website.[13]Cerebras[1] andAmazon Web Services[14][15] offer services to fine-tune the GPT-J model for company-specific tasks.Graphcore offers both fine-tuning and hosting services for the untuned GPT-J, as well as offering to host the fine-tuned models after they are produced.[16] CoreWeave offers hosting services for both the untuned GPT-J and fine-tuned variants.[17][18]
In March 2023,Databricks released Dolly, anApache-licensed, instruction-following model created by fine-tuning GPT-J on theStanford Alpaca dataset.[19]NovelAI's Sigurd[20] and Genji-JP 6B[21] models are both fine-tuned versions of GPT-J. They also offer further fine-tuning services to produce and host custom models.[22]
EleutherAI has received praise from Cerebras,[1] GPT-3 Demo,[4] NLP Cloud,[13] and Databricks[19] for making the model open-source, and its open-source status is often cited as a major advantage when choosing which model to use.[10][16][23]
In general we have found that across a large suite of setups including regular, linear, and local self-attention, it either matches or surpasses all other methods currently available for injecting positional information into transformers.
| Concepts |
| ||||||
|---|---|---|---|---|---|---|---|
| Applications | |||||||
| Implementations |
| ||||||
| People |
| ||||||
| Architectures |
| ||||||
