This model was released on 2019-10-02 and added to Hugging Face Transformers on 2020-11-16.
DistilBERT
DistilBERT is pretrained by knowledge distillation to create a smaller model with faster inference and requires less compute to train. Through a triple loss objective during pretraining, language modeling loss, distillation loss, cosine-distance loss, DistilBERT demonstrates similar performance to a larger transformer language model.
You can find all the original DistilBERT checkpoints under theDistilBERT organization.
Click on the DistilBERT models in the right sidebar for more examples of how to apply DistilBERT to different language tasks.
The example below demonstrates how to classify text withPipeline,AutoModel, and from the command line.
from transformersimport pipelineclassifier = pipeline( task="text-classification", model="distilbert-base-uncased-finetuned-sst-2-english", dtype=torch.float16, device=0)result = classifier("I love using Hugging Face Transformers!")print(result)# Output: [{'label': 'POSITIVE', 'score': 0.9998}]
Notes
- DistilBERT doesn’t have
token_type_ids, you don’t need to indicate which token belongs to which segment. Justseparate your segments with the separation tokentokenizer.sep_token(or[SEP]). - DistilBERT doesn’t have options to select the input positions (
position_idsinput). This could be added ifnecessary though, just let us know if you need this option.
DistilBertConfig
classtransformers.DistilBertConfig
<source>(vocab_size = 30522max_position_embeddings = 512sinusoidal_pos_embds = Falsen_layers = 6n_heads = 12dim = 768hidden_dim = 3072dropout = 0.1attention_dropout = 0.1activation = 'gelu'initializer_range = 0.02qa_dropout = 0.1seq_classif_dropout = 0.2pad_token_id = 0**kwargs)
Parameters
- vocab_size (
int,optional, defaults to 30522) —Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be represented bytheinputs_idspassed when callingDistilBertModel orTFDistilBertModel. - max_position_embeddings (
int,optional, defaults to 512) —The maximum sequence length that this model might ever be used with. Typically set this to something largejust in case (e.g., 512 or 1024 or 2048). - sinusoidal_pos_embds (
boolean,optional, defaults toFalse) —Whether to use sinusoidal positional embeddings. - n_layers (
int,optional, defaults to 6) —Number of hidden layers in the Transformer encoder. - n_heads (
int,optional, defaults to 12) —Number of attention heads for each attention layer in the Transformer encoder. - dim (
int,optional, defaults to 768) —Dimensionality of the encoder layers and the pooler layer. - hidden_dim (
int,optional, defaults to 3072) —The size of the “intermediate” (often named feed-forward) layer in the Transformer encoder. - dropout (
float,optional, defaults to 0.1) —The dropout probability for all fully connected layers in the embeddings, encoder, and pooler. - attention_dropout (
float,optional, defaults to 0.1) —The dropout ratio for the attention probabilities. - activation (
strorCallable,optional, defaults to"gelu") —The non-linear activation function (function or string) in the encoder and pooler. If string,"gelu","relu","silu"and"gelu_new"are supported. - initializer_range (
float,optional, defaults to 0.02) —The standard deviation of the truncated_normal_initializer for initializing all weight matrices. - qa_dropout (
float,optional, defaults to 0.1) —The dropout probabilities used in the question answering modelDistilBertForQuestionAnswering. - seq_classif_dropout (
float,optional, defaults to 0.2) —The dropout probabilities used in the sequence classification and the multiple choice modelDistilBertForSequenceClassification.
This is the configuration class to store the configuration of aDistilBertModel or aTFDistilBertModel. Itis used to instantiate a DistilBERT model according to the specified arguments, defining the model architecture.Instantiating a configuration with the defaults will yield a similar configuration to that of the DistilBERTdistilbert-base-uncased architecture.
Configuration objects inherit fromPreTrainedConfig and can be used to control the model outputs. Read thedocumentation fromPreTrainedConfig for more information.
Examples:
>>>from transformersimport DistilBertConfig, DistilBertModel>>># Initializing a DistilBERT configuration>>>configuration = DistilBertConfig()>>># Initializing a model (with random weights) from the configuration>>>model = DistilBertModel(configuration)>>># Accessing the model configuration>>>configuration = model.config
DistilBertTokenizer
classtransformers.DistilBertTokenizer
<source>(vocab_filedo_lower_case = Truedo_basic_tokenize = Truenever_split = Noneunk_token = '[UNK]'sep_token = '[SEP]'pad_token = '[PAD]'cls_token = '[CLS]'mask_token = '[MASK]'tokenize_chinese_chars = Truestrip_accents = Noneclean_up_tokenization_spaces = True**kwargs)
Parameters
- vocab_file (
str) —File containing the vocabulary. - do_lower_case (
bool,optional, defaults toTrue) —Whether or not to lowercase the input when tokenizing. - do_basic_tokenize (
bool,optional, defaults toTrue) —Whether or not to do basic tokenization before WordPiece. - never_split (
Iterable,optional) —Collection of tokens which will never be split during tokenization. Only has an effect whendo_basic_tokenize=True - unk_token (
str,optional, defaults to"[UNK]") —The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be thistoken instead. - sep_token (
str,optional, defaults to"[SEP]") —The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences forsequence classification or for a text and a question for question answering. It is also used as the lasttoken of a sequence built with special tokens. - pad_token (
str,optional, defaults to"[PAD]") —The token used for padding, for example when batching sequences of different lengths. - cls_token (
str,optional, defaults to"[CLS]") —The classifier token which is used when doing sequence classification (classification of the whole sequenceinstead of per-token classification). It is the first token of the sequence when built with special tokens. - mask_token (
str,optional, defaults to"[MASK]") —The token used for masking values. This is the token used when training this model with masked languagemodeling. This is the token which the model will try to predict. - tokenize_chinese_chars (
bool,optional, defaults toTrue) —Whether or not to tokenize Chinese characters.This should likely be deactivated for Japanese (see thisissue).
- strip_accents (
bool,optional) —Whether or not to strip all accents. If this option is not specified, then it will be determined by thevalue forlowercase(as in the original BERT). - clean_up_tokenization_spaces (
bool,optional, defaults toTrue) —Whether or not to cleanup spaces after decoding, cleanup consists in removing potential artifacts likeextra spaces.
Construct a DistilBERT tokenizer. Based on WordPiece.
This tokenizer inherits fromPreTrainedTokenizer which contains most of the main methods. Users should refer tothis superclass for more information regarding those methods.
build_inputs_with_special_tokens
<source>(token_ids_0: listtoken_ids_1: typing.Optional[list[int]] = None)→List[int]
Parameters
- token_ids_0 (
List[int]) —List of IDs to which the special tokens will be added. - token_ids_1 (
List[int],optional) —Optional second list of IDs for sequence pairs.
Returns
List[int]
List ofinput IDs with the appropriate special tokens.
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating andadding special tokens. A BERT sequence has the following format:
- single sequence:
[CLS] X [SEP] - pair of sequences:
[CLS] A [SEP] B [SEP]
convert_tokens_to_string
<source>(tokens)
Converts a sequence of tokens (string) in a single string.
get_special_tokens_mask
<source>(token_ids_0: listtoken_ids_1: typing.Optional[list[int]] = Nonealready_has_special_tokens: bool = False)→List[int]
Parameters
- token_ids_0 (
List[int]) —List of IDs. - token_ids_1 (
List[int],optional) —Optional second list of IDs for sequence pairs. - already_has_special_tokens (
bool,optional, defaults toFalse) —Whether or not the token list is already formatted with special tokens for the model.
Returns
List[int]
A list of integers in the range [0, 1]: 1 for a special token, 0 for a sequence token.
Retrieve sequence ids from a token list that has no special tokens added. This method is called when addingspecial tokens using the tokenizerprepare_for_model method.
DistilBertTokenizerFast
classtransformers.DistilBertTokenizerFast
<source>(vocab_file = Nonetokenizer_file = Nonedo_lower_case = Trueunk_token = '[UNK]'sep_token = '[SEP]'pad_token = '[PAD]'cls_token = '[CLS]'mask_token = '[MASK]'tokenize_chinese_chars = Truestrip_accents = None**kwargs)
Parameters
- vocab_file (
str) —File containing the vocabulary. - do_lower_case (
bool,optional, defaults toTrue) —Whether or not to lowercase the input when tokenizing. - unk_token (
str,optional, defaults to"[UNK]") —The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be thistoken instead. - sep_token (
str,optional, defaults to"[SEP]") —The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences forsequence classification or for a text and a question for question answering. It is also used as the lasttoken of a sequence built with special tokens. - pad_token (
str,optional, defaults to"[PAD]") —The token used for padding, for example when batching sequences of different lengths. - cls_token (
str,optional, defaults to"[CLS]") —The classifier token which is used when doing sequence classification (classification of the whole sequenceinstead of per-token classification). It is the first token of the sequence when built with special tokens. - mask_token (
str,optional, defaults to"[MASK]") —The token used for masking values. This is the token used when training this model with masked languagemodeling. This is the token which the model will try to predict. - clean_text (
bool,optional, defaults toTrue) —Whether or not to clean the text before tokenization by removing any control characters and replacing allwhitespaces by the classic one. - tokenize_chinese_chars (
bool,optional, defaults toTrue) —Whether or not to tokenize Chinese characters. This should likely be deactivated for Japanese (seethisissue). - strip_accents (
bool,optional) —Whether or not to strip all accents. If this option is not specified, then it will be determined by thevalue forlowercase(as in the original BERT). - wordpieces_prefix (
str,optional, defaults to"##") —The prefix for subwords.
Construct a “fast” DistilBERT tokenizer (backed by HuggingFace’stokenizers library). Based on WordPiece.
This tokenizer inherits fromPreTrainedTokenizerFast which contains most of the main methods. Users shouldrefer to this superclass for more information regarding those methods.
build_inputs_with_special_tokens
<source>(token_ids_0token_ids_1 = None)→List[int]
Parameters
- token_ids_0 (
List[int]) —List of IDs to which the special tokens will be added. - token_ids_1 (
List[int],optional) —Optional second list of IDs for sequence pairs.
Returns
List[int]
List ofinput IDs with the appropriate special tokens.
Build model inputs from a sequence or a pair of sequence for sequence classification tasks by concatenating andadding special tokens. A BERT sequence has the following format:
- single sequence:
[CLS] X [SEP] - pair of sequences:
[CLS] A [SEP] B [SEP]
DistilBertModel
classtransformers.DistilBertModel
<source>(config: PreTrainedConfig)
Parameters
- config (PreTrainedConfig) —Model configuration class with all the parameters of the model. Initializing with a config file does notload the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.
The bare Distilbert Model outputting raw hidden-states without any specific head on top.
This model inherits fromPreTrainedModel. Check the superclass documentation for the generic methods thelibrary implements for all its model (such as downloading or saving, resizing the input embeddings, pruning headsetc.)
This model is also a PyTorchtorch.nn.Module subclass.Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usageand behavior.
forward
<source>(input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.Tensor] = None**kwargs: typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs])→transformers.modeling_outputs.BaseModelOutput ortuple(torch.FloatTensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, num_choices)) —Indices of input sequence tokens in the vocabulary.Indices can be obtained usingAutoTokenizer. SeePreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length),optional) —Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that arenot masked,
- 0 for tokens that aremasked.
- inputs_embeds (
torch.FloatTensorof shape(batch_size, num_choices, hidden_size),optional) —Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. Thisis useful if you want more control over how to convertinput_idsindices into associated vectors than themodel’s internal embedding lookup matrix. - position_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.n_positions - 1].
Returns
transformers.modeling_outputs.BaseModelOutput ortuple(torch.FloatTensor)
Atransformers.modeling_outputs.BaseModelOutput or a tuple oftorch.FloatTensor (ifreturn_dict=False is passed or whenconfig.return_dict=False) comprising variouselements depending on the configuration (DistilBertConfig) and inputs.
last_hidden_state (
torch.FloatTensorof shape(batch_size, sequence_length, hidden_size)) — Sequence of hidden-states at the output of the last layer of the model.hidden_states (
tuple(torch.FloatTensor),optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, +one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.FloatTensor),optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attentionheads.
TheDistilBertModel forward method, overrides the__call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps whilethe latter silently ignores them.
DistilBertForMaskedLM
classtransformers.DistilBertForMaskedLM
<source>(config: PreTrainedConfig)
Parameters
- config (PreTrainedConfig) —Model configuration class with all the parameters of the model. Initializing with a config file does notload the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.
DistilBert Model with amasked language modeling head on top.
This model inherits fromPreTrainedModel. Check the superclass documentation for the generic methods thelibrary implements for all its model (such as downloading or saving, resizing the input embeddings, pruning headsetc.)
This model is also a PyTorchtorch.nn.Module subclass.Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usageand behavior.
forward
<source>(input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Nonelabels: typing.Optional[torch.LongTensor] = Noneposition_ids: typing.Optional[torch.Tensor] = None**kwargs: typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs])→transformers.modeling_outputs.MaskedLMOutput ortuple(torch.FloatTensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, num_choices)) —Indices of input sequence tokens in the vocabulary.Indices can be obtained usingAutoTokenizer. SeePreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length),optional) —Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that arenot masked,
- 0 for tokens that aremasked.
- inputs_embeds (
torch.FloatTensorof shape(batch_size, num_choices, hidden_size),optional) —Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. Thisis useful if you want more control over how to convertinput_idsindices into associated vectors than themodel’s internal embedding lookup matrix. - labels (
torch.LongTensorof shape(batch_size, sequence_length),optional) —Labels for computing the masked language modeling loss. Indices should be in[-100, 0, ..., config.vocab_size](seeinput_idsdocstring) Tokens with indices set to-100are ignored (masked), theloss is only computed for the tokens with labels in[0, ..., config.vocab_size]. - position_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.n_positions - 1].
Returns
transformers.modeling_outputs.MaskedLMOutput ortuple(torch.FloatTensor)
Atransformers.modeling_outputs.MaskedLMOutput or a tuple oftorch.FloatTensor (ifreturn_dict=False is passed or whenconfig.return_dict=False) comprising variouselements depending on the configuration (DistilBertConfig) and inputs.
loss (
torch.FloatTensorof shape(1,),optional, returned whenlabelsis provided) — Masked language modeling (MLM) loss.logits (
torch.FloatTensorof shape(batch_size, sequence_length, config.vocab_size)) — Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).hidden_states (
tuple(torch.FloatTensor),optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, +one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.FloatTensor),optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attentionheads.
TheDistilBertForMaskedLM forward method, overrides the__call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps whilethe latter silently ignores them.
Example:
>>>from transformersimport AutoTokenizer, DistilBertForMaskedLM>>>import torch>>>tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")>>>model = DistilBertForMaskedLM.from_pretrained("distilbert-base-uncased")>>>inputs = tokenizer("The capital of France is <mask>.", return_tensors="pt")>>>with torch.no_grad():... logits = model(**inputs).logits>>># retrieve index of <mask>>>>mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]>>>predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)>>>tokenizer.decode(predicted_token_id)...>>>labels = tokenizer("The capital of France is Paris.", return_tensors="pt")["input_ids"]>>># mask labels of non-<mask> tokens>>>labels = torch.where(inputs.input_ids == tokenizer.mask_token_id, labels, -100)>>>outputs = model(**inputs, labels=labels)>>>round(outputs.loss.item(),2)...
DistilBertForSequenceClassification
classtransformers.DistilBertForSequenceClassification
<source>(config: PreTrainedConfig)
Parameters
- config (PreTrainedConfig) —Model configuration class with all the parameters of the model. Initializing with a config file does notload the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.
DistilBert Model transformer with a sequence classification/regression head on top (a linear layer on top of thepooled output) e.g. for GLUE tasks.
This model inherits fromPreTrainedModel. Check the superclass documentation for the generic methods thelibrary implements for all its model (such as downloading or saving, resizing the input embeddings, pruning headsetc.)
This model is also a PyTorchtorch.nn.Module subclass.Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usageand behavior.
forward
<source>(input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Nonelabels: typing.Optional[torch.LongTensor] = Noneposition_ids: typing.Optional[torch.Tensor] = None**kwargs: typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs])→transformers.modeling_outputs.SequenceClassifierOutput ortuple(torch.FloatTensor)
Parameters
- input_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.Indices can be obtained usingAutoTokenizer. SeePreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length),optional) —Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that arenot masked,
- 0 for tokens that aremasked.
- inputs_embeds (
torch.Tensorof shape(batch_size, sequence_length, hidden_size),optional) —Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. Thisis useful if you want more control over how to convertinput_idsindices into associated vectors than themodel’s internal embedding lookup matrix. - labels (
torch.LongTensorof shape(batch_size,),optional) —Labels for computing the sequence classification/regression loss. Indices should be in[0, ..., config.num_labels - 1]. Ifconfig.num_labels == 1a regression loss is computed (Mean-Square loss), Ifconfig.num_labels > 1a classification loss is computed (Cross-Entropy). - position_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.n_positions - 1].
Returns
transformers.modeling_outputs.SequenceClassifierOutput ortuple(torch.FloatTensor)
Atransformers.modeling_outputs.SequenceClassifierOutput or a tuple oftorch.FloatTensor (ifreturn_dict=False is passed or whenconfig.return_dict=False) comprising variouselements depending on the configuration (DistilBertConfig) and inputs.
loss (
torch.FloatTensorof shape(1,),optional, returned whenlabelsis provided) — Classification (or regression if config.num_labels==1) loss.logits (
torch.FloatTensorof shape(batch_size, config.num_labels)) — Classification (or regression if config.num_labels==1) scores (before SoftMax).hidden_states (
tuple(torch.FloatTensor),optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, +one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.FloatTensor),optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attentionheads.
TheDistilBertForSequenceClassification forward method, overrides the__call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps whilethe latter silently ignores them.
Example of single-label classification:
>>>import torch>>>from transformersimport AutoTokenizer, DistilBertForSequenceClassification>>>tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")>>>model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased")>>>inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")>>>with torch.no_grad():... logits = model(**inputs).logits>>>predicted_class_id = logits.argmax().item()>>>model.config.id2label[predicted_class_id]...>>># To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`>>>num_labels =len(model.config.id2label)>>>model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", num_labels=num_labels)>>>labels = torch.tensor([1])>>>loss = model(**inputs, labels=labels).loss>>>round(loss.item(),2)...
Example of multi-label classification:
>>>import torch>>>from transformersimport AutoTokenizer, DistilBertForSequenceClassification>>>tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")>>>model = DistilBertForSequenceClassification.from_pretrained("distilbert-base-uncased", problem_type="multi_label_classification")>>>inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")>>>with torch.no_grad():... logits = model(**inputs).logits>>>predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) >0.5]>>># To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`>>>num_labels =len(model.config.id2label)>>>model = DistilBertForSequenceClassification.from_pretrained(..."distilbert-base-uncased", num_labels=num_labels, problem_type="multi_label_classification"...)>>>labels = torch.sum(... torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1...).to(torch.float)>>>loss = model(**inputs, labels=labels).loss
DistilBertForMultipleChoice
classtransformers.DistilBertForMultipleChoice
<source>(config: PreTrainedConfig)
Parameters
- config (PreTrainedConfig) —Model configuration class with all the parameters of the model. Initializing with a config file does notload the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.
The Distilbert Model with a multiple choice classification head on top (a linear layer on top of the pooled output and asoftmax) e.g. for RocStories/SWAG tasks.
This model inherits fromPreTrainedModel. Check the superclass documentation for the generic methods thelibrary implements for all its model (such as downloading or saving, resizing the input embeddings, pruning headsetc.)
This model is also a PyTorchtorch.nn.Module subclass.Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usageand behavior.
forward
<source>(input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Nonelabels: typing.Optional[torch.LongTensor] = Noneposition_ids: typing.Optional[torch.Tensor] = None**kwargs: typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs])→transformers.modeling_outputs.MultipleChoiceModelOutput ortuple(torch.FloatTensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, num_choices, sequence_length)) —Indices of input sequence tokens in the vocabulary.Indices can be obtained usingAutoTokenizer. SeePreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length),optional) —Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that arenot masked,
- 0 for tokens that aremasked.
- inputs_embeds (
torch.FloatTensorof shape(batch_size, num_choices, sequence_length, hidden_size),optional) —Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. Thisis useful if you want more control over how to convertinput_idsindices into associated vectors than themodel’s internal embedding lookup matrix. - labels (
torch.LongTensorof shape(batch_size,),optional) —Labels for computing the multiple choice classification loss. Indices should be in[0, ..., num_choices-1]wherenum_choicesis the size of the second dimension of the input tensors. (Seeinput_idsabove) - position_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.n_positions - 1].
Returns
transformers.modeling_outputs.MultipleChoiceModelOutput ortuple(torch.FloatTensor)
Atransformers.modeling_outputs.MultipleChoiceModelOutput or a tuple oftorch.FloatTensor (ifreturn_dict=False is passed or whenconfig.return_dict=False) comprising variouselements depending on the configuration (DistilBertConfig) and inputs.
loss (
torch.FloatTensorof shape(1,),optional, returned whenlabelsis provided) — Classification loss.logits (
torch.FloatTensorof shape(batch_size, num_choices)) —num_choices is the second dimension of the input tensors. (seeinput_ids above).Classification scores (before SoftMax).
hidden_states (
tuple(torch.FloatTensor),optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, +one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.FloatTensor),optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attentionheads.
TheDistilBertForMultipleChoice forward method, overrides the__call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps whilethe latter silently ignores them.
Examples:
>>>from transformersimport AutoTokenizer, DistilBertForMultipleChoice>>>import torch>>>tokenizer = AutoTokenizer.from_pretrained("distilbert-base-cased")>>>model = DistilBertForMultipleChoice.from_pretrained("distilbert-base-cased")>>>prompt ="In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced.">>>choice0 ="It is eaten with a fork and a knife.">>>choice1 ="It is eaten while held in the hand.">>>labels = torch.tensor(0).unsqueeze(0)# choice0 is correct (according to Wikipedia ;)), batch size 1>>>encoding = tokenizer([[prompt, choice0], [prompt, choice1]], return_tensors="pt", padding=True)>>>outputs = model(**{k: v.unsqueeze(0)for k, vin encoding.items()}, labels=labels)# batch size is 1>>># the linear classifier still needs to be trained>>>loss = outputs.loss>>>logits = outputs.logits
DistilBertForTokenClassification
classtransformers.DistilBertForTokenClassification
<source>(config: PreTrainedConfig)
Parameters
- config (PreTrainedConfig) —Model configuration class with all the parameters of the model. Initializing with a config file does notload the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.
The Distilbert transformer with a token classification head on top (a linear layer on top of the hidden-statesoutput) e.g. for Named-Entity-Recognition (NER) tasks.
This model inherits fromPreTrainedModel. Check the superclass documentation for the generic methods thelibrary implements for all its model (such as downloading or saving, resizing the input embeddings, pruning headsetc.)
This model is also a PyTorchtorch.nn.Module subclass.Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usageand behavior.
forward
<source>(input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Nonelabels: typing.Optional[torch.LongTensor] = Noneposition_ids: typing.Optional[torch.Tensor] = None**kwargs: typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs])→transformers.modeling_outputs.TokenClassifierOutput ortuple(torch.FloatTensor)
Parameters
- input_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.Indices can be obtained usingAutoTokenizer. SeePreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length),optional) —Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that arenot masked,
- 0 for tokens that aremasked.
- inputs_embeds (
torch.Tensorof shape(batch_size, sequence_length, hidden_size),optional) —Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. Thisis useful if you want more control over how to convertinput_idsindices into associated vectors than themodel’s internal embedding lookup matrix. - labels (
torch.LongTensorof shape(batch_size, sequence_length),optional) —Labels for computing the token classification loss. Indices should be in[0, ..., config.num_labels - 1]. - position_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.n_positions - 1].
Returns
transformers.modeling_outputs.TokenClassifierOutput ortuple(torch.FloatTensor)
Atransformers.modeling_outputs.TokenClassifierOutput or a tuple oftorch.FloatTensor (ifreturn_dict=False is passed or whenconfig.return_dict=False) comprising variouselements depending on the configuration (DistilBertConfig) and inputs.
loss (
torch.FloatTensorof shape(1,),optional, returned whenlabelsis provided) — Classification loss.logits (
torch.FloatTensorof shape(batch_size, sequence_length, config.num_labels)) — Classification scores (before SoftMax).hidden_states (
tuple(torch.FloatTensor),optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, +one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.FloatTensor),optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attentionheads.
TheDistilBertForTokenClassification forward method, overrides the__call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps whilethe latter silently ignores them.
Example:
>>>from transformersimport AutoTokenizer, DistilBertForTokenClassification>>>import torch>>>tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")>>>model = DistilBertForTokenClassification.from_pretrained("distilbert-base-uncased")>>>inputs = tokenizer(..."HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"...)>>>with torch.no_grad():... logits = model(**inputs).logits>>>predicted_token_class_ids = logits.argmax(-1)>>># Note that tokens are classified rather then input words which means that>>># there might be more predicted token classes than words.>>># Multiple token classes might account for the same word>>>predicted_tokens_classes = [model.config.id2label[t.item()]for tin predicted_token_class_ids[0]]>>>predicted_tokens_classes...>>>labels = predicted_token_class_ids>>>loss = model(**inputs, labels=labels).loss>>>round(loss.item(),2)...
DistilBertForQuestionAnswering
classtransformers.DistilBertForQuestionAnswering
<source>(config: PreTrainedConfig)
Parameters
- config (PreTrainedConfig) —Model configuration class with all the parameters of the model. Initializing with a config file does notload the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.
The Distilbert transformer with a span classification head on top for extractive question-answering tasks likeSQuAD (a linear layer on top of the hidden-states output to computespan start logits andspan end logits).
This model inherits fromPreTrainedModel. Check the superclass documentation for the generic methods thelibrary implements for all its model (such as downloading or saving, resizing the input embeddings, pruning headsetc.)
This model is also a PyTorchtorch.nn.Module subclass.Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usageand behavior.
forward
<source>(input_ids: typing.Optional[torch.Tensor] = Noneattention_mask: typing.Optional[torch.Tensor] = Noneinputs_embeds: typing.Optional[torch.Tensor] = Nonestart_positions: typing.Optional[torch.Tensor] = Noneend_positions: typing.Optional[torch.Tensor] = Noneposition_ids: typing.Optional[torch.Tensor] = None**kwargs: typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs])→transformers.modeling_outputs.QuestionAnsweringModelOutput ortuple(torch.FloatTensor)
Parameters
- input_ids (
torch.LongTensorof shape(batch_size, num_choices)) —Indices of input sequence tokens in the vocabulary.Indices can be obtained usingAutoTokenizer. SeePreTrainedTokenizer.encode() andPreTrainedTokenizer.call() for details.
- attention_mask (
torch.Tensorof shape(batch_size, sequence_length),optional) —Mask to avoid performing attention on padding token indices. Mask values selected in[0, 1]:- 1 for tokens that arenot masked,
- 0 for tokens that aremasked.
- inputs_embeds (
torch.FloatTensorof shape(batch_size, num_choices, hidden_size),optional) —Optionally, instead of passinginput_idsyou can choose to directly pass an embedded representation. Thisis useful if you want more control over how to convertinput_idsindices into associated vectors than themodel’s internal embedding lookup matrix. - start_positions (
torch.Tensorof shape(batch_size,),optional) —Labels for position (index) of the start of the labelled span for computing the token classification loss.Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequenceare not taken into account for computing the loss. - end_positions (
torch.Tensorof shape(batch_size,),optional) —Labels for position (index) of the end of the labelled span for computing the token classification loss.Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequenceare not taken into account for computing the loss. - position_ids (
torch.Tensorof shape(batch_size, sequence_length),optional) —Indices of positions of each input sequence tokens in the position embeddings. Selected in the range[0, config.n_positions - 1].
Returns
transformers.modeling_outputs.QuestionAnsweringModelOutput ortuple(torch.FloatTensor)
Atransformers.modeling_outputs.QuestionAnsweringModelOutput or a tuple oftorch.FloatTensor (ifreturn_dict=False is passed or whenconfig.return_dict=False) comprising variouselements depending on the configuration (DistilBertConfig) and inputs.
loss (
torch.FloatTensorof shape(1,),optional, returned whenlabelsis provided) — Total span extraction loss is the sum of a Cross-Entropy for the start and end positions.start_logits (
torch.FloatTensorof shape(batch_size, sequence_length)) — Span-start scores (before SoftMax).end_logits (
torch.FloatTensorof shape(batch_size, sequence_length)) — Span-end scores (before SoftMax).hidden_states (
tuple(torch.FloatTensor),optional, returned whenoutput_hidden_states=Trueis passed or whenconfig.output_hidden_states=True) — Tuple oftorch.FloatTensor(one for the output of the embeddings, if the model has an embedding layer, +one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.
attentions (
tuple(torch.FloatTensor),optional, returned whenoutput_attentions=Trueis passed or whenconfig.output_attentions=True) — Tuple oftorch.FloatTensor(one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).Attentions weights after the attention softmax, used to compute the weighted average in the self-attentionheads.
TheDistilBertForQuestionAnswering forward method, overrides the__call__ special method.
Although the recipe for forward pass needs to be defined within this function, one should call the
Moduleinstance afterwards instead of this since the former takes care of running the pre and post processing steps whilethe latter silently ignores them.
Example:
>>>from transformersimport AutoTokenizer, DistilBertForQuestionAnswering>>>import torch>>>tokenizer = AutoTokenizer.from_pretrained("distilbert-base-uncased")>>>model = DistilBertForQuestionAnswering.from_pretrained("distilbert-base-uncased")>>>question, text ="Who was Jim Henson?","Jim Henson was a nice puppet">>>inputs = tokenizer(question, text, return_tensors="pt")>>>with torch.no_grad():... outputs = model(**inputs)>>>answer_start_index = outputs.start_logits.argmax()>>>answer_end_index = outputs.end_logits.argmax()>>>predict_answer_tokens = inputs.input_ids[0, answer_start_index : answer_end_index +1]>>>tokenizer.decode(predict_answer_tokens, skip_special_tokens=True)...>>># target is "nice puppet">>>target_start_index = torch.tensor([14])>>>target_end_index = torch.tensor([15])>>>outputs = model(**inputs, start_positions=target_start_index, end_positions=target_end_index)>>>loss = outputs.loss>>>round(loss.item(),2)...