ALBERT

Parameters

• vocab_size (int,optional, defaults to 30000) —Vocabulary size of the ALBERT model. Defines the number of different tokens that can be represented by theinputs_ids passed when callingAlbertModel orTFAlbertModel.
• embedding_size (int,optional, defaults to 128) —Dimensionality of vocabulary embeddings.
• hidden_size (int,optional, defaults to 4096) —Dimensionality of the encoder layers and the pooler layer.
• num_hidden_layers (int,optional, defaults to 12) —Number of hidden layers in the Transformer encoder.
• num_hidden_groups (int,optional, defaults to 1) —Number of groups for the hidden layers, parameters in the same group are shared.
• num_attention_heads (int,optional, defaults to 64) —Number of attention heads for each attention layer in the Transformer encoder.
• intermediate_size (int,optional, defaults to 16384) —The dimensionality of the “intermediate” (often named feed-forward) layer in the Transformer encoder.
• inner_group_num (int,optional, defaults to 1) —The number of inner repetition of attention and ffn.
• hidden_act (str orCallable,optional, defaults to"gelu_new") —The non-linear activation function (function or string) in the encoder and pooler. If string,"gelu","relu","silu" and"gelu_new" are supported.
• hidden_dropout_prob (float,optional, defaults to 0) —The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
• attention_probs_dropout_prob (float,optional, defaults to 0) —The dropout ratio for the attention probabilities.
• max_position_embeddings (int,optional, defaults to 512) —The maximum sequence length that this model might ever be used with. Typically set this to something large(e.g., 512 or 1024 or 2048).
• type_vocab_size (int,optional, defaults to 2) —The vocabulary size of thetoken_type_ids passed when callingAlbertModel orTFAlbertModel.
• initializer_range (float,optional, defaults to 0.02) —The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
• layer_norm_eps (float,optional, defaults to 1e-12) —The epsilon used by the layer normalization layers.
• classifier_dropout_prob (float,optional, defaults to 0.1) —The dropout ratio for attached classifiers.
• pad_token_id (int,optional, defaults to 0) —Padding token id.
• bos_token_id (int,optional, defaults to 2) —Beginning of stream token id.
• eos_token_id (int,optional, defaults to 3) —End of stream token id.

>>>from transformersimport AlbertConfig, AlbertModel>>># Initializing an ALBERT-xxlarge style configuration>>>albert_xxlarge_configuration = AlbertConfig()>>># Initializing an ALBERT-base style configuration>>>albert_base_configuration = AlbertConfig(...    hidden_size=768,...    num_attention_heads=12,...    intermediate_size=3072,...)>>># Initializing a model (with random weights) from the ALBERT-base style configuration>>>model = AlbertModel(albert_xxlarge_configuration)>>># Accessing the model configuration>>>configuration = model.config

Parameters

• vocab_file (str) —SentencePiece file (generally has a.spm extension) thatcontains the vocabulary necessary to instantiate a tokenizer.
• do_lower_case (bool,optional, defaults toTrue) —Whether or not to lowercase the input when tokenizing.
• remove_space (bool,optional, defaults toTrue) —Whether or not to strip the text when tokenizing (removing excess spaces before and after the string).
• keep_accents (bool,optional, defaults toFalse) —Whether or not to keep accents when tokenizing.
• bos_token (str,optional, defaults to"[CLS]") —The beginning of sequence token that was used during pretraining. Can be used a sequence classifier token.

  When building a sequence using special tokens, this is not the token that is used for the beginning ofsequence. The token used is thecls_token.

• eos_token (str,optional, defaults to"[SEP]") —The end of sequence token. .. note:: When building a sequence using special tokens, this is not the tokenthat is used for the end of sequence. The token used is thesep_token.
• unk_token (str,optional, defaults to"<unk>") —The unknown token. A token that is not in the vocabulary cannot be converted to an ID and is set to be thistoken instead.
• sep_token (str,optional, defaults to"[SEP]") —The separator token, which is used when building a sequence from multiple sequences, e.g. two sequences forsequence classification or for a text and a question for question answering. It is also used as the lasttoken of a sequence built with special tokens.
• pad_token (str,optional, defaults to"<pad>") —The token used for padding, for example when batching sequences of different lengths.
• cls_token (str,optional, defaults to"[CLS]") —The classifier token which is used when doing sequence classification (classification of the whole sequenceinstead of per-token classification). It is the first token of the sequence when built with special tokens.
• mask_token (str,optional, defaults to"[MASK]") —The token used for masking values. This is the token used when training this model with masked languagemodeling. This is the token which the model will try to predict.

(token_ids_0: listtoken_ids_1: typing.Optional[list[int]] = None)→List[int]

Parameters

• loss (*optional*, returned whenlabels is provided,torch.FloatTensor of shape(1,)) —Total loss as the sum of the masked language modeling loss and the next sequence prediction(classification) loss.
• prediction_logits (torch.FloatTensor of shape(batch_size, sequence_length, config.vocab_size)) —Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax).
• sop_logits (torch.FloatTensor of shape(batch_size, 2)) —Prediction scores of the next sequence prediction (classification) head (scores of True/False continuationbefore SoftMax).
• hidden_states (tuple[torch.FloatTensor],optional, returned whenoutput_hidden_states=True is passed or whenconfig.output_hidden_states=True) —Tuple oftorch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, +one for the output of each layer) of shape(batch_size, sequence_length, hidden_size).

  Hidden-states of the model at the output of each layer plus the optional initial embedding outputs.

• attentions (tuple[torch.FloatTensor],optional, returned whenoutput_attentions=True is passed or whenconfig.output_attentions=True) —Tuple oftorch.FloatTensor (one for each layer) of shape(batch_size, num_heads, sequence_length, sequence_length).

  Attentions weights after the attention softmax, used to compute the weighted average in the self-attentionheads.

Parameters

• config (AlbertConfig) —Model configuration class with all the parameters of the model. Initializing with a config file does notload the weights associated with the model, only the configuration. Check out thefrom_pretrained() method to load the model weights.

(input_ids: typing.Optional[torch.LongTensor] = Noneattention_mask: typing.Optional[torch.FloatTensor] = Nonetoken_type_ids: typing.Optional[torch.LongTensor] = Noneposition_ids: typing.Optional[torch.LongTensor] = Noneinputs_embeds: typing.Optional[torch.FloatTensor] = Nonelabels: typing.Optional[torch.LongTensor] = None**kwargs: typing_extensions.Unpack[transformers.utils.generic.TransformersKwargs])→transformers.models.albert.modeling_albert.AlbertForPreTrainingOutput ortuple(torch.FloatTensor)

Example:

>>>from transformersimport AutoTokenizer, AlbertForMultipleChoice>>>import torch>>>tokenizer = AutoTokenizer.from_pretrained("albert/albert-xxlarge-v2")>>>model = AlbertForMultipleChoice.from_pretrained("albert/albert-xxlarge-v2")>>>prompt ="In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced.">>>choice0 ="It is eaten with a fork and a knife.">>>choice1 ="It is eaten while held in the hand.">>>labels = torch.tensor(0).unsqueeze(0)# choice0 is correct (according to Wikipedia ;)), batch size 1>>>encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)>>>outputs = model(**{k: v.unsqueeze(0)for k, vin encoding.items()}, labels=labels)# batch size is 1>>># the linear classifier still needs to be trained>>>loss = outputs.loss>>>logits = outputs.logits