Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also orlearn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also.Learn more about diff comparisons here.
base repository:huggingface/swift-transformers
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base:main
Choose a base ref
Loading
...
head repository:huggingface/swift-transformers
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare:bpe-tokenizers
Choose a head ref
Loading
  • 17commits
  • 0files changed
  • 1contributor

Commits on Jul 17, 2023

  1. BPE tokenizer tests.

    Llama fails because we don't find normalizers or pre-processors.
    @pcuenca
    pcuenca committedJul 17, 2023
    Configuration menu
    Copy the full SHA
    b04e144View commit details
    Browse the repository at this point in the history

  2. Load tokenizers using Hub config.

    Note that some models still don't have a tokenizer_config.json file.
    @pcuenca
    pcuenca committedJul 17, 2023
    Configuration menu
    Copy the full SHA
    43384caView commit details
    Browse the repository at this point in the history

  3. Make Llama tokenization work.

    Breaks the other bpe tokenizers.
    @pcuenca
    pcuenca committedJul 17, 2023
    Configuration menu
    Copy the full SHA
    a0f7b1dView commit details
    Browse the repository at this point in the history

  4. MissingNormalizer.swift

    @pcuenca
    pcuenca committedJul 17, 2023
    Configuration menu
    Copy the full SHA
    2b18425View commit details
    Browse the repository at this point in the history

  5. Add (some) PostProcessors and Decoders.

    Llama tests now pass.
    @pcuenca
    pcuenca committedJul 17, 2023
    Configuration menu
    Copy the full SHA
    5a89bd2View commit details
    Browse the repository at this point in the history

Commits on Jul 18, 2023

  1. Configuration menu
    Copy the full SHA
    34f7c15View commit details
    Browse the repository at this point in the history

Commits on Jul 31, 2023

  1. Configuration menu
    Copy the full SHA
    7392e2fView commit details
    Browse the repository at this point in the history

  2. Support fallback tokenizer_config.json

    Currently available for gpt2, will add for others if necessary.
    @pcuenca
    pcuenca committedJul 31, 2023
    Configuration menu
    Copy the full SHA
    fe1d6afView commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    88c3767View commit details
    Browse the repository at this point in the history

  4. Support generic "PreTrainedTokenizer" class.

    Some models use this name instead of a concrete tokenizer class name.Currently using BPETokenizer as its implementation, to be fixed when wegeneralize to other tokenizer types.
    @pcuenca
    pcuenca committedJul 31, 2023
    Configuration menu
    Copy the full SHA
    2389579View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    3d2763eView commit details
    Browse the repository at this point in the history

Commits on Aug 1, 2023

  1. Configuration menu
    Copy the full SHA
    cc23e17View commit details
    Browse the repository at this point in the history

  2. Fix test case to include specials.

    TODO: support stripping special tokens.
    @pcuenca
    pcuenca committedAug 1, 2023
    Configuration menu
    Copy the full SHA
    2e2b9c6View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    1143edfView commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    67f4628View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    8f9e974View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    e6701a8View commit details
    Browse the repository at this point in the history
Loading
[8]ページ先頭
©2009-2025 Movatter.jp