Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

add FastSentenceTransformer for easily finetuning SentenceTransformer models#3719

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Open
electroglyph wants to merge20 commits intounslothai:main
base:main
Choose a base branch
Loading
fromelectroglyph:FST

Conversation

@electroglyph
Copy link

@electroglyphelectroglyph commentedDec 12, 2025•
edited
Loading

supersedes#3718

example training code:

fromunslothimportFastSentenceTransformerfromsentence_transformersimport (SentenceTransformerTrainer,SentenceTransformerTrainingArguments,)fromdatasetsimportDatasetimporttorchmodel_name="Snowflake/snowflake-arctic-embed-m-v1.5"model=FastSentenceTransformer.from_pretrained(model_name,load_in_4bit=True,device_map="cuda",)model=FastSentenceTransformer.get_peft_model(model,r=16,target_modules=["query","key","value","dense"],lora_alpha=16,lora_dropout=0,bias="none",# task_type="FEATURE_EXTRACTION",)train_dataset=Dataset.from_dict(    {"sentence_A": ["The cat sits outside","A man is playing guitar","I love pasta",        ],"sentence_B": ["A man is playing guitar","The woman loves that cat","Do you like pizza?",        ],"label": [0.0,0.5,1.0],    })fromsentence_transformers.lossesimportCoSENTLossloss=CoSENTLoss(model)args=SentenceTransformerTrainingArguments(output_dir="test_trainer_output",num_train_epochs=1,per_device_train_batch_size=2,learning_rate=2e-4,fp16=nottorch.cuda.is_bf16_supported(),bf16=torch.cuda.is_bf16_supported(),logging_steps=1,save_strategy="no",report_to="none",)trainer=SentenceTransformerTrainer(model=model,args=args,train_dataset=train_dataset,loss=loss,)trainer.train()print("Training finished successfully!")

shimmyshimmer reacted with thumbs up emojishimmyshimmer reacted with heart emoji
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello@electroglyph, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly expands Unsloth's capabilities by introducingFastSentenceTransformer, a new class designed to streamline the finetuning ofSentenceTransformer models. By integrating Unsloth's optimized model loading and PEFT (LoRA) application, users can now efficiently train embedding models with reduced memory footprint and improved speed, broadening the range of models that can benefit from Unsloth's performance enhancements.

Highlights

  • New FastSentenceTransformer Class: IntroducesFastSentenceTransformer to enable easy and optimized finetuning ofSentenceTransformer models within the Unsloth framework.
  • Optimized Model Loading and PEFT Integration: Leverages Unsloth'sFastModel for efficient loading of base models and seamlessly integrates PEFT (LoRA) for memory-efficient finetuning of the underlying transformer.
  • Automatic Pooling Mode Detection: Includes logic to automatically detect the pooling mode (e.g., 'cls', 'mean', 'max') from theSentenceTransformer model's configuration, ensuring correct setup.
  • Seamless SentenceTransformer Wrapper: Wraps the loaded base model into asentence_transformers.SentenceTransformer instance, complete withTransformer,Pooling, andNormalize modules, making it compatible with thesentence-transformers ecosystem.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on ourdocumentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either/gemini <command> or@gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

FeatureCommandDescription
Code Review/gemini reviewPerforms a code review for the current pull request in its current state.
Pull Request Summary/gemini summaryProvides a summary of the current pull request in its current state.
Comment@gemini-code-assistResponds in comments when explicitly tagged, both in pull request comments and review comments.
Help/gemini helpDisplays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a.gemini/ folder in the base of the repository. Detailed instructions can be foundhere.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on@gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign uphere.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with theGemini Code Assist IDE Extension.

Footnotes

  1. Review thePrivacy Notices,Generative AI Prohibited Use Policy,Terms of Service, and learn how to configure Gemini Code Assist in GitHubhere. Gemini can make mistakes, so double check it anduse code with caution.↩

@electroglyph
Copy link
Author

just getting this started, i'll get to the docs (and any suggestions) tomorrow

Sneakr reacted with rocket emoji

Copy link
Contributor

@gemini-code-assistgemini-code-assistbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Code Review

This pull request introducesFastSentenceTransformer, a new class for easily finetuning SentenceTransformer models with Unsloth's optimizations. The implementation is solid, providingfrom_pretrained andget_peft_model methods that correctly integrate with the existingFastModel framework. The code handles loading quantized models, applying PEFT, and constructing aSentenceTransformer object. My review includes a couple of suggestions to improve code style and maintainability, such as moving imports to the top level and refactoring a conditional block to be more concise.

@shimmyshimmer
Copy link
Collaborator

Thank you amazing!! Please let us know if you'd like to collab on a blog as well :)

Sneakr reacted with hooray emoji

@electroglyph
Copy link
Author

electroglyph commentedDec 13, 2025•
edited
Loading

Thank you amazing!! Please let us know if you'd like to collab on a blog as well :)

absolutely, that would be great!

unslothai/unsloth-zoo#383 will add XLMRobertaModel support

shimmyshimmer reacted with thumbs up emoji

@electroglyph
Copy link
Author

here is a colab notebook to test the current code:https://colab.research.google.com/drive/1Wu-lB33o8JdeKT1R38uGLbd0yCqluYML?usp=sharing

@Datta0
Copy link
Collaborator

Ok I tested your notebook and it seems to work fine. Imma review it in the morning

Copy link
Collaborator

@Datta0Datta0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Great work! I have some queries and comments :)

@electroglyphelectroglyph marked this pull request as ready for reviewDecember 16, 2025 11:08
Copy link

@chatgpt-codex-connectorchatgpt-codex-connectorbot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When yousign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

@electroglyph
Copy link
Author

electroglyph commentedDec 17, 2025•
edited
Loading

here's current compatibility status, i tested training the top 100 encoder embedding models (by download number) and 72 out of 100 can be trained right now:https://0x0.st/PrxF.txt

i'm going to see if i can get some easy wins and increase it a bit

after mpnet patch, up to 76:https://0x0.st/PrYx.txt

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@Datta0Datta0Datta0 left review comments

@chatgpt-codex-connectorchatgpt-codex-connector[bot]chatgpt-codex-connector[bot] left review comments

+1 more reviewer

@gemini-code-assistgemini-code-assist[bot]gemini-code-assist[bot] left review comments

Reviewers whose approvals may not affect merge requirements

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@electroglyph@shimmyshimmer@Datta0
[8]ページ先頭
©2009-2025 Movatter.jp