Repository files navigation

A repository to build aSmall Language Model from scratch, inspired byQWEN 3 andGPT 2.
This project focuses on implementing and experimenting with different tokenizers and data preprocessing strategies to prepare text data for training.

• Python
• HuggingFace

• 📂Data Load & Preprocessing
  Efficient pipeline to load raw text data and prepare it for training.

• 🔎Data Diversity Check (MinHash)
  UsesMinHash to detect near-duplicate samples and ensure dataset diversity.

• 🔤Custom BPE Tokenizer
  Implements Byte-Pair Encoding (BPE) tokenizer from scratch for experimentation.

• 📘BLT Tokenizer
  Implementation ofFacebook Research's BLT Tokenizer.

• 📝SentencePiece Tokenizer
  Integration ofGoogle's SentencePiece Tokenizer.

1. Clone the repository:

   git clone https://github.com/ltrc/mini-project-slm-sourabiiit7.gitcd small-language-model

2. Install dependencies:

   pip install -r requirements.txt

3. Update configuration:Modify parameters as needed in:

   config/config.json

4. Run the project:

   python src/main.py

5. To run Perform PEFT on Google colab:

   Run the cells in  PEFT.ipynb

• Facebook Research BLT
• Google SentencePiece
• MinHash (Datasketch library)
• https://github.com/rasbt/LLMs-from-scratch

• ChatGPT
• GitHub Copilot
• Google Translate

• Google Colab Free Tier
• Google Drive For storage
• Google Translate