list of efficient attention modules

• UpdatedAug 23, 2021
• Python

Abstractive and Extractive Text summarization using Transformers.

• UpdatedJun 9, 2023
• Jupyter Notebook

Master thesis with code investigating methods for incorporating long-context reasoning in low-resource languages, without the need to pre-train from scratch. We investigated if multilingual models could inherit these properties by making it an Efficient Transformer (s.a. the Longformer architecture).

• UpdatedAug 19, 2021
• Jupyter Notebook

2020 AI研习社 金融用户评论分类

• UpdatedMay 17, 2020
• Python

Convert pretrained RoBerta models to various long-document transformer models

• UpdatedApr 5, 2022
• Python

Longformer Encoder Decoder model for the legal domain, trained for long document abstractive summarization task.

• UpdatedFeb 26, 2021

using transformers to do text classification.

• UpdatedNov 10, 2021
• Jupyter Notebook

This GitHub repository implements a novel approach for detecting Initial Public Offering (IPO) underpricing using pre-trained Transformers. The models, extended to handle large S-1 filings, leverage both textual information and financial indicators, outperforming traditional machine learning methods.

• UpdatedDec 2, 2024
• Python

Industrial Text Scoring using Multimodal Deep Natural Language Processing 🚀 | Code for IEA AIE 2022 paper

• UpdatedJan 1, 2023
• Python

[제 13회 투빅스 컨퍼런스] YoYAK - Yes or Yes, Attention with gap-sentence for Korean long sequence

• UpdatedApr 4, 2022
• Jupyter Notebook

A summarization website that can generate summaries from either YouTube videos or PDF files.

• UpdatedSep 25, 2024
• Python

Fine-tuned Longformer for Summarization of Machine Learning Articles

• UpdatedJun 22, 2023
• Jupyter Notebook

Kaggle NLP competition - Top 2% solution (36/2060)

• UpdatedMar 17, 2022
• Jupyter Notebook

• UpdatedAug 30, 2024
• Python

This project applies the Longformer model to sentiment analysis using the IMDB movie review dataset. The Longformer model, introduced in "Longformer: The Long-Document Transformer," tackles long document processing with sliding-window and global attention mechanisms. The implementation leverages PyTorch, following the paper's architecture

• UpdatedApr 7, 2023
• Python

Project as part of COMP34812: Natural Language Understanding

• UpdatedMay 5, 2023
• Jupyter Notebook

A hyperpartisan news article classification system using BERT-based techniques. The goal was to leverage state-of-the-art transformer models like BERT, ROBERTa, and Longformer to accurately classify news articles as hyperpartisan or non-hyperpartisan.

• UpdatedOct 24, 2023
• Jupyter Notebook

This project was developed for a Kaggle competition focused on detecting Personally Identifiable Information (PII) in student writing. The primary objective was to build a robust model capable of identifying PII with high recall. The DeBERTa v3 transformer model was chosen for this task after comparing its performance with other transformer models.

• UpdatedJun 28, 2024
• Jupyter Notebook

Longformer model trained on HotpotQA

• UpdatedJan 8, 2023
• Python

Focus - Understanding contextual retrievability.

• UpdatedNov 5, 2024
• Jupyter Notebook