ReFocus

🔔News

Introduction

Download Training Data

ReFocus Prompting

Installation

conda create -n refocus python=3.11conda activate refocuspip install pyautogen==0.3.0pip install 'pyautogen[jupyter-executor]'pip install Pillow joblib matplotlib opencv-python numpy networkx scipy datasets

Quick Start

Task Data

Run a Task

export OPENAI_API_KEY=<your_key>

python src/main_chartQA.pypython src/main_tablevqa.pypython src/main_charxiv.py

ReFocus Finetuning

Download the Finetuned Model

<|image|>\n{question}\nThought:

<|image_1|>\n{question}\nThought: The areas to focus on in the image have bounding box coordinates:

Finetune Quickstart

git clone https://github.com/microsoft/Phi-3CookBook.git# create a new conda environmentconda create -n phi3v python=3.10conda activate phi3v# install pytorchconda install pytorch==2.1.2 torchvision==0.16.2 torchaudio==2.1.2 pytorch-cuda=12.1 -c pytorch -c nvidia# other libraries needed to run the example codepip install -r requirements.txt# (optional) flash attention -- Ampere+ GPUs (e.g., A100, H100)pip install ninjaMAX_JOBS=32 pip install flash-attn==2.4.2 --no-build-isolation# (optional) QLoRA -- Turing+ GPUs (e.g., RTX 8000)pip install bitsandbytes==0.43.1

mv finetune_hf_trainer_chartqa_vcot.py Phi-3CookBook/code/04.Finetuning/vision_finetuning/

cd Phi-3CookBook/code/04.Finetuning/vision_finetuningpython -m torch.distributed.run --nproc_per_node=8 finetune_hf_trainer_chartqa_vcot.py --full_train --data_dir data/chartqa_vcot --bf16 --use_flash_attention --batch_size 48 --output_dir outputs/chartqa_vcot_loop --learning_rate 1e-6 --num_train_epochs 2 --output_bbox 1

Coordinate Acquisition

python helper/get_coordinates_for_chart.pypython helper/get_coordinates_for_table.py