Repository files navigation

LoRA Project Website

This is the code used to generate the LoRA Dataset.

You can use this code to generate complex logical questions and related images, like this:

Q: If we do not have milk, is there another dairy product that does not necessarily contain fat but is rich in protein that can be subsituted for breakfast?
A: Yogurt

Q: Can we use the food between eggs and bread to make a meal for vegetarians?
A: No

First, we construct a knowledge base by using Owlready2.

cd ontologypython ontology.py

Next we generate questions based on the ontology and logical operators.You can generate logical questions with 2 logical operators and 3 logical operators.

You can generate basic logical questions by using:

cd question_generationpython generate_questions_logic2.py

The detailed steps can also be found in thebasic_logic_questions_generation.ipynb file.

More complex logical reasoning questions are generated by using:

cd question_generationpython generate_questions_logic3.py

The detailed steps can also be found in thecomplex_reasoning_questions_generation.ipynb file.

After generating the questions using the template, you can convert these questions into natural language questions using the following script:

cd question_generationpython generate_questions_natural_language_logic2.pypython generate_questions_natural_language_logic3.py

The answers are batch generated by using SparQL_query algorithm.

The algorithm can be found here:

cd answer_generationSparQL_query.ipynb

To generate the answers for logic 2 questions, run:

cd answer_generationpython generate_answers_logic2.py

To generate the answers for logic 3 questions, run:

cd answer_generationpython generate_answers_logic3.py

The details can also be found in theanswer_generation.ipynb file.

The images are generated based on answers and noise objects. Each image contains the answers objects and the noise objects. We use Blender script to generate the realistic kitchen scene.

Firstly, install blender v2.93.

Secondly, you need to generate visible_objects.csv file by using:

cd image_generationpython visible_object_processing.py

The generated visible objects file are saved in the folder:

cd image_generationlogic2_and_visible.csv

You can choose different blender background. The blender background file isVqa_Background_1.blend. We provided four kitchen background scenarios in the fileimage_generation/blender_background/Vqa_Backgrounds_Full.blend.

You can then render some images by using:

cd image_generationblender Vqa_Background_1.blend --background --python blend_image_generation.py

https://drive.google.com/drive/folders/1H4msZ4ae1msEDApxOT7cr1A1QwQ33Pkn

@inproceedings{gao2023lora,  title={LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering},  author={Gao, Jingying and Wu, Qi and Blair, Alan and Pagnucco, Maurice},  booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},  year={2023}}