- Notifications
You must be signed in to change notification settings - Fork26
A package to read and convert object detection datasets (COCO, YOLO, PascalVOC, LabelMe, CVAT, OpenImage, ...) and evaluate them with COCO and PascalVOC metrics.
License
laclouis5/globox
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
This framework can:
- parse all kinds of object detection datasets (ImageNet, COCO, YOLO, PascalVOC, OpenImage, CVAT, LabelMe, etc.) and show statistics,
- convert them to other formats (ImageNet, COCO, YOLO, PascalVOC, OpenImage, CVAT, LabelMe, etc.),
- and evaluate predictions using standard object detection metrics such as
$AP_{[.5:.05:.95]}$ ,$AP_{50}$ ,$mAP$ ,$AR_{1}$ ,$AR_{10}$ ,$AR_{100}$ .
This framework can be used both as a library in your own code and as a command line tool. This tool is designed to be simple to use, fast and correct.
You can install the package using pip:
pip install globox
The library has three main components:
BoundingBox: represents a bounding box with a label and an optional confidence scoreAnnotation: represent the bounding boxes annotations for one imageAnnotationSet: represents annotations for a set of images (a database)
TheAnnotationSet class contains static methods to read different dataset formats:
# COCOcoco=AnnotationSet.from_coco(file_path="path/to/file.json")# YOLOv5yolo=AnnotationSet.from_yolo_v5(folder="path/to/files/",image_folder="path/to/images/")# Pascal VOCpascal=AnnotationSet.from_pascal_voc(folder="path/to/files/")
Annotation offers file-level granularity for compatible datasets:
annotation=Annotation.from_labelme(file_path="path/to/file.xml")
For more specific implementations theBoundingBox class contains lots of utilities to parse bounding boxes in different formats, like thecreate() method.
AnnotationsSets are set-like objects. They can be combined and annotations can be added:
gts=coco|yologts.add(annotation)
Iterators and efficient lookup byimage_id's are easy to use:
ifannotationingts:print("This annotation is present.")if"image_123.jpg"ingts.image_ids:print("Annotation of image 'image_123.jpg' is present.")forboxingts.all_boxes:print(box.label,box.area,box.is_ground_truth)forannotationingts:nb_boxes=len(annotation.boxes)print(f"{annotation.image_id}:{nb_boxes} boxes")
Datasets stats can printed to the console:
coco_gts.show_stats()
Database Stats ┏━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━┓┃ Label ┃ Images ┃ Boxes ┃┡━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━┩│ aeroplane │ 10 │ 15 ││ bicycle │ 7 │ 14 ││ bird │ 4 │ 6 ││ boat │ 7 │ 11 ││ bottle │ 9 │ 13 ││ bus │ 5 │ 6 ││ car │ 6 │ 14 ││ cat │ 4 │ 5 ││ chair │ 9 │ 15 ││ cow │ 6 │ 14 ││ diningtable │ 7 │ 7 ││ dog │ 6 │ 8 ││ horse │ 7 │ 7 ││ motorbike │ 3 │ 5 ││ person │ 41 │ 91 ││ pottedplant │ 6 │ 7 ││ sheep │ 4 │ 10 ││ sofa │ 10 │ 10 ││ train │ 5 │ 6 ││ tvmonitor │ 8 │ 9 │├─────────────┼────────┼───────┤│ Total │ 100 │ 273 │└─────────────┴────────┴───────┘Datasets can be converted to and saved in other formats:
# ImageNetgts.save_imagenet(save_dir="pascalVOC_db/")# YOLO Darknetgts.save_yolo_darknet(save_dir="yolo_train/",label_to_id={"cat":0,"dog":1,"racoon":2})# YOLOv5gts.save_yolo_v5(save_dir="yolo_train/",label_to_id={"cat":0,"dog":1,"racoon":2},)# CVATgts.save_cvat(path="train.xml")
COCO Evaluation is also supported:
evaluator=COCOEvaluator(ground_truths=gts,predictions=dets)ap=evaluator.ap()ar_100=evaluator.ar_100()ap_75=evaluator.ap_75()ap_small=evaluator.ap_small()...
All COCO standard metrics can be displayed in a pretty printed table with:
evaluator.show_summary()
which outputs:
COCO Evaluation┏━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━┳...┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓┃ Label ┃ AP 50:95 ┃ AP 50 ┃ ┃ AR S ┃ AR M ┃ AR L ┃┡━━━━━━━━━━━╇━━━━━━━━━━╇━━━━━━━━╇...╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩│ airplane │ 22.7% │ 25.2% │ │ nan% │ 90.0% │ 0.0% ││ apple │ 46.4% │ 57.4% │ │ 48.5% │ nan% │ nan% ││ backpack │ 54.8% │ 85.1% │ │ 100.0% │ 72.0% │ 0.0% ││ banana │ 73.6% │ 96.4% │ │ nan% │ 100.0% │ 70.0% │. . . . . . . .. . . . . . . .. . . . . . . .├───────────┼──────────┼────────┼...┼────────┼────────┼────────┤│ Total │ 50.3% │ 69.7% │ │ 65.4% │ 60.3% │ 55.3% │└───────────┴──────────┴────────┴...┴────────┴────────┴────────┘The array of results can be saved in CSV format:
evaluator.save_csv("where/to/save/results.csv")
Custom evaluations can be achieved with:
evaluation=evaluator.evaluate(iou_threshold=0.33,max_detections=1_000,size_range=(0.0,10_000))ap=evaluation.ap()cat_ar=evaluation["cat"].ar
Evaluations are cached by(iou_threshold, max_detections, size_range) keys. This means that repetead queries to the evaluator are fast!
If you only need to use Globox from the command line like an application, you can install the package throughpipx:
pipx install globox
Globox will then be in your shell path and usable from anywhere.
Get a summary of annotations for one dataset:
globox summary /yolo/folder/ --format yolo
Convert annotations from one format to another one:
globox convert input/yolo/folder/ output_coco_file_path.json --format yolo --save_fmt coco
Evaluate a set of detections with COCO metrics, display them and save them in a CSV file:
globox evaluate groundtruths/ predictions.json --format yolo --format_dets coco -s results.csv
Show the help message for an exhaustive list of options:
globox summary -hglobox convert -hglobox evaluate -h
Clone the repo with its test data:
git clone https://github.com/laclouis5/globox --recurse-submodules=tests/globox_test_datacd globoxInstall dependencies withuv:
uv sync --dev
Run the tests:
uv run pytest tests
Speed benchmark can be executed with:
uv run python tests/benchmark.py -n 5
The following speed test is performed using Python 3.11 andtimeit with 5 iterations on a 2021 MacBook Pro 14" (M1 Pro 8 Cores and 16 GB of RAM). The dataset is COCO 2017 Validation which comprises 5k images and 36 781 bounding boxes.
| Task | COCO | CVAT | OpenImage | LabelMe | PascalVOC | YOLO | TXT |
|---|---|---|---|---|---|---|---|
| Parsing | 0.22s | 0.12s | 0.44s | 0.60s | 0.97s | 1.45s | 1.12s |
| Saving | 0.32s | 0.17s | 0.14s | 1.06s | 1.08s | 0.91s | 0.85s |
AnnotationSet.show_stats(): 0.02 s- Evalaution: 0.30 s
- Basic data structures and utilities
- Parsers (ImageNet, COCO, YOLO, Pascal, OpenImage, CVAT, LabelMe)
- Parser tests
- Database summary and stats
- Database converters
- Visualization options
- COCO Evaluation
- Tests with a huge load (5k images)
- CLI interface
- Make
image_sizeoptional and raise err when required (bbox conversion) - Make file saving atomic with a temporary to avoid file corruption
- Pip package!
- PascalVOC Evaluation
- Parsers for TFRecord and TensorFlow
- UI interface?
This repo is based on the work ofRafael Padilla.
Feel free to contribute, any help you can offer with this project is most welcome.
About
A package to read and convert object detection datasets (COCO, YOLO, PascalVOC, LabelMe, CVAT, OpenImage, ...) and evaluate them with COCO and PascalVOC metrics.