coco_instance_segmentation.ipynb

Kernel: Python 3

TIDE COCO Example

This example downloads some Mask R-CNN results files from detectron and then evaluates them using the COCO dataset.

Grabbing Mask R-CNN results

For demonstration purposes, we'll be using Mask R-CNN results (Model ID 36229740), but you'll want to use your own results in practice.

# Download the Mask R-CNN results to test on. Only for demonstration purposes.
import urllib.request # For downloading the sample Mask R-CNN annotations

bbox_file = 'mask_rcnn_bbox.json'
mask_file = 'mask_rcnn_mask.json'

urllib.request.urlretrieve('https://dl.fbaipublicfiles.com/detectron/35861795/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_1x.yaml.02_31_37.KqyEK4tT/output/test/coco_2014_minival/generalized_rcnn/bbox_coco_2014_minival_results.json', bbox_file)
urllib.request.urlretrieve('https://dl.fbaipublicfiles.com/detectron/35861795/12_2017_baselines/e2e_mask_rcnn_R-101-FPN_1x.yaml.02_31_37.KqyEK4tT/output/test/coco_2014_minival/generalized_rcnn/segmentations_coco_2014_minival_results.json', mask_file)

print('Results Downloaded!')

Results Downloaded!

Running TIDE

# Import the TIDE evaluation toolkit
from tidecv import TIDE

# Import the datasets we want to use
import tidecv.datasets as datasets

# Load the dataset
# This will automatically download COCO's 2017 annotations for you. You can specify your own COCO-style dataset too!
# See datasets.py for a list of all the supported datasets.
gt = datasets.COCO()

# Load the results in COCO format
bbox_results = datasets.COCOResult(bbox_file) # These files were downloaded above.
mask_results = datasets.COCOResult(mask_file) # Replace them with your own in practice.

# Create a TIDE object to use for evaluation
tide = TIDE()

# Run the evaluations on the standard COCO metrics (i.e., a range of AP with IoU thresholds [50:95])
tide.evaluate_range(gt, bbox_results, mode=TIDE.BOX ) # Several options are available here, see the functions
tide.evaluate_range(gt, mask_results, mode=TIDE.MASK) # evaluate and evaluate_range for more details.

# Summarize the evaluations run so far in the console
tide.summarize()

-- mask_rcnn_bbox --

bbox AP @ [50-95]: 40.01
                                         bbox AP @ [50-95]
===================================================================================================
  Thresh       50       55       60       65       70       75       80       85       90       95  
---------------------------------------------------------------------------------------------------
    AP      61.80    59.76    57.07    53.72    49.59    43.66    35.87    25.77    11.68     1.19  
===================================================================================================

                         Main Errors
=============================================================
  Type      Cls      Loc     Both     Dupe      Bkg     Miss  
-------------------------------------------------------------
   dAP     3.40     6.65     1.18     0.19     3.96     7.53  
=============================================================

        Special Error
=============================
  Type   FalsePos   FalseNeg  
-----------------------------
   dAP      16.28      15.57  
=============================

-- mask_rcnn_mask --

mask AP @ [50-95]: 35.92
                                         mask AP @ [50-95]
===================================================================================================
  Thresh       50       55       60       65       70       75       80       85       90       95  
---------------------------------------------------------------------------------------------------
    AP      58.30    55.61    52.48    48.82    44.05    37.95    30.33    20.96     9.54     1.17  
===================================================================================================

                         Main Errors
=============================================================
  Type      Cls      Loc     Both     Dupe      Bkg     Miss  
-------------------------------------------------------------
   dAP     3.05     9.38     0.58     0.32     4.31     7.43  
=============================================================

        Special Error
=============================
  Type   FalsePos   FalseNeg  
-----------------------------
   dAP      15.50      18.03  
=============================

# Plot summaries for the evaluations run so far
tide.plot()