YOLO V3 Trained on Open Images Data

Contributed by: Julian W. Francis

Detect and localize objects in an image

YOLO (You Only Look Once) Version 3 is an object detection model published in April 2018. It is a single-stage architecture that goes straight from image pixels to bounding box coordinates and class probabilities. Compared to previous versions of YOLO, it performs bounding box regression and classification at three different scales and uses three anchor boxes instead of two. Additionally, residual blocks and skip connections are added and L2 losses for object confidence and class predictions are now replaced by cross-entropy error terms.

Number of layers: 349 | Parameter count: 65,252,682 | Trained size: 263 MB |

Training Set Information

Open Images, a dataset for image recognition, segmentation and captioning, consisting a total of 16 million bounding boxes for 600 object classes on 1.9 million images.

Performance

This model achieves an mAP-50 of 51.5% and an AP50 of 57.9% on the MS-COCO 2014 test set.

Examples

Download Example Notebook

Open in Wolfram Cloud

Resource retrieval

Get the pre-trained net:

In[1]:=

Out[1]=

Label list

Define the label list for this model. Integers in the model’s output correspond to elements in the label list:

In[2]:=

Evaluation function

Write an evaluation function to scale the result to the input image size and suppress the least probable detections:

In[3]:=

$ClearAll[IoU] IoU := IoU = With[ { IoUCompiled = Compile[ { {box1, _Real, 2}, {box2, _Real, 2} }, Module[ {area1, area2, x1, y1, x2, y2, w, h, int}, area1 = (box1[[2, 1]] - box1[[1, 1]])*(box1[[2, 2]] - box1[[1, 2]]); area2 = (box2[[2, 1]] - box2[[1, 1]])*(box2[[2, 2]] - box2[[1, 2]]); x1 = Max[box1[[1, 1]], box2[[1, 1]]]; y1 = Max[box1[[1, 2]], box2[[1, 2]]]; x2 = Min[box1[[2, 1]], box2[[2, 1]]]; y2 = Min[box1[[2, 2]], box2[[2, 2]]]; w = Max[0., x2 - x1]; h = Max[0., y2 - y1]; int = w*h; int/(area1 + area2 - int) ], RuntimeAttributes -> {Listable}, Parallelization -> True, RuntimeOptions -> "Speed" ] }, IoUCompiled @@ Replace[{##}, Rectangle -> List, Infinity, Heads -> True] & ];$

In[4]:=

$nonMaxSuppression[nmsThreshold_ ][dets_] := DeleteCases[ Function[detection, { detection[[1]], Function[ overlapBoxLabels, Select[ detection[[2]], #[[2]] > Max@Extract[ overlapBoxLabels[[All, All, 2]], Position[overlapBoxLabels[[All, All, 1]], #[[1]]] ] & ] ] [Select[ dets, (IoU[detection[[1]], #[[1]]] > nmsThreshold && ! (detection[[1]] === #[[1]])) &][[All, 2]]] } ] /@ dets, {_, {}} ];$

In[5]:=

$netOutputDecoder[threshold_ : .5][output_] := Module[{probs = output["Objectness"]*output["ClassProb"], detectionBoxes}, detectionBoxes = Union@Flatten@ SparseArray[UnitStep[probs - threshold]]["NonzeroPositions"][[ All, 1]]; Map[ Function[{detectionBox}, { Rectangle @@ output["Boxes"][[detectionBox]], Map[ {labels[[#]], probs[[detectionBox, #]]} &, Flatten@Position[probs[[detectionBox]], x_ /; x > threshold] ] } ], detectionBoxes ] ]; imageConformer[dims_, fitting_][image_] := First[ConformImages[{image}, dims, fitting, Padding -> 0.5]]; deconformRectangles[{}, _, _, _] := {}; deconformRectangles[rboxes_List, image_Image, netDims_List, "Fit"] := With[{netAspectRatio = netDims[[2]]/netDims[[1]]}, With[ { boxes = Map[{#[[1]], #[[2]]} &, rboxes], padding = If[ImageAspectRatio[image] < netAspectRatio, {0, (ImageDimensions[image][[1]]*netAspectRatio - ImageDimensions[image][[2]])/2}, {(ImageDimensions[image][[2]]*(1/netAspectRatio) - ImageDimensions[image][[1]])/2, 0} ], scale = If[ImageAspectRatio[image] < netAspectRatio, ImageDimensions[image][[1]]/netDims[[1]], ImageDimensions[image][[2]]/netDims[[2]] ] }, Map[Rectangle[Round[#[[1]]], Round[#[[2]]]] &, Transpose[Transpose[boxes, {2, 3, 1}]*scale - padding, {3, 1, 2}]] ] ]; detectionsDeconformer[image_Image, netDims_List, fitting_String ][ objects_] := Transpose[{deconformRectangles[objects[[All, 1]], image, netDims, fitting ], objects[[All, 2]]}]; filterClasses[ All ][ detections_ ] := detections; filterClasses[classes_][ detections_ ] := {#[[1]], Select[#[[2]], Function[det, MemberQ[classes, det[[1]]]]]} & /@ detections;$

In[6]:=

$Options[ netevaluate ] = { TargetDevice -> "CPU", AcceptanceThreshold -> .5, MaxOverlapFraction -> .45 }; netevaluate[img_Image, category_ : All, opts : OptionsPattern[]] := Module[{net}, net = NetModel["YOLO V3 Trained on Open Images Data"]; nonMaxSuppression[OptionValue[MaxOverlapFraction ]]@ detectionsDeconformer[img, {608, 608}, "Fit"]@ filterClasses[category ]@ netOutputDecoder[OptionValue[AcceptanceThreshold]]@ (net[#, TargetDevice -> OptionValue[TargetDevice]] &)@ imageConformer[{608, 608}, "Fit"]@img ];$

Basic usage

Obtain the detected bounding boxes with their corresponding classes and confidences for a given image:

In[7]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/e1883d1e-28d2-441e-8294-48c4f32efdf4"]

In[8]:=

Out[8]=

Inspect which classes are detected:

In[9]:=

Out[9]=

Visualize the detection:

In[10]:=

$displayObject[object_] := {White, object[[1]], Text[Style[object[[2]], Black, 8, Background -> GrayLevel[1, .6]], {35, 35} + object[[1, 1]], Background -> GrayLevel[1, .6]]}$

In[11]:=

$HighlightImage[testImage, displayObject /@ ({#[[1]], StringRiffle[#[[2, All, 1]], ",\n "]} & /@ detection)]$

Out[11]=

Network result

The network computes 22,743 bounding boxes, the probability of having an object in each box and the conditioned probability that the object is of any given class:

In[12]:=

Out[12]=

In[13]:=

Out[13]=

Visualize all the boxes predicted by the net scaled by their “objectness” measures:

In[14]:=

In[15]:=

Graphics[
MapThread[{EdgeForm[Opacity[#1 + .01]], #2} &, {res["Objectness"], rectangles}],
BaseStyle -> {FaceForm[], EdgeForm[{Thin, Black}]}
]

Out[15]=

Visualize all the boxes scaled by the probability that they contain an animal:

In[16]:=

Out[16]=

In[17]:=

Graphics[
MapThread[{EdgeForm[Opacity[#1 + .01]], #2} &, {res[
"Objectness"] Extract[res["ClassProb"], {All, idx}], rectangles}],
BaseStyle -> {FaceForm[], EdgeForm[{Thin, Black}]}
]

Out[17]=

Superimpose the animal prediction on top of the scaled input received by the net:

In[18]:=

HighlightImage[
Image[NetExtract[NetModel["YOLO V3 Trained on Open Images Data"], "Input"][imageConformer[{608, 608}, "Fit"]@testImage], Interleaving -> False],
MapThread[{EdgeForm[{Thickness[#1/100], Opacity[#1 + .01]}], #2} &, {res["Objectness"] Extract[
res["ClassProb"], {All, idx}], rectangles}],
BaseStyle -> {FaceForm[], EdgeForm[{Thin, Red}]}
]

Out[18]=

Class filtering

Obtain a test image:

In[19]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/4d54f050-34ea-4b75-b054-27383eaabbf9"]

Obtain bounding boxes for the specified classes only (“Vehicle registration plate” and “Window”):

In[20]:=

Out[20]=

Visualize the detection:

In[21]:=

$displayObject[object_] := {White, object[[1]], Text[Style[object[[2]], Black, 8, Background -> GrayLevel[1, .6]], {35, 35} + object[[1, 1]], Background -> GrayLevel[1, .6]]}$

In[22]:=

$HighlightImage[testImage, displayObject /@ ({#[[1]], StringRiffle[#[[2, All, 1]], ",\n "]} & /@ detection)]$

Out[22]=

Net information

Inspect the number of parameters of all arrays in the net:

In[23]:=

$NetInformation[ NetModel["YOLO V3 Trained on Open Images Data"], \ "ArraysElementCounts"]$

Out[23]=

Obtain the total number of parameters:

In[24]:=

$NetInformation[ NetModel["YOLO V3 Trained on Open Images Data"], \ "ArraysTotalElementCount"]$

Out[24]=

Obtain the layer type counts:

In[25]:=

Out[25]=

Display the summary graphic:

In[26]:=

Out[26]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[27]:=

Out[27]=

Export also creates a net.params file containing parameters:

In[28]:=

Out[28]=

Get the size of the parameter file:

In[29]:=

Out[29]=

The size is similar to the byte count of the resource object:

In[30]:=

Out[30]=

Construction Notebook

Download Construction Notebook

Open in Wolfram Cloud

Requirements

Wolfram Language 12.0 (April 2019) or above

Resource History

Date Created: 30 September 2019
Latest Update: 2 March 2020

Reference

J. Redmon, A. Farhadi, "YOLOv3: An Incremental Improvement," arXiv:1804.02767 (2018)
Available from: https://pjreddie.com/darknet/yolo
Rights: Unrestricted Use