YOLO V3 Trained on Open Images Data

Contributed by: Julian W. Francis

Detect and localize objects in an image

YOLO (You Only Look Once) Version 3 is an object detection model published in April 2018. It is a single-stage architecture that goes straight from image pixels to bounding box coordinates and class probabilities. Compared to previous versions of YOLO, it performs bounding box regression and classification at three different scales and uses three anchor boxes instead of two. Additionally, residual blocks and skip connections are added and L2 losses for object confidence and class predictions are now replaced by cross-entropy error terms.

Number of layers: 349 | Parameter count: 65,252,682 | Trained size: 263 MB |

Training Set Information

Performance

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["YOLO V3 Trained on Open Images Data"]
Out[1]=

Label list

Define the label list for this model. Integers in the model’s output correspond to elements in the label list:

In[2]:=
labels = {
  "Tortoise", "Container", "Magpie", "Sea turtle", "Football", "Ambulance", "Ladder", "Toothbrush", "Syringe", "Sink", "Toy", "Organ", "Cassette deck", "Apple", "Human eye", "Cosmetics", "Paddle", "Snowman", "Beer", "Chopsticks", "Human beard", "Bird", "Parking meter", "Traffic light", "Croissant", "Cucumber", "Radish", "Towel", "Doll", "Skull", "Washing machine", "Glove", "Tick", "Belt", "Sunglasses", "Banjo", "Cart", "Ball", "Backpack", "Bicycle", "Home appliance", "Centipede", "Boat", "Surfboard", "Boot", "Headphones", "Hot dog", "Shorts", "Fast food", "Bus", "Boy", "Screwdriver", "Bicycle wheel", "Barge", "Laptop", "Miniskirt", "Drill", "Dress", "Bear", "Waffle", "Pancake", "Brown bear", "Woodpecker", "Blue jay", "Pretzel", "Bagel", "Tower", "Teapot", "Person", "Bow and arrow", "Swimwear", "Beehive", "Brassiere", "Bee", "Bat", "Starfish", "Popcorn", "Burrito", "Chainsaw", "Balloon", "Wrench", "Tent", "Vehicle registration plate", "Lantern", "Toaster", "Flashlight", "Billboard", "Tiara", "Limousine", "Necklace", "Carnivore", "Scissors", "Stairs", "Computer keyboard", "Printer", "Traffic sign", "Chair", "Shirt", "Poster", "Cheese", "Sock", "Fire hydrant", "Land vehicle", "Earrings", "Tie", "Watercraft", "Cabinetry", "Suitcase", "Muffin", "Bidet", "Snack", "Snowmobile", "Clock", "Medical equipment", "Cattle", "Cello", "Jet ski", "Camel", "Coat", "Suit", "Desk", "Cat", "Bronze sculpture", "Juice", "Gondola", "Beetle", "Cannon", "Computer mouse", "Cookie",
    "Office building", "Fountain", "Coin", "Calculator", "Cocktail", "Computer monitor", "Box", "Stapler", "Christmas tree", "Cowboy hat", "Hiking equipment", "Studio couch", "Drum", "Dessert", "Wine rack", "Drink", "Zucchini", "Ladle", "Human mouth", "Dairy", "Dice", "Oven", "Dinosaur", "Ratchet", "Couch", "Cricket ball", "Winter melon", "Spatula", "Whiteboard", "Pencil sharpener", "Door", "Hat", "Shower", "Eraser", "Fedora", "Guacamole", "Dagger", "Scarf", "Dolphin", "Sombrero", "Tin can", "Mug", "Tap", "Harbor seal", "Stretcher", "Can opener", "Goggles", "Human body", "Roller skates", "Coffee cup", "Cutting board", "Blender", "Plumbing fixture", "Stop sign", "Office supplies", "Volleyball", "Vase", "Slow cooker", "Wardrobe", "Coffee", "Whisk",
    "Paper towel", "Personal care", "Food", "Sun hat", "Tree house", "Flying disc", "Skirt", "Gas stove", "Salt and pepper shakers", "Mechanical fan", "Face powder", "Fax", "Fruit", "French fries", "Nightstand", "Barrel", "Kite", "Tart", "Treadmill", "Fox", "Flag",
    "Horn", "Window blind", "Human foot", "Golf cart", "Jacket", "Egg", "Street light", "Guitar", "Pillow", "Human leg", "Isopod", "Grape", "Human ear", "Power plugs and sockets", "Panda", "Giraffe", "Woman", "Door handle", "Rhinoceros", "Bathtub", "Goldfish", "Houseplant", "Goat", "Baseball bat", "Baseball glove",
    "Mixing bowl", "Marine invertebrates", "Kitchen utensil", "Light switch", "House", "Horse", "Stationary bicycle", "Hammer", "Ceiling fan", "Sofa bed", "Adhesive tape", "Harp", "Sandal", "Bicycle helmet", "Saucer", "Harpsichord", "Human hair", "Heater", "Harmonica", "Hamster", "Curtain", "Bed", "Kettle", "Fireplace", "Scale", "Drinking straw", "Insect", "Hair dryer", "Kitchenware", "Indoor rower", "Invertebrate", "Food processor", "Bookcase", "Refrigerator", "Wood-burning stove", "Punching bag", "Common fig",
    "Cocktail shaker", "Jaguar", "Golf ball", "Fashion accessory", "Alarm clock", "Filing cabinet", "Artichoke", "Table", "Tableware",
    "Kangaroo", "Koala", "Knife", "Bottle", "Bottle opener", "Lynx", "Lavender", "Lighthouse", "Dumbbell", "Human head", "Bowl", "Humidifier", "Porch", "Lizard", "Billiard table", "Mammal", "Mouse", "Motorcycle", "Musical instrument", "Swim cap", "Frying pan", "Snowplow", "Bathroom cabinet", "Missile", "Bust", "Man", "Waffle iron", "Milk", "Ring binder", "Plate", "Mobile phone", "Baked goods", "Mushroom", "Crutch", "Pitcher", "Mirror", "Lifejacket", "Table tennis racket", "Pencil case", "Musical keyboard", "Scoreboard", "Briefcase", "Kitchen knife", "Nail", "Tennis ball", "Plastic bag", "Oboe", "Chest of drawers", "Ostrich", "Piano", "Girl", "Plant", "Potato", "Hair spray", "Sports equipment", "Pasta", "Penguin", "Pumpkin", "Pear", "Infant bed", "Polar bear", "Mixer", "Cupboard", "Jacuzzi", "Pizza", "Digital clock", "Pig", "Reptile", "Rifle", "Lipstick", "Skateboard", "Raven", "High heels", "Red panda", "Rose", "Rabbit",
    "Sculpture", "Saxophone", "Shotgun", "Seafood", "Submarine sandwich", "Snowboard", "Sword", "Picture frame", "Sushi", "Loveseat", "Ski", "Squirrel", "Tripod", "Stethoscope", "Submarine", "Scorpion", "Segway", "Training bench", "Snake", "Coffee table", "Skyscraper", "Sheep", "Television", "Trombone", "Tea", "Tank", "Taco", "Telephone", "Torch", "Tiger", "Strawberry",
    "Trumpet", "Tree", "Tomato", "Train", "Tool", "Picnic basket", "Cooking spray", "Trousers", "Bowling equipment", "Football helmet", "Truck", "Measuring cup", "Coffeemaker", "Violin", "Vehicle", "Handbag", "Paper cutter", "Wine", "Weapon", "Wheel", "Worm", "Wok", "Whale", "Zebra", "Auto part", "Jug", "Pizza cutter", "Cream", "Monkey", "Lion", "Bread", "Platter", "Chicken", "Eagle", "Helicopter", "Owl", "Duck", "Turtle", "Hippopotamus", "Crocodile", "Toilet", "Toilet paper", "Squid", "Clothing", "Footwear", "Lemon", "Spider", "Deer", "Frog", "Banana", "Rocket", "Wine glass", "Countertop", "Tablet computer", "Waste container", "Swimming pool", "Dog", "Book", "Elephant", "Shark", "Candle", "Leopard", "Axe", "Hand dryer", "Soap dispenser", "Porcupine", "Flower", "Canary", "Cheetah", "Palm tree", "Hamburger", "Maple", "Building", "Fish", "Lobster", "Asparagus", "Furniture", "Hedgehog", "Airplane", "Spoon", "Otter",
    "Bull", "Oyster", "Horizontal bar", "Convenience store", "Bomb", "Bench", "Ice cream", "Caterpillar", "Butterfly", "Parachute", "Orange", "Antelope", "Beaker", "Moths and butterflies", "Window", "Closet", "Castle", "Jellyfish", "Goose", "Mule", "Swan", "Peach", "Coconut", "Seat belt", "Raccoon", "Chisel", "Fork", "Lamp", "Camera", "Squash", "Racket", "Human face", "Human arm", "Vegetable", "Diaper", "Unicycle", "Falcon", "Chime", "Snail", "Shellfish", "Cabbage", "Carrot", "Mango", "Jeans", "Flowerpot", "Pineapple", "Drawer", "Stool", "Envelope", "Cake", "Dragonfly", "Sunflower", "Microwave oven", "Honeycomb", "Marine mammal", "Sea lion", "Ladybug", "Shelf", "Watch", "Candy", "Salad", "Parrot", "Handgun", "Sparrow", "Van", "Grinder", "Spice rack", "Light bulb", "Corded phone", "Sports uniform", "Tennis racket", "Wall clock", "Serving tray", "Kitchen & dining room table", "Dog bed", "Cake stand", "Cat furniture", "Bathroom accessory", "Facial tissue holder", "Pressure cooker", "Kitchen appliance", "Tire", "Ruler", "Luggage and bags", "Microphone", "Broccoli", "Umbrella", "Pastry", "Grapefruit", "Band-aid", "Animal", "Bell pepper", "Turkey", "Lily", "Pomegranate", "Doughnut", "Glasses", "Human nose", "Pen", "Ant", "Car", "Aircraft", "Human hand", "Skunk", "Teddy bear", "Watermelon", "Cantaloupe", "Dishwasher", "Flute", "Balance beam", "Sandwich", "Shrimp", "Sewing machine", "Binoculars", "Rays and skates", "Ipod", "Accordion", "Willow", "Crab", "Crown", "Seahorse", "Perfume", "Alpaca", "Taxi", "Canoe", "Remote control", "Wheelchair", "Rugby ball", "Armadillo", "Maracas", "Helmet"};

Evaluation function

Write an evaluation function to scale the result to the input image size and suppress the least probable detections:

In[3]:=
ClearAll[IoU]
IoU := IoU =
   With[
    {
     IoUCompiled = Compile[
       {
        {box1, _Real, 2}, {box2, _Real, 2}
        },
       Module[
        {area1, area2, x1, y1, x2, y2, w, h, int},
        area1 = (box1[[2, 1]] - box1[[1, 1]])*(box1[[2, 2]] - box1[[1, 2]]);
        area2 = (box2[[2, 1]] - box2[[1, 1]])*(box2[[2, 2]] - box2[[1, 2]]);
        x1 = Max[box1[[1, 1]], box2[[1, 1]]];
        y1 = Max[box1[[1, 2]], box2[[1, 2]]];
        x2 = Min[box1[[2, 1]], box2[[2, 1]]];
        y2 = Min[box1[[2, 2]], box2[[2, 2]]];
        w = Max[0., x2 - x1];
        h = Max[0., y2 - y1];
        int = w*h;
        int/(area1 + area2 - int)
        ],
       RuntimeAttributes -> {Listable},
       Parallelization -> True,
       RuntimeOptions -> "Speed"
       ]
     },
    IoUCompiled @@ Replace[{##}, Rectangle -> List, Infinity, Heads -> True] &
    ];
In[4]:=
nonMaxSuppression[nmsThreshold_ ][dets_] :=
  DeleteCases[ Function[detection,
     {
      detection[[1]],
       Function[
        overlapBoxLabels,
        Select[
         detection[[2]], #[[2]] > Max@Extract[
             overlapBoxLabels[[All, All, 2]],
             Position[overlapBoxLabels[[All, All, 1]], #[[1]]] ] &
         ]
        ]
       [Select[
         dets, (IoU[detection[[1]], #[[1]]] > nmsThreshold && ! (detection[[1]] === #[[1]])) &][[All, 2]]]
      }
     ] /@ dets,
   {_, {}}
   ];
In[5]:=
netOutputDecoder[threshold_ : .5][output_] := Module[{probs = output["Objectness"]*output["ClassProb"], detectionBoxes},
   detectionBoxes = Union@Flatten@
      SparseArray[UnitStep[probs - threshold]]["NonzeroPositions"][[
       All, 1]];
   Map[
    Function[{detectionBox},
     {
      Rectangle @@ output["Boxes"][[detectionBox]],
      Map[
       {labels[[#]], probs[[detectionBox, #]]} &,
       Flatten@Position[probs[[detectionBox]], x_ /; x > threshold]
       ]
      }
     ],
    detectionBoxes
    ]
   ];
imageConformer[dims_, fitting_][image_] := First[ConformImages[{image}, dims, fitting, Padding -> 0.5]];
deconformRectangles[{}, _, _, _] := {};
deconformRectangles[rboxes_List, image_Image, netDims_List, "Fit"] := With[{netAspectRatio = netDims[[2]]/netDims[[1]]},
   With[
    {
     boxes = Map[{#[[1]], #[[2]]} &, rboxes],
     padding = If[ImageAspectRatio[image] < netAspectRatio,
       {0, (ImageDimensions[image][[1]]*netAspectRatio - ImageDimensions[image][[2]])/2},
       {(ImageDimensions[image][[2]]*(1/netAspectRatio) - ImageDimensions[image][[1]])/2, 0}
       ],
     scale = If[ImageAspectRatio[image] < netAspectRatio,
       ImageDimensions[image][[1]]/netDims[[1]],
       ImageDimensions[image][[2]]/netDims[[2]]
       ]
     },
    Map[Rectangle[Round[#[[1]]], Round[#[[2]]]] &, Transpose[Transpose[boxes, {2, 3, 1}]*scale - padding, {3, 1, 2}]]
    ]
   ];
detectionsDeconformer[image_Image, netDims_List, fitting_String ][
   objects_] := Transpose[{deconformRectangles[objects[[All, 1]], image, netDims, fitting ], objects[[All, 2]]}];
filterClasses[ All ][ detections_ ] := detections;
filterClasses[classes_][
   detections_ ] := {#[[1]], Select[#[[2]], Function[det, MemberQ[classes, det[[1]]]]]} & /@ detections;
In[6]:=
Options[ netevaluate ] = {
      TargetDevice -> "CPU",
      AcceptanceThreshold -> .5,
      MaxOverlapFraction -> .45
   };
netevaluate[img_Image, category_ : All, opts : OptionsPattern[]] := Module[{net},
   net = NetModel["YOLO V3 Trained on Open Images Data"];
   nonMaxSuppression[OptionValue[MaxOverlapFraction ]]@
    detectionsDeconformer[img, {608, 608}, "Fit"]@
     filterClasses[category ]@
      netOutputDecoder[OptionValue[AcceptanceThreshold]]@
       (net[#, TargetDevice -> OptionValue[TargetDevice]] &)@
        imageConformer[{608, 608}, "Fit"]@img
   ];

Basic usage

Obtain the detected bounding boxes with their corresponding classes and confidences for a given image:

In[7]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/e1883d1e-28d2-441e-8294-48c4f32efdf4"]
In[8]:=
detection = netevaluate[testImage, All, AcceptanceThreshold -> .1]
Out[8]=

Inspect which classes are detected:

In[9]:=
classes = DeleteDuplicates@Flatten@detection[[All, 2, All, 1]]
Out[9]=

Visualize the detection:

In[10]:=
displayObject[object_] := {White, object[[1]], Text[Style[object[[2]], Black, 8, Background -> GrayLevel[1, .6]], {35, 35} + object[[1, 1]], Background -> GrayLevel[1, .6]]}
In[11]:=
HighlightImage[testImage, displayObject /@ ({#[[1]], StringRiffle[#[[2, All, 1]], ",\n "]} & /@
     detection)]
Out[11]=

Network result

The network computes 22,743 bounding boxes, the probability of having an object in each box and the conditioned probability that the object is of any given class:

In[12]:=
res = NetModel["YOLO V3 Trained on Open Images Data"][testImage]
Out[12]=
In[13]:=
Dimensions /@ res
Out[13]=

Visualize all the boxes predicted by the net scaled by their “objectness” measures:

In[14]:=
rectangles = Rectangle @@@ res["Boxes"];
In[15]:=
Graphics[
 MapThread[{EdgeForm[Opacity[#1 + .01]], #2} &, {res["Objectness"], rectangles}],
 BaseStyle -> {FaceForm[], EdgeForm[{Thin, Black}]}
 ]
Out[15]=

Visualize all the boxes scaled by the probability that they contain an animal:

In[16]:=
idx = Position[labels, "Animal"][[1, 1]]
Out[16]=
In[17]:=
Graphics[
 MapThread[{EdgeForm[Opacity[#1 + .01]], #2} &, {res[
     "Objectness"] Extract[res["ClassProb"], {All, idx}], rectangles}],
 BaseStyle -> {FaceForm[], EdgeForm[{Thin, Black}]}
 ]
Out[17]=

Superimpose the animal prediction on top of the scaled input received by the net:

In[18]:=
HighlightImage[
 Image[NetExtract[NetModel["YOLO V3 Trained on Open Images Data"], "Input"][imageConformer[{608, 608}, "Fit"]@testImage], Interleaving -> False],
 MapThread[{EdgeForm[{Thickness[#1/100], Opacity[#1 + .01]}], #2} &, {res["Objectness"] Extract[
     res["ClassProb"], {All, idx}], rectangles}],
 BaseStyle -> {FaceForm[], EdgeForm[{Thin, Red}]}
 ]
Out[18]=

Class filtering

Obtain a test image:

In[19]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/4d54f050-34ea-4b75-b054-27383eaabbf9"]

Obtain bounding boxes for the specified classes only (“Vehicle registration plate” and “Window”):

In[20]:=
detection = netevaluate[testImage, {"Vehicle registration plate", "Window"}, AcceptanceThreshold -> .01]
Out[20]=

Visualize the detection:

In[21]:=
displayObject[object_] := {White, object[[1]], Text[Style[object[[2]], Black, 8, Background -> GrayLevel[1, .6]], {35, 35} + object[[1, 1]], Background -> GrayLevel[1, .6]]}
In[22]:=
HighlightImage[testImage, displayObject /@ ({#[[1]], StringRiffle[#[[2, All, 1]], ",\n "]} & /@
     detection)]
Out[22]=

Net information

Inspect the number of parameters of all arrays in the net:

In[23]:=
NetInformation[
 NetModel["YOLO V3 Trained on Open Images Data"], \
"ArraysElementCounts"]
Out[23]=

Obtain the total number of parameters:

In[24]:=
NetInformation[
 NetModel["YOLO V3 Trained on Open Images Data"], \
"ArraysTotalElementCount"]
Out[24]=

Obtain the layer type counts:

In[25]:=
NetInformation[
 NetModel["YOLO V3 Trained on Open Images Data"], "LayerTypeCounts"]
Out[25]=

Display the summary graphic:

In[26]:=
NetInformation[
 NetModel["YOLO V3 Trained on Open Images Data"], "SummaryGraphic"]
Out[26]=

Export to MXNet

Export the net into a format that can be opened in MXNet:

In[27]:=
jsonPath = Export[FileNameJoin[{$TemporaryDirectory, "net.json"}], NetModel["YOLO V3 Trained on Open Images Data"], "MXNet"]
Out[27]=

Export also creates a net.params file containing parameters:

In[28]:=
paramPath = FileNameJoin[{DirectoryName[jsonPath], "net.params"}]
Out[28]=

Get the size of the parameter file:

In[29]:=
FileByteCount[paramPath]
Out[29]=

The size is similar to the byte count of the resource object:

In[30]:=
ResourceObject["YOLO V3 Trained on Open Images Data"]["ByteCount"]
Out[30]=

Requirements

Wolfram Language 12.0 (April 2019) or above

Resource History

Reference