Wolfram Research

Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA Swiss Alps and PASCAL VOC Data

Navigate a drone in a forest environment

Released in 2017, these two models constitute a system for autonomous path navigation in unstructured, outdoor environments such as forests. Specifically, this model is trained for steering in a forest environment. The system consists of two main submodules: a navigation net (TrailNet DNN) and an obstacle detection net. The navigation net is a two-headed classifier used to estimate rotation directions and lateral translations given an input image, with a total of three categories each. These output probabilities are later combined to predict a final rotation angle. It is based on ResNet-18, with batch normalization layers removed and ReLUs replaced with shifted ReLUs.

The obstacle detection net is an object detection model based on YOLO V1 with a few modifications, such as removal of batch normalizations and replacement of leaky ReLUs by ReLUs. In case a detected object occupies a large proportion of the space within the image frame, the vehicle is forced to stop.

Number of models: 2

Training Set Information

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \
Swiss Alps and PASCAL VOC Data"]
Out[1]=

NetModel parameters

This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:

In[2]:=
NetModel["Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \
Swiss Alps and PASCAL VOC Data", "ParametersInformation"]
Out[2]=

Pick a non-default net by specifying the parameter:

In[3]:=
NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}]
Out[3]=

Pick a non-default uninitialized net:

In[4]:=
NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
  "Task" -> "ObjectDetection"}, "UninitializedEvaluationNet"]
Out[4]=

Evaluation functions

Evaluation function for the navigation model

Define an evaluation function to calculate the turning angle in radians:

In[5]:=
getTurnAngle[img_, b1_, b2_] := 
  With[{netOut = 
     NetModel[
       "Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \
Swiss Alps and PASCAL VOC Data"][img]},
    b1*(netOut["view_orientation"][[3]] - 
       netOut["view_orientation"][[1]]) + 
    b2*(netOut["lateral_offset"][[3]] - 
       netOut["lateral_offset"][[1]])
   ];
getTurnAngle[img_] := getTurnAngle[img, 10, 10]

Evaluation function for object detection model

Write an evaluation function to scale the result to the input image size and suppress the least probable detections:

In[6]:=
nonMaxSuppression[overlapThreshold_][detection_] := 
  Module[{boxes, confidence},
   	Fold[
    		{list, new} \[Function] 
     If[NoneTrue[list[[All, 1]], 
       IoU[#, new[[1]]] > overlapThreshold &], Append[list, new], 
      list],
    		Sequence @@ TakeDrop[Reverse@SortBy[detection, Last], 1]
    	]
   ];
ClearAll[IoU]
IoU := IoU =  
   With[{c = Compile[
       	{{box1, _Real, 2}, {box2, _Real, 2}},
       	Module[{area1, area2, x1, y1, x2, y2, w, h, int},
        		
        area1 = (box1[[2, 1]] - box1[[1, 1]]) (box1[[2, 2]] - 
            box1[[1, 2]]);
        		
        area2 = (box2[[2, 1]] - box2[[1, 1]]) (box2[[2, 2]] - 
            box2[[1, 2]]);	
        		x1 = Max[box1[[1, 1]], box2[[1, 1]]];
        		y1 = Max[box1[[1, 2]], box2[[1, 2]]];
        		x2 = Min[box1[[2, 1]], box2[[2, 1]]];
        		y2 = Min[box1[[2, 2]], box2[[2, 2]]];	
        		w = Max[0., x2 - x1];
        		h = Max[0., y2 - y1];
        		int = w * h;
        		
        		int / (area1 + area2 - int)
        	],
       	RuntimeAttributes -> {Listable},
       	Parallelization -> True,
       	RuntimeOptions -> "Speed"
       ]},
    	c @@ Replace[{##}, Rectangle -> List, Infinity, Heads -> True] &
    ];
getBoxes[center_, boxCord_, imgW_, imgH_, scaling_ : 1] := Module[
   {bx, by, w, h},
   bx = (center[[1]] + boxCord[[1]])*imgW/7.;
   by = (center[[2]] + boxCord[[2]])*imgH/7.;
   w = boxCord[[3]]*scaling*imgW;
   h = boxCord[[4]]*scaling*imgH;
   
   {{bx - w/2, by - h/2}, {bx + w/2, by + h/2}}
   ];
detectObstacles[img_, detectionThreshold_ : .2, 
   overlapThreshold_ : .4, boxScaling_ : 0.7] :=
  Module[
   {w, h, scale, coords, confidence, classes, boxes, classProb, 
    bestClass, probable, finals, grid, yolo, imgW, imgH},
   
   {coords, classes, confidence} = 
    Values@NetModel[{"Micro Aerial Vehicle Trail Navigation Nets \
Trained on IDSIA Swiss Alps and PASCAL VOC Data", 
        "Task" -> "ObjectDetection"}]@img;
   
   (* transform coordinates into rectangular boxes *)
   
   grid = Flatten[Table[{i, j}, {j, 0, 6}, {i, 0, 6}], 1];
   {imgW, imgH} = ImageDimensions@img;
   boxes = 
    Table[ getBoxes[grid[[i]], coords[[i, b]], imgW, imgH, 
      boxScaling], {i, 1, 49}, {b, 1, 2} ]; 
   boxes = ArrayReshape[boxes, {98, 2, 2}];
   
   (* each class probability is rescaled with the box objectivness *)
   classProb = 
    Table[classes[[i, c]]*confidence[[i, b]], {i, 1, 49}, {b, 1, 
      2}, {c, 1, 20}]; 
   classProb = ArrayReshape[classProb, {98, 20}];
   
   (* filter by probability*)
   (* 
   very small probability are thresholded *)
   
   probable = 
    Position[Max /@ classProb , p_ /; p - 10^-2 > detectionThreshold];	
   If[Length[probable] == 0, Return[{}]];
   
   (* gather the boxes of the same class and perform non-
   max suppression *)
   bestClass = Last@*Ordering /@ classProb;
   finals = Join @@ Values @ GroupBy[
       MapThread[
        {#1, #2, #3[[#2]]} &,
        {Extract[boxes, probable], Extract[bestClass, probable], 
         Extract[classProb, probable]}
        ],
       #[[2]] &, nonMaxSuppression[overlapThreshold]
       ];
   {Rectangle[Sequence @@ #[[1]]], #[[2]], #[[3]]} & /@ finals	
     ];

Label list for the object detection model

Define the label list for this model. Integers in the model's output correspond to elements in the label list:

In[7]:=
labels = {"aeroplane", "bicycle", "bird", "boat", "bottle", "bus", 
   "car", "cat", "chair", "cow", "diningtable", "dog", "horse", 
   "motorbike", "person", "pottedplant", "sheep", "sofa", "train", 
   "tvmonitor"};

Basic usage: navigation model

Calculate the turn angle in radians given a test image:

In[8]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/6f7b9ae3-11d8-4f92-8a6c-7afac8376c12"]
Out[8]=

Basic usage: object detection model

Obtain the detected bounding boxes with their corresponding classes and confidences for a given image:

In[9]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/5ae4031f-b1f2-464f-ac59-f04d23f90722"]
In[10]:=
detection = detectObstacles[testImage]
Out[10]=

Inspect which classes are detected:

In[11]:=
classes = DeleteDuplicates@detection[[All, 2]]
Out[11]=
In[12]:=
labels[[classes]]
Out[12]=

Visualize the detection:

In[13]:=
HighlightImage[testImage, 
 MapThread[{White, 
    Inset[Style[labels[[#2]], Black, FontSize -> Scaled[1/25], 
      Background -> GrayLevel[1, .6]], Last[#1], {Right, Top}], #1} &,
   Transpose@detection]]
Out[13]=

Detection results

Define an image:

In[14]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/7b0bf1f1-baec-4c4c-ab0a-4a6f01459bb9"]

The network computes 98 bounding boxes and the probability that the objects in each box are of any given class:

In[15]:=
res = NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained \
on IDSIA Swiss Alps and PASCAL VOC Data", 
     "Task" -> "ObjectDetection"}][testImage];
In[16]:=
Dimensions /@ res
Out[16]=

Visualize all the boxes predicted by the net scaled by their "Confidence" measures:

In[17]:=
rectangles = res["Boxes"];
rectangles = 
  ArrayReshape[ArrayReshape[rectangles, {98, 4}], {98, 2, 2}];
rectangles = Rectangle @@@ rectangles;
objectness = ArrayFlatten[res["Confidence"], 1];
In[18]:=
Graphics[
 MapThread[{EdgeForm[Opacity[#1 + 0.12]], #2} &, {objectness, 
   rectangles}],
 BaseStyle -> {FaceForm[], EdgeForm[{Thin, Black}]}
 ]
Out[18]=

Net information

Inspect the number of parameters of all arrays in the net:

In[19]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "ObjectDetection"}], "ArraysElementCounts"]
Out[19]=
In[20]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "Navigation"}], "ArraysElementCounts"]
Out[20]=

Obtain the total number of parameters:

In[21]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "ObjectDetection"}], "ArraysTotalElementCount"]
Out[21]=
In[22]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "Navigation"}], "ArraysTotalElementCount"]
Out[22]=

Obtain the layer type counts:

In[23]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "ObjectDetection"}], "LayerTypeCounts"]
Out[23]=
In[24]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "Navigation"}], "LayerTypeCounts"]
Out[24]=

Display the summary graphic:

In[25]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "ObjectDetection"}], "SummaryGraphic"]
Out[25]=
In[26]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", 
   "Task" -> "Navigation"}], "SummaryGraphic"]
Out[26]=

Export to MXNet

Export the nets into a format that can be opened in MXNet:

In[27]:=
jsonPaths = Map[
  Export[
    FileNameJoin[{$TemporaryDirectory, # <> ".json"}],
    NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> #}], "MXNet"
    ] &,
  {"ObjectDetection", "Navigation"}
  ]
Out[27]=

Export also creates a net.params file containing parameters:

In[28]:=
paramPaths = 
 Map[FileNameJoin[{DirectoryName[First@jsonPaths], # <> 
      ".params"}] &, {"ObjectDetection", "Navigation"}]
Out[28]=

Get the size of the parameter files:

In[29]:=
AssociationThread[{"ObjectDetection", "Navigation"}, 
 Map[FileByteCount, paramPaths]]
Out[29]=

Resource History

Reference

  • N. Smolyanskiy, A. Kamenev, J. Smith, S. Birchfield, "Toward Low-Flying Autonomous MAV Trail Navigation Using Deep Neural Networks for Environmental Awareness," arXiv:1705.02550 (2017)
  • (available from https://github.com/NVIDIA-AI-IOT/redtail)
  • Rights: Model License