Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA Swiss Alps and PASCAL VOC Data

Navigate a drone in a forest environment

Released in 2017, these two models constitute a system for autonomous path navigation in unstructured, outdoor environments such as forests. Specifically, this model is trained for steering in a forest environment. The system consists of two main submodules: a navigation net (TrailNet DNN) and an obstacle detection net. The navigation net is a two-headed classifier used to estimate rotation directions and lateral translations given an input image, with a total of three categories each. These output probabilities are later combined to predict a final rotation angle. It is based on ResNet-18, with batch normalization layers removed and ReLUs replaced with shifted ReLUs.

The obstacle detection net is an object detection model based on YOLO V1 with a few modifications, such as removal of batch normalizations and replacement of leaky ReLUs by ReLUs. In case a detected object occupies a large proportion of the space within the image frame, the vehicle is forced to stop.

Number of models: 2

Training Set Information

Examples

Resource retrieval

Get the pre-trained net:

In[1]:=
NetModel["Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \
Swiss Alps and PASCAL VOC Data"]
Out[1]=

NetModel parameters

This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:

In[2]:=
NetModel["Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \
Swiss Alps and PASCAL VOC Data", "ParametersInformation"]
Out[2]=

Pick a non-default net by specifying the parameter:

In[3]:=
NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}]
Out[3]=

Pick a non-default uninitialized net:

In[4]:=
NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}, "UninitializedEvaluationNet"]
Out[4]=

Evaluation functions

Evaluation function for the navigation model

Define an evaluation function to calculate the turning angle in radians:

In[5]:=
getTurnAngle[img_, b1_, b2_] := With[{netOut = NetModel[
       "Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \
Swiss Alps and PASCAL VOC Data"][img]},
    b1*(netOut["view_orientation"][[3]] - netOut["view_orientation"][[1]]) + b2*(netOut["lateral_offset"][[3]] - netOut["lateral_offset"][[1]])
   ];
getTurnAngle[img_] := getTurnAngle[img, 10, 10]

Evaluation function for object detection model

Write an evaluation function to scale the result to the input image size and suppress the least probable detections:

In[6]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/d6421455-104c-49a6-b76e-ee492bed6245"]

Label list for the object detection model

Define the label list for this model. Integers in the model's output correspond to elements in the label list:

In[7]:=
labels = {"aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"};

Basic usage: navigation model

Calculate the turn angle in radians given a test image:

In[8]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/4056bc3f-7df8-4c6e-8418-a0a200903b88"]
Out[8]=

Basic usage: object detection model

Obtain the detected bounding boxes with their corresponding classes and confidences for a given image:

In[9]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/d566d376-4a03-4f1d-be00-285776ff2c9f"]
In[10]:=
detection = detectObstacles[testImage]
Out[10]=

Inspect which classes are detected:

In[11]:=
classes = DeleteDuplicates@detection[[All, 2]]
Out[11]=
In[12]:=
labels[[classes]]
Out[12]=

Visualize the detection:

In[13]:=
HighlightImage[testImage, MapThread[{White, Inset[Style[labels[[#2]], Black, FontSize -> Scaled[1/25], Background -> GrayLevel[1, .6]], Last[#1], {Right, Top}], #1} &,
   Transpose@detection]]
Out[13]=

Detection results

Define an image:

In[14]:=
(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/91115a35-2f65-417e-952e-ced73a0d5570"]

The network computes 98 bounding boxes and the probability that the objects in each box are of any given class:

In[15]:=
res = NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained \
on IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}][testImage];
In[16]:=
Dimensions /@ res
Out[16]=

Visualize all the boxes predicted by the net scaled by their "Confidence" measures:

In[17]:=
rectangles = res["Boxes"];
rectangles = ArrayReshape[ArrayReshape[rectangles, {98, 4}], {98, 2, 2}];
rectangles = Rectangle @@@ rectangles;
objectness = ArrayFlatten[res["Confidence"], 1];
In[18]:=
Graphics[
 MapThread[{EdgeForm[Opacity[#1 + 0.12]], #2} &, {objectness, rectangles}],
 BaseStyle -> {FaceForm[], EdgeForm[{Thin, Black}]}
 ]
Out[18]=

Net information

Inspect the number of parameters of all arrays in the net:

In[19]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "ArraysElementCounts"]
Out[19]=
In[20]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "ArraysElementCounts"]
Out[20]=

Obtain the total number of parameters:

In[21]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "ArraysTotalElementCount"]
Out[21]=
In[22]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "ArraysTotalElementCount"]
Out[22]=

Obtain the layer type counts:

In[23]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "LayerTypeCounts"]
Out[23]=
In[24]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "LayerTypeCounts"]
Out[24]=

Display the summary graphic:

In[25]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "SummaryGraphic"]
Out[25]=
In[26]:=
Information[
 NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "SummaryGraphic"]
Out[26]=

Export to MXNet

Export the nets into a format that can be opened in MXNet:

In[27]:=
jsonPaths = Map[
  Export[
    FileNameJoin[{$TemporaryDirectory, # <> ".json"}],
    NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \
IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> #}], "MXNet"
    ] &,
  {"ObjectDetection", "Navigation"}
  ]
Out[27]=

Export also creates a net.params file containing parameters:

In[28]:=
paramPaths = Map[FileNameJoin[{DirectoryName[First@jsonPaths], # <> ".params"}] &, {"ObjectDetection", "Navigation"}]
Out[28]=

Get the size of the parameter files:

In[29]:=
AssociationThread[{"ObjectDetection", "Navigation"}, Map[FileByteCount, paramPaths]]
Out[29]=

Requirements

Wolfram Language 12.1 (March 2020) or above

Resource History

Reference

  • N. Smolyanskiy, A. Kamenev, J. Smith, S. Birchfield, "Toward Low-Flying Autonomous MAV Trail Navigation Using Deep Neural Networks for Environmental Awareness," arXiv:1705.02550 (2017)
  • Available from: https://github.com/NVIDIA-AI-IOT/redtail
  • Rights: Model License