Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA Swiss Alps and PASCAL VOC Data

Navigate a drone in a forest environment

Released in 2017, these two models constitute a system for autonomous path navigation in unstructured, outdoor environments such as forests. Specifically, this model is trained for steering in a forest environment. The system consists of two main submodules: a navigation net (TrailNet DNN) and an obstacle detection net. The navigation net is a two-headed classifier used to estimate rotation directions and lateral translations given an input image, with a total of three categories each. These output probabilities are later combined to predict a final rotation angle. It is based on ResNet-18, with batch normalization layers removed and ReLUs replaced with shifted ReLUs.

The obstacle detection net is an object detection model based on YOLO V1 with a few modifications, such as removal of batch normalizations and replacement of leaky ReLUs by ReLUs. In case a detected object occupies a large proportion of the space within the image frame, the vehicle is forced to stop.

Number of models: 2

Training Set Information

The TrailNet DNN was first trained on the IDSIA dataset to learn the rotation directions and then on a private unreleased dataset collected in the Pacific Northwest to predict translation offsets. The IDSIA dataset is a dataset for the visual perception of forest trails consisting of 17,119 training images and 7,355 test images, with three classes of objects. The private dataset used to learn the translation offsets consists of footages of hikes recorded by three cameras capturing a diversity of cross-seasonal landscape scenes. The object detection net was retrained on the PASCAL VOC datasets, presumably on a combination of PASCAL VOC 2007 and PASCAL VOC 2012.

Examples

Download Example Notebook

Open in Wolfram Cloud

Resource retrieval

Get the pre-trained net:

In[1]:=

$NetModel["Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \ Swiss Alps and PASCAL VOC Data"]$

Out[1]=

NetModel parameters

This model consists of a family of individual nets, each identified by a specific parameter combination. Inspect the available parameters:

In[2]:=

$NetModel["Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \ Swiss Alps and PASCAL VOC Data", "ParametersInformation"]$

Out[2]=

Pick a non-default net by specifying the parameter:

In[3]:=

$NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}]$

Out[3]=

Pick a non-default uninitialized net:

In[4]:=

$NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}, "UninitializedEvaluationNet"]$

Out[4]=

Evaluation functions

Evaluation function for the navigation model

Define an evaluation function to calculate the turning angle in radians:

In[5]:=

$getTurnAngle[img_, b1_, b2_] := With[{netOut = NetModel[ "Micro Aerial Vehicle Trail Navigation Nets Trained on IDSIA \ Swiss Alps and PASCAL VOC Data"][img]}, b1*(netOut["view_orientation"][[3]] - netOut["view_orientation"][[1]]) + b2*(netOut["lateral_offset"][[3]] - netOut["lateral_offset"][[1]]) ]; getTurnAngle[img_] := getTurnAngle[img, 10, 10]$

Evaluation function for object detection model

Write an evaluation function to scale the result to the input image size and suppress the least probable detections:

In[6]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/d6421455-104c-49a6-b76e-ee492bed6245"]

Label list for the object detection model

Define the label list for this model. Integers in the model's output correspond to elements in the label list:

In[7]:=

labels = {"aeroplane", "bicycle", "bird", "boat", "bottle", "bus", "car", "cat", "chair", "cow", "diningtable", "dog", "horse", "motorbike", "person", "pottedplant", "sheep", "sofa", "train", "tvmonitor"};

Basic usage: navigation model

Calculate the turn angle in radians given a test image:

In[8]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/4056bc3f-7df8-4c6e-8418-a0a200903b88"]

Out[8]=

Basic usage: object detection model

Obtain the detected bounding boxes with their corresponding classes and confidences for a given image:

In[9]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/d566d376-4a03-4f1d-be00-285776ff2c9f"]

In[10]:=

Out[10]=

Inspect which classes are detected:

In[11]:=

Out[11]=

In[12]:=

Out[12]=

Visualize the detection:

In[13]:=

HighlightImage[testImage, MapThread[{White, Inset[Style[labels[[#2]], Black, FontSize -> Scaled[1/25], Background -> GrayLevel[1, .6]], Last[#1], {Right, Top}], #1} &,
Transpose@detection]]

Out[13]=

Detection results

Define an image:

In[14]:=

(* Evaluate this cell to get the example input *) CloudGet["https://www.wolframcloud.com/obj/91115a35-2f65-417e-952e-ced73a0d5570"]

The network computes 98 bounding boxes and the probability that the objects in each box are of any given class:

In[15]:=

$res = NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained \ on IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}][testImage];$

In[16]:=

Out[16]=

Visualize all the boxes predicted by the net scaled by their "Confidence" measures:

In[17]:=

rectangles = res["Boxes"];
rectangles = ArrayReshape[ArrayReshape[rectangles, {98, 4}], {98, 2, 2}];
rectangles = Rectangle @@@ rectangles;
objectness = ArrayFlatten[res["Confidence"], 1];

In[18]:=

Graphics[
MapThread[{EdgeForm[Opacity[#1 + 0.12]], #2} &, {objectness, rectangles}],
BaseStyle -> {FaceForm[], EdgeForm[{Thin, Black}]}
]

Out[18]=

Net information

Inspect the number of parameters of all arrays in the net:

In[19]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "ArraysElementCounts"]$

Out[19]=

In[20]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "ArraysElementCounts"]$

Out[20]=

Obtain the total number of parameters:

In[21]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "ArraysTotalElementCount"]$

Out[21]=

In[22]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "ArraysTotalElementCount"]$

Out[22]=

Obtain the layer type counts:

In[23]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "LayerTypeCounts"]$

Out[23]=

In[24]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "LayerTypeCounts"]$

Out[24]=

Display the summary graphic:

In[25]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "ObjectDetection"}], "SummaryGraphic"]$

Out[25]=

In[26]:=

$Information[ NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> "Navigation"}], "SummaryGraphic"]$

Out[26]=

Export to MXNet

Export the nets into a format that can be opened in MXNet:

In[27]:=

$jsonPaths = Map[ Export[ FileNameJoin[{$TemporaryDirectory, # <> ".json"}], NetModel[{"Micro Aerial Vehicle Trail Navigation Nets Trained on \ IDSIA Swiss Alps and PASCAL VOC Data", "Task" -> #}], "MXNet" ] &, {"ObjectDetection", "Navigation"} ]$

Out[27]=

Export also creates a net.params file containing parameters:

In[28]:=

Out[28]=

Get the size of the parameter files:

In[29]:=

Out[29]=

Construction Notebook

Download Construction Notebook

Open in Wolfram Cloud

Requirements

Wolfram Language 12.1 (March 2020) or above

Resource History

Date Created: 20 February 2020
Latest Update: 2 March 2020

Reference

N. Smolyanskiy, A. Kamenev, J. Smith, S. Birchfield, "Toward Low-Flying Autonomous MAV Trail Navigation Using Deep Neural Networks for Environmental Awareness," arXiv:1705.02550 (2017)
Available from: https://github.com/NVIDIA-AI-IOT/redtail
Rights: Model License