Wolfram Function Repository

Instant-use add-on functions for the Wolfram Language

Function Repository Resource:

 FocalLossLayer

Source Notebook

A NetGraph layer implementing focal loss

Contributed by: Michael Sollami

ResourceFunction["FocalLossLayer"][α,γ]

represents a net layer that implements focal loss.

Details and Options

ResourceFunction["FocalLossLayer"] exposes the following ports for use in NetGraph etc.:
"Input"scalar values between 0 and 1, or arrays of these
"Target"scalar values between 0 and 1, or arrays of these
"Loss"real number
ResourceFunction["FocalLossLayer"] is a variant of binary cross-entropy (see CrossEntropyLossLayer) that downweights well-classified examples.
ResourceFunction["FocalLossLayer"] is defined in the 2017 paper "Focal Loss for Dense Object Detection".
The paper's authors suggest default values of 0.25 and 2 for α and γ, respectively.
ResourceFunction["FocalLossLayer"] takes the same options as NetGraph.

Examples

Basic Examples (4) 

Create a focal loss layer:

In[1]:=
loss = ResourceFunction["FocalLossLayer"][.25, 2]
Out[1]=

Apply it to a given input and target:

In[2]:=
loss[<|"Input" -> RandomReal[1, {10, 10}], "Target" -> RandomReal[1, {10, 10}]|>]
Out[2]=

Use FocalLossLayer with single probabilities for the input and target:

In[3]:=
fl = ResourceFunction["FocalLossLayer"][.25, 2, "Input" -> "Real"]; fl[<|"Input" -> 0.2, "Target" -> 0.8|>]
Out[4]=

Apply FocalLossLayer with inputs and targets being matrices of binary-class probabilities:

In[5]:=
enc = NetEncoder[{"Image", {6, 11}, ColorSpace -> "Grayscale"}]; loss = ResourceFunction["FocalLossLayer"][.25, 2, "Input" -> enc, "Target" -> enc]
Out[6]=

Apply the layer to an input and target:

In[7]:=
loss[<|"Input" -> \!\(\* GraphicsBox[ TagBox[RasterBox[{{128, 128, 128, 128, 128, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 128, 128, 128, 128, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 128, 128, 128, 128, 128}}, {{0, 11}, {6, 0}}, {0, 255}, ColorFunction->GrayLevel], BoxForm`ImageTag[ "Byte", ColorSpace -> Automatic, Interleaving -> None, Magnification -> 3], Selectable->False], DefaultBaseStyle->"ImageGraphics", ImageSize->Magnification[3], ImageSizeRaw->{6, 11}, PlotRange->{{0, 6}, {0, 11}}]\), "Target" -> \!\(\* GraphicsBox[ TagBox[RasterBox[{{128, 128, 128, 128, 128, 128}, {128, 230, 230, 230, 230, 128}, {128, 230, 230, 230, 230, 128}, {128, 230, 230, 230, 230, 128}, {128, 230, 230, 230, 230, 128}, {128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}, { 128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}}, {{0, 11}, {6, 0}}, {0, 255}, ColorFunction->GrayLevel], BoxForm`ImageTag[ "Byte", ColorSpace -> Automatic, Interleaving -> None, Magnification -> 3], Selectable->False], DefaultBaseStyle->"ImageGraphics", ImageSize->Magnification[3], ImageSizeRaw->{6, 11}, PlotRange->{{0, 6}, {0, 11}}]\)|>]
Out[7]=

Use a focal loss layer during training:

In[8]:=
floss = ResourceFunction["FocalLossLayer"][.25, 2]; net = NetInitialize@ NetChain[{8, 1, LogisticSigmoid, PartLayer[1]}, "Input" -> 3, "Output" -> "Boolean"]; trainData = Flatten@Table[{i, j, k} -> (i > j) && (k > i), {i, 10}, {j, 10}, {k, 10}]; result = NetTrain[net, trainData, LossFunction -> floss, TimeGoal -> 3]
Out[11]=
In[12]:=
Graphics3D[ Table[{If[result[{i, j, k}], Blue, LightGray], Point[{i, j, k}]}, {i, 10}, {j, 10}, {k, 10}], Boxed -> True]
Out[12]=

Scope (1) 

Define a FocalLossLayer operating on color images:

In[13]:=
rgbloss = ResourceFunction["FocalLossLayer"][1, 2, "Input" -> NetEncoder[{"Image", ColorSpace -> "RGB"}], "Target" -> NetEncoder[{"Image", ColorSpace -> "RGB"}]]
Out[13]=
In[14]:=
DiscretePlot[ rgbloss[<|"Input" -> Image@ConstantArray[{n, n, n}, {10, 10}], "Target" -> Image@ConstantArray[1, {3, 10, 10}]|>], {n, 0, 1, 0.01}, Joined -> False, Filling -> Axis, PlotRange -> Most]
Out[14]=

Properties and Relations (1) 

Plot a comparison of FocalLossLayer (abbreviated as "FL") and standard CrossEntropy (abbreviated as "CE"). Note that setting γ>0 reduces the relative loss for well-classified examples (pt>.5), thus focusing on the harder examples:

In[15]:=
Clear[fl, vals, exprs]; fl[p_, \[Gamma]_] := -(1 - p)^\[Gamma] Log[p]; vals = {0, .5, 1, 2, 5, 10}; exprs = Table[fl[p, \[Gamma]], {\[Gamma], vals}]; Plot[exprs, {p, 0, 1}, PlotRange -> {{0, 1}, {0, 5}}, PlotLegends -> (HoldForm[\[Gamma] = #] & /@ vals), FrameTicks -> {{0}~Join~Range[0.2, .8, .2]~Join~{1}, Range[0, 5]}, Frame -> {{True, False}, {True, False}}, FrameLabel -> {"probability of ground truth label", "loss"}, FrameStyle -> 13, PlotRangePadding -> Scaled[.0], ImageMargins -> 5, Epilog -> {Inset[ Style[Column[{Text[ HoldForm[ FL[Subscript[p, t]] = -(1 - Subscript[p, t])^\[Gamma] Log[Subscript[p, t]]]], Text[ HoldForm[CE[Subscript[p, t]] = -Log[Subscript[p, t]]]]}], 16], {.42, 4.1}], Inset[Style[ \!\(\*UnderscriptBox[\("\<well-classified examples\>"\), StyleBox["⏞", FontSize->14, "NodeID" -> 53]]\), 13, Darker@Gray], {.78, .95}] } ]
Out[736]=

Publisher

Michael Sollami

Version History

  • 1.0.0 – 27 January 2020

Related Resources

License Information