Function Repository Resource:

FocalLossLayer

Source Notebook

A NetGraph layer implementing focal loss

Contributed by: Michael Sollami

ResourceFunction["FocalLossLayer"][α,γ]

represents a net layer that implements focal loss.

Details and Options

ResourceFunction["FocalLossLayer"] exposes the following ports for use in NetGraph etc.:
"Input"scalar values between 0 and 1, or arrays of these
"Target"scalar values between 0 and 1, or arrays of these
"Loss"real number
ResourceFunction["FocalLossLayer"] is a variant of binary cross-entropy (see CrossEntropyLossLayer) that downweights well-classified examples.
ResourceFunction["FocalLossLayer"] is defined in the 2017 paper "Focal Loss for Dense Object Detection".
The paper's authors suggest default values of 0.25 and 2 for α and γ, respectively.
ResourceFunction["FocalLossLayer"] takes the same options as NetGraph.

Examples

Basic Examples (4) 

Create a focal loss layer:

In[1]:=
loss = ResourceFunction["FocalLossLayer"][.25, 2]
Out[1]=

Apply it to a given input and target:

In[2]:=
loss[<|"Input" -> RandomReal[1, {10, 10}], "Target" -> RandomReal[1, {10, 10}]|>]
Out[2]=

Use FocalLossLayer with single probabilities for the input and target:

In[3]:=
fl = ResourceFunction["FocalLossLayer"][.25, 2, "Input" -> "Real"];
fl[<|"Input" -> 0.2, "Target" -> 0.8|>]
Out[4]=

Apply FocalLossLayer with inputs and targets being matrices of binary-class probabilities:

In[5]:=
enc = NetEncoder[{"Image", {6, 11}, ColorSpace -> "Grayscale"}];
loss = ResourceFunction["FocalLossLayer"][.25, 2, "Input" -> enc, "Target" -> enc]
Out[6]=

Apply the layer to an input and target:

In[7]:=
loss[<|"Input" -> \!\(\*
GraphicsBox[
TagBox[RasterBox[{{128, 128, 128, 128, 128, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 128, 128, 128, 128, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 26, 26, 26, 26, 128}, {128, 128, 128, 128, 128, 128}}, {{0, 11}, {6, 0}}, {0, 255},
ColorFunction->GrayLevel],
BoxForm`ImageTag[
       "Byte", ColorSpace -> Automatic, Interleaving -> None, Magnification -> 3],
Selectable->False],
DefaultBaseStyle->"ImageGraphics",
ImageSize->Magnification[3],
ImageSizeRaw->{6, 11},
PlotRange->{{0, 6}, {0, 11}}]\), "Target" -> \!\(\*
GraphicsBox[
TagBox[RasterBox[{{128, 128, 128, 128, 128, 128}, {128, 230, 230, 230,
         230, 128}, {128, 230, 230, 230, 230, 128}, {128, 230, 230, 230, 230, 128}, {128, 230, 230, 230, 230, 128}, {128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}, {
        128, 128, 128, 128, 128, 128}, {128, 128, 128, 128, 128, 128}}, {{0, 11}, {6, 0}}, {0, 255},
ColorFunction->GrayLevel],
BoxForm`ImageTag[
       "Byte", ColorSpace -> Automatic, Interleaving -> None, Magnification -> 3],
Selectable->False],
DefaultBaseStyle->"ImageGraphics",
ImageSize->Magnification[3],
ImageSizeRaw->{6, 11},
PlotRange->{{0, 6}, {0, 11}}]\)|>]
Out[7]=

Use a focal loss layer during training:

In[8]:=
floss = ResourceFunction["FocalLossLayer"][.25, 2];
net = NetInitialize@
   NetChain[{8, 1, LogisticSigmoid, PartLayer[1]}, "Input" -> 3, "Output" -> "Boolean"];
trainData = Flatten@Table[{i, j, k} -> (i > j) && (k > i), {i, 10}, {j, 10}, {k,
      10}];
result = NetTrain[net, trainData, LossFunction -> floss, TimeGoal -> 3]
Out[11]=
In[12]:=
Graphics3D[
 Table[{If[result[{i, j, k}], Blue, LightGray], Point[{i, j, k}]}, {i,
    10}, {j, 10}, {k, 10}], Boxed -> True]
Out[12]=

Scope (1) 

Define a FocalLossLayer operating on color images:

In[13]:=
rgbloss = ResourceFunction["FocalLossLayer"][1, 2, "Input" -> NetEncoder[{"Image", ColorSpace -> "RGB"}],
  "Target" -> NetEncoder[{"Image", ColorSpace -> "RGB"}]]
Out[13]=
In[14]:=
DiscretePlot[
 rgbloss[<|"Input" -> Image@ConstantArray[{n, n, n}, {10, 10}], "Target" -> Image@ConstantArray[1, {3, 10, 10}]|>], {n, 0, 1, 0.01}, Joined -> False, Filling -> Axis, PlotRange -> Most]
Out[14]=

Properties and Relations (1) 

Plot a comparison of FocalLossLayer (abbreviated as "FL") and standard CrossEntropy (abbreviated as "CE"). Note that setting γ>0 reduces the relative loss for well-classified examples (pt>.5), thus focusing on the harder examples:

In[15]:=
Clear[fl, vals, exprs];
fl[p_, \[Gamma]_] := -(1 - p)^\[Gamma] Log[p];
vals = {0, .5, 1, 2, 5, 10};
exprs = Table[fl[p, \[Gamma]], {\[Gamma], vals}];
Plot[exprs, {p, 0, 1}, PlotRange -> {{0, 1}, {0, 5}}, PlotLegends -> (HoldForm[\[Gamma] = #] & /@ vals),
 FrameTicks -> {{0}~Join~Range[0.2, .8, .2]~Join~{1}, Range[0, 5]},
 Frame -> {{True, False}, {True, False}}, FrameLabel -> {"probability of ground truth label", "loss"}, FrameStyle -> 13, PlotRangePadding -> Scaled[.0], ImageMargins -> 5, Epilog -> {Inset[
    Style[Column[{Text[
        HoldForm[
         FL[Subscript[p, t]] = -(1 - Subscript[p, t])^\[Gamma] Log[Subscript[p, t]]]], Text[
        HoldForm[CE[Subscript[p, t]] = -Log[Subscript[p, t]]]]}], 16], {.42, 4.1}],
   Inset[Style[
\!\(\*UnderscriptBox[\("\<well-classified examples\>"\), 
StyleBox["⏞",
FontSize->14,
"NodeID" -> 53]]\), 13, Darker@Gray], {.78, .95}]
   }
 ]
Out[736]=

Publisher

Michael Sollami

Version History

  • 1.0.0 – 27 January 2020

Related Resources

License Information