Wolfram Research

Function Repository Resource:

QuantileRegression

Source Notebook

Computes quantile regression fits over a time series, a list of numbers, or a list of numeric pairs.

Contributed by: Anton Antonov

ResourceFunction["QuantileRegression"][data,knots,probs]

does quantile regression over the times series or data array data using the knots specification knots for the probabilties probs .

ResourceFunction["QuantileRegression"][data,knots,probs,opts]

does quantile regression with the options opts .

Details and Options

QuantileRegression works on time series, lists of numbers, and lists of numeric pairs.
The curves computed with quantile regression are called regression quantiles.
The regression quantiles corresponding to the specified probabilities are linear combinations of B-splines generated over the specified knots.
In other words, QuantileRegression computes fits using a B-spline functions basis. The basis is specified with the knots argument and the option InterpolationOrder.
QuantilesRegression takes the following options:
InterpolationOrder 3 interpolation order
Method LinearProgramming method for the quantile regression computations

Examples

Basic Examples

Make a random signal:

In[1]:=
SeedRandom[23];
n = 200;
randData = Transpose[{Range[n], RandomReal[{0, 100.}, n]}];

Compute Quantile Regression with 5 knots for the probabilities 0.25 and 0.75:

In[2]:=
qFuncs = ResourceFunction["QuantileRegression"][randData, 5, {0.25, 0.75}];

Here are the formulas of the obtained regression quantiles:

In[3]:=
Simplify /@ Through[qFuncs[x]]
Out[3]=

Here is a plot of the original data and the obtained regression quantiles:

In[4]:=
ListLinePlot[{randData, qFuncs[[1]] /@ randData[[All, 1]], qFuncs[[2]] /@ randData[[All, 1]]}, PlotLegends -> {"data", 0.25`, 0.75`}, PlotStyle -> {Thin, Thick, Thick}, PlotTheme -> "Detailed"]
Out[4]=

Find the fraction of the data points that are under the second regression quantile:

In[5]:=
Length[Select[randData, #[[2]] < qFuncs[[2]][#[[1]]] &]]/
  Length[randData] // N
Out[5]=

The obtained fraction is close to the second probability, 0.75, given to QuantileRegression.

Scope

Here is a Quantile regression computation over a numerical vector:

In[6]:=
vecData = Sin[Range[200]/6] + RandomReal[{-0.5, 0.5}, 200];
qFunc = First@ResourceFunction["QuantileRegression"][vecData, 12, 0.5];
ListLinePlot[{vecData, qFunc[#] & /@ Range[Length[vecData]]}, PlotRange -> All, PlotTheme -> "Detailed", PlotLegends -> {"vector", "fit"}]
Out[7]=

Here is a Quantile regression computation over a times series object:

In[8]:=
finData = FinancialData["GE", {{2010, 1, 1}, {2019, 10, 15}, "Week"}];
qFunc = First@ResourceFunction["QuantileRegression"][finData, 12, 0.5];
DateListPlot[{finData, {#, qFunc[#]} & /@ finData["Times"]}, Joined -> {False, True}, PlotLegends -> {"time series", "fit"}]
Out[9]=

The second argument -- the knots specification -- can be an integer specifying the number of knots, or a list of numbers specifying the knots of the B-spline basis:

In[10]:=
qFuncs = ResourceFunction["QuantileRegression"][randData, randData[[1 ;; -1 ;; 3, 1]], 0.5];

Options

InterpolationOrder

The option InterpolationOrder specifies the polynomial order of the B-spline basis. Its values are expected to be non-negative integers:

In[11]:=
qFuncs = Table[
   First@ResourceFunction["QuantileRegression"][randData, 5, 0.5, InterpolationOrder -> i], {i, {0, 1, 3}}];
In[12]:=
ListLinePlot[
 Prepend[(#1 /@ Range[Length[randData]] &) /@ qFuncs, randData], PlotStyle -> {Thin, Thick, Thick, Thick}, PlotLegends -> Prepend[{"0.5, interpolation order 0", "0.5, interpolation order 1",
     "0.5, interpolation order 3"}, "data"], PlotTheme -> "Detailed"]
Out[12]=

Method

QuantileRegression uses LinearProgramming. Additional parameters can be passed to LinearProgramming with the Method option:

In[13]:=
qFuncs = ResourceFunction["QuantileRegression"][randData, 5, 0.5, Method -> {LinearProgramming, Tolerance -> 10^(-2), Method -> "CLP"}];
In[14]:=
ListLinePlot[{randData, qFuncs[[1]] /@ randData[[All, 1]]}, PlotStyle -> {Thin, Thick}, PlotLegends -> {"data", 0.5`}, PlotTheme -> "Detailed"]
Out[14]=

Applications

Fit for heteroscedastic data

Here is heteroscedastic data (the variance is not constant with respect to the predictor variable):

In[15]:=
distData = Table[{x, Exp[-x^2] + RandomVariate[
      NormalDistribution[0, .15 \[Sqrt](Abs[1.5 - x]/1.5)]]}, {x, -3, 3, .01}];
ListPlot[distData, PlotTheme -> "Detailed", ImageSize -> Medium]
Out[14]=

Find quantile regression fits:

In[16]:=
probs = {0.1, 0.25, 0.5, 0.75, 0.9};
qFuncs = ResourceFunction["QuantileRegression"][distData, 4, probs];

Plot the data and the regression quantiles:

In[17]:=
ListPlot[Prepend[(Transpose[{distData[[All, 1]], #1 /@ distData[[All, 1]]}] &) /@ qFuncs, distData], Joined -> Prepend[Table[True, Length[probs]], False], PlotLegends -> Prepend[probs, "data"], PlotTheme -> "Detailed", FrameLabel -> {"Time", "Temperature"}, ImageSize -> Medium]
Out[17]=

Note that the regression quantiles clearly outline the heteroscedastic nature of the data.

Find variance anomalies

A certain contextual type of anomaly is a subset of points that have variance very different than other subsets. Using Quantile regression (1) we can evaluate the regressor dependent variance for each point using the regression quantiles 0.25 and 0.75, and (2) find the points that have outlier variances.

Here we compute and plot the variance estimates for a signal:

In[18]:=
qFuncs = ResourceFunction["QuantileRegression"][distData, 4, {0.25, 0.75}];
variances = Map[Abs@*Subtract @@ Through[qFuncs[#]] &, distData[[All, 1]]];
ListPlot[variances, PlotRange -> All]
Out[16]=

Here we find the lower and upper thresholds for the variance outliers:

In[19]:=
medianDistances = variances - Median[variances];
{lowerThreshold, upperThreshold} = Quantile[medianDistances, {0.2, 0.8}]
Out[17]=

Here we find the outlier positions:

In[20]:=
varianceOutlierPositions = Flatten@Position[medianDistances, x_?NumberQ /; (x < lowerThreshold || upperThreshold < x)];

Here we plot the data and the found outliers:

In[21]:=
ListPlot[{distData, distData[[varianceOutlierPositions]]}, PlotRange -> All, PlotLegends -> {"data", "variance outliers"}]
Out[21]=

Fit and anomalies for financial time series

Here is a financial time series:

In[22]:=
finData = FinancialData["GE", {{2016, 1, 1}, {2019, 10, 15}, "Day"}]
Out[22]=
In[23]:=
finData // Head
Out[23]=

Do a quantile regression fit and plot it:

In[24]:=
qFunc = First@ResourceFunction["QuantileRegression"][finData, 50, 0.5];
DateListPlot[{finData, {#, qFunc[#]} & /@ finData["Times"]}, PlotTheme -> "Detailed", AspectRatio -> 1/4, ImageSize -> Large, PlotLegends -> {"data", "fit"}]
Out[19]=

Here are the errors of the found fit:

In[25]:=
DateListPlot[
 Map[{#[[1]], (qFunc[#[[1]]] - #[[2]])/#[[2]]} &, QuantityMagnitude[finData["Path"]]], Joined -> False, PlotRange -> All, Filling -> Axis, PlotTheme -> "Detailed", AspectRatio -> 1/4, ImageSize -> Large]
Out[25]=

Find anomalies positions in the list of fit errors:

In[26]:=
pos = FindAnomalies[
  Map[(qFunc[#[[1]]] - #[[2]])/#[[2]] &, QuantityMagnitude[finData["Path"]]], "AnomalyPositions"]
Out[26]=

Plot the data, fit, and anomalies:

In[27]:=
DateListPlot[{finData, {#, qFunc[#]} & /@ finData["Times"], finData["Path"][[pos]]}, PlotTheme -> "Detailed", AspectRatio -> 1/4, ImageSize -> Large, Joined -> {False, True, False}, PlotStyle -> {{GrayLevel[0.5`], Opacity[0.5`]}, {Red, Thick}, {Blue, PointSize[0.01`]}}, PlotLegends -> {"data", "fit", "anomalies"}]
Out[27]=

Analyze temperature time series

Get temperature data:

In[28]:=
tempData = WeatherData[{"Orlando", "Florida", "USA"}, "Temperature", {{2016, 1, 1}, {2019, 1, 1}, "Day"}]
Out[28]=

Convert the time series into a list of numeric pairs:

In[29]:=
tempData = QuantityMagnitude[tempData["Path"]];

Compute Quantile regression fits:

In[30]:=
probs = Sort[Join[Range[0.1, 0.9, 0.1], {0.01, 0.99}]];
qFuncs = ResourceFunction["QuantileRegression"][tempData, 10, probs];

Plot the data and the regression quantiles:

In[31]:=
DateListPlot[
 Prepend[(Transpose[{tempData[[All, 1]], #1 /@ tempData[[All, 1]]}] &) /@ qFuncs, tempData], Joined -> Prepend[Table[True, Length[probs]], False], PlotLegends -> Prepend[probs, "temperature data"], AspectRatio -> 1/4, PlotTheme -> "Detailed", FrameLabel -> {"Time", "Temperature"}, ImageSize -> Large]
Out[31]=

Find an estimate of the conditional Cumulative Distribution Function (CDF) at the date 2017-10-01:

In[32]:=
ListLinePlot[
 Transpose[{Through[qFuncs[AbsoluteTime[{2017, 10, 1}]]], probs}], Sequence[
 PlotTheme -> "Detailed", FrameLabel -> {"Temperature", "Probability"}]]
Out[32]=

Find outliers in the temperature data -- we define outliers as points below or above the 0.02 and 0.98 regression quantiles respectively:

In[33]:=
qFuncs = ResourceFunction["QuantileRegression"][tempData, 10, {0.02, 0.98}];
In[34]:=
bottomOutliers = Select[tempData, #[[2]] < qFuncs[[1]][#[[1]]] &];
topOutliers = Select[tempData, #[[2]] > qFuncs[[-1]][#[[1]]] &];
In[35]:=
DateListPlot[{tempData, {#, qFuncs[[1]][#]} & /@ tempData[[All, 1]], {#, qFuncs[[-1]][#]} & /@ tempData[[All, 1]], bottomOutliers, topOutliers}, Sequence[
 PlotTheme -> "Detailed", PlotLegends -> {
   "original data", "bottom regression quantile", "top regression quantile", "bottom outliers", "top outliers"}, Joined -> {False, True, True, False, False}, PlotStyle -> {{Gray, 
PointSize[0.002]}, {Black, Thin}, {Black, Thin}, {
PointSize[0.012]}, {
PointSize[0.012]}}, AspectRatio -> 1/4, FrameLabel -> {"Time", "Temperature"}, ImageSize -> Large]]
Out[35]=

Properties and Relations

QuantileRegression can be compared with FindFormula, Fit , LinearModelFit, and NonlinearModelFit:

In[36]:=
qFunc = First@
   ResourceFunction["QuantileRegression"][distData, 4, 0.5];
In[37]:=
fFunc = LinearModelFit[distData, Table[x^i, {i, 0, 6, 1}], x][
   "Function"];
In[38]:=
ListLinePlot[{distData, {#, qFunc[#]} & /@ distData[[All, 1]], {#, fFunc[#]} & /@ distData[[All, 1]]}, PlotStyle -> {Thin, Thick, Thick}, PlotLegends -> {"data", 0.5`, "mean"}, PlotTheme -> "Detailed"]
Out[38]=

Quantile regression is much more robust than Linear regression. In order to demonstrate that let us add a few large outliers in the data:

In[39]:=
distData3 = SortBy[Join[distData, {{1.5, 10}, {2.2, 30}}], First];

Here we apply Quantile regression and Linear regression as in the previous example:

In[40]:=
qFunc = First@
   ResourceFunction["QuantileRegression"][distData3, 4, 0.5];
In[41]:=
fFunc = LinearModelFit[distData3, Table[x^i, {i, 0, 6, 1}], x][
   "Function"];

Here is a plot of the obtained curves – note that the curve corresponding to Linear regression is different and a worse fit than the one from the previous example:

In[42]:=
ListLinePlot[{distData3, {#, qFunc[#]} & /@ distData3[[All, 1]], {#, fFunc[#]} & /@ distData3[[All, 1]]}, PlotStyle -> {Thin, Thick, Thick}, PlotLegends -> {"data", 0.5`, "mean"}, PlotTheme -> "Detailed"]
Out[42]=

Possible Issues

Slow computations

Because of the Linear programming formulation for some data and knots specifications the computations can be slow.

Fitting for probabilities 0 and 1

For most data the quantile regression fitting for probabilities 0 and 1 produces regression quantiles that “too far away from the data.”

Find regression quantiles for probabilities 0 and 0.5 and plot them:

In[43]:=
probs = {0, 0.5};
qFuncs = ResourceFunction["QuantileRegression"][distData, 6, probs];
ListPlot[Prepend[(Transpose[{distData[[All, 1]], #1 /@ distData[[All, 1]]}] &) /@ qFuncs, distData], Joined -> Prepend[Table[True, Length[probs]], False], PlotLegends -> Prepend[probs, "data"], PlotTheme -> "Detailed", FrameLabel -> {"Time", "Temperature"}, ImageSize -> Medium]
Out[41]=

Find regression quantiles for probabilities 0.5, and 1 and plot them:

In[44]:=
probs = {0.5, 1};
qFuncs = ResourceFunction["QuantileRegression"][distData, 6, probs];
ListPlot[Prepend[(Transpose[{distData[[All, 1]], #1 /@ distData[[All, 1]]}] &) /@ qFuncs, distData], Joined -> Prepend[Table[True, Length[probs]], False], PlotLegends -> Prepend[probs, "data"], PlotTheme -> "Detailed", FrameLabel -> {"Time", "Temperature"}, ImageSize -> Medium]
Out[32]=

One way to fix this is to use probabilities that are close to 0 and 1 from above and below respectively:

In[45]:=
probs = {0.0001, 0.5, 0.9999};
qFuncs = ResourceFunction["QuantileRegression"][distData, 6, probs];
ListPlot[Prepend[(Transpose[{distData[[All, 1]], #1 /@ distData[[All, 1]]}] &) /@ qFuncs, distData], Joined -> Prepend[Table[True, Length[probs]], False], PlotLegends -> Prepend[probs, "data"], PlotTheme -> "Detailed", FrameLabel -> {"Time", "Temperature"}, ImageSize -> Medium]
Out[46]=

Over-fitting

Consider the following non-linear data:

In[47]:=
nlData = {
Sequence[{0, 2.52}, {1., 2.83}, {2., 3}, {3., 3.2}, {4.1, 3.35}, {5., 3.47}, {6, 3.57}, {7, 3.66}, {8, 3.76}, {8.5, 3.81}, {9, 3.85}, {
    9.5, 3.89}, {10.1, 3.94}, {10.5, 3.98}, {11, 4.01}, {11.5, 4.06}, {12, 4.09}, {12.5, 4.15}, {13, 4.19}, {13.5, 4.25}, {
    14, 4.3}, {14.5, 4.35}, {15, 4.41}, {15.6, 4.47}, {16, 4.53}, {
    16.5, 4.6}, {17., 4.68}, {17.5, 4.77}, {18, 4.85}, {18.5, 4.96}, {
    19, 5.11}, {19.55, 5.34}, {19.7, 5.44}, {19.9, 5.58}, {20.1, 5.91}, {20.3, 6.27}, {20.5, 7.14}, {20.6, 7.14}, {20.8, 7.81}, {
    20.9, 8.32}, {21, 7.75}, {21.2, 9.07}, {21.4, 9.49}, {21.5, 9.71}, {21.6, 9.83}, {21.8, 10}, {22, 10.18}, {22.1, 10.21}, {
    22.2, 10.25}, {22.3, 10.27}, {22.5, 10.3}, {22.7, 10.42}, {22.9, 10.47}, {23.1, 10.52}, {23.3, 10.59}, {23.5, 10.63}, {23.7, 10.67}, {24, 10.74}, {24.2, 10.78}, {24.4, 10.8}, {24.6, 10.82}, {
    24.8, 10.84}, {25, 10.87}]};
ListPlot[nlData, Sequence[PlotTheme -> "Detailed", PlotStyle -> Red]]
Out[40]=

Make a quantile regression fit with 20 knots:

In[48]:=
qFunc20 = First@ResourceFunction["QuantileRegression"][nlData, 20, {0.5}, InterpolationOrder -> 2];

Make a quantile regression fit with 40 knots:

In[49]:=
qFunc40 = First@ResourceFunction["QuantileRegression"][nlData, 40, {0.5}, InterpolationOrder -> 2];

Plot the regression quantiles and the data:

In[50]:=
Plot[{qFunc20[x], qFunc40[x]}, {x, Min[nlData[[All, 1]]], Max[nlData[[All, 1]]]}, Epilog -> {Red, Point[nlData]}, PlotTheme -> "Detailed"]
Out[50]=

We can see that the regression quantile computed with 40 knots is “over-fitted” between 0 and 8 -- the B-spline basis knots are too densely placed between 0 and 8.

Intersecting regression quantiles

When regression quantiles are over-fitted then the estimate of the conditional Cumulative Distribution Function (CDF) can be problematic -- the estimated CDF is not a monotonically increasing function.

Compute regression quantiles using “too many” knots:

In[51]:=
probs = Sort[Join[Range[0.1, 0.9, 0.1], {0.01, 0.99}]];
qFuncs = ResourceFunction["QuantileRegression"][tempData, 30, probs];

Plot the regression quantiles:

In[52]:=
Block[{data = tempData[[1 ;; 300]]}, qFuncs = ResourceFunction["QuantileRegression"][data, 20, probs]; DateListPlot[
  Prepend[(Transpose[{data[[All, 1]], #1 /@ data[[All, 1]]}] &) /@ qFuncs, data], Joined -> Prepend[Table[True, Length[probs]], False], PlotLegends -> Prepend[probs, "temperature data"], AspectRatio -> 1/4, PlotTheme -> "Detailed", FrameLabel -> {"Time", "Temperature"}, ImageSize -> Large]]
Out[52]=

Here is the estimated conditional CDF:

In[53]:=
ListLinePlot[Transpose[{Through[qFuncs[tempData[[100, 1]]]], probs}], PlotTheme -> "Detailed", FrameLabel -> {"Temperature", "Probability"}]
Out[53]=

Rescaling

For certain data it is beneficial to rescale the predictor values, predicted values, or both before doing the quantile regression computations:

In[54]:=
finData2 = N@{10532750, 1342440, 714972, 1289802, 1302765, 814231, 830680, 416649, 80017, 24343, 1808, 1939, 25848, 32532, 21807, 26108, 12581, 36315, 34544, 3827, 7631, 7259};
In[55]:=
ResourceFunction["QuantileRegression"][finData2, 4, probs]
Out[55]=
In[56]:=
qFuncs = ResourceFunction["QuantileRegression"][Rescale[finData2], 4, probs];
Through[qFuncs[10]]
Out[53]=

Neat Examples

Compute and plot regression quantiles over symmetric data:

In[57]:=
SeedRandom[20];
distData = Table[{x, Exp[-x^2] + RandomVariate[
      NormalDistribution[0, .15 \[Sqrt](Abs[1.5 - x]/1.5)]]}, {x, -3, 3, .01}];
Grid[Table[Block[{tempData = RandomSample[distData, 100], data, probs},
   data = Join[tempData, {#[[1]], -#[[2]]} & /@ tempData];
   data = SortBy[Transpose[Rescale /@ Transpose[data]], #[[1]] &];
   probs = {0.02, 0.48, 0.52, 0.98};
   qFuncs = ResourceFunction["QuantileRegression"][data, 5, probs, Method -> {LinearProgramming, Method -> "InteriorPoint", Tolerance -> 10^(-2)}]; ListPlot[
    Prepend[(Transpose[{data[[All, 1]], #1 /@ data[[All, 1]]}] &) /@ qFuncs, data], Joined -> Prepend[Table[True, Length[probs]], False], Filling -> Prepend[Table[(i + 1) -> {i + 2}, {i, Length[probs] - 1}], 1 -> None], Frame -> True, FrameTicks -> False, Axes -> False, ImageSize -> Small, AspectRatio -> 1]
   ], 3, 3]]
Out[56]=

Resource History

Source Metadata

AuthorNotes

The workflows described above are simplified and streamlined with software monad QRMon.

License Information