Wolfram Language Paclet Repository
Community-contributed installable additions to the Wolfram Language
Long form data transformation
Introduction
In this notebook we consider in more detail the (tabular) data transformation operations Long form.
Remark: This notebook is an expanded version of the notebook used in the Wolfram U session [AAv2].
Remark: We use several Wolfram Function Repository (WFR) functions: and .
Load the paclet
In[34]:=
Needs["AntonAntonov`DataReshapers`"]
What kind of data?
We concentrate on using data frames (datasets).
More precisely tabular data and simple collections of tabular data.
I.e. in WL -- datasets or lists or associations of datasets.
What workflows considered?
Here is a flow chart that shows the targeted workflows:
In[35]:=
plWorkflows=ImageCrop@Import["https://github.com/antononcube/ConversationalAgents/raw/master/ConceptualDiagrams/Tabular-data-transformation-workflows.jpg"]
Out[35]=
Long form in more detail
Remark: This “Illustrating example 2” from the tech note . (Also, session [AAv1].)
In[1]:=
Column names as data
Column names as data
Take a dataset with time series specified through separate columns and convert into easier to query dataset:
Get the Lake Mead levels data
In[36]:=
ExampleData[{"Statistics","LakeMeadLevels"},"LongDescription"]
Out[36]=
Elevation of Lake Mead at Hoover Dam, in feet, in years 1935-2009. The observation on January of 1935 was missing.
Get the Lake Mead levels dataset
In[37]:=
dsLakeData=ResourceFunction["ExampleDataset"][{"Statistics","LakeMeadLevels"}]
Out[37]=
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Convert to long form
In[38]:=
dsLong=
[dsLakeData,"Year","VariablesTo""Month","ValuesTo""Elevation"]
 | LongFormDataset |
Out[38]=
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Modify month from string value to integer value
In[39]:=
dsLong=dsLong[All,Join[#,"Month"DateList[{#Month,{"MonthShort"}}]〚2〛]&]
Out[39]=
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Group by year and show short form of the result
For each year find the corresponding time series (of lake levels)
Show a sample of the time series found above
Calendar time series
Calendar time series
The previous time series are relative in time. What if we want to do calendar time series?
First we have to modify the month column to be "ObservationTime":
Split the long form dataset by year and make time series with the columns "ObservationTime" and "Elevation":
Plot the obtained time series:
Alternative solution without long form
Alternative solution without long form
(This sub-section was not discussed in Wolfram U recording of the “Q&A Introduction” session. It is given in this notebook in order to make a more convenient reference.)
We can apply the Split-Transform-Combine pattern directly without using long form:
Note that:
◼
In the transformation above we implicitly rely on the sorted order of the month name columns.
◼
Using Rest and First in the third argument given to GroupBy corresponds to the transformations to get the long form.
Long form applications
Many applications of Long form transformation come from the fact that metadata is converted to data
In[6]:=
Combinations of the heterogenous data
Combinations of the heterogenous data
Using long forms makes easier the programmatic manipulation of heterogenous data.
Get datasets with different number of rows and columns that correspond to items and variables of different kinds:
Convert all datasets into long form datasets:
For each long form dataset change the automatic key column to include dataset's name:
Join all long form datasets into one dataset and show a sample:
Long form vs sparse array
Sparse matrices (sparse arrays) have representation very similar to that of long form. Here is a random sparse matrix:
Generate random (sparse) matrix
Matrix form of the random matrix
Matrix plot of the random matrix
Compare the array rules of that sparse matrix with its long form representation.
Here are the array rules:
Here is tabulation of the array rules:
Here is the long form representation:
Sort the long form dataset:
Cross tabulate the long form
Matrix form of the corresponding sparse matrix:
Here is comparison of the representations of sparse arrays vs long form:
References