%% Calculate Statistics by Group
% Create a tall table, then calculate the mean flight delay for each year in the data.
%
% Create a tall table for the airlinesmall.csv data set. The data contains information about arrival and departure times of US flights. Remove rows of missing data from the table and extract the ArrDelay, DepDelay, and Year variables. These variables are vectors of arrival and departure delays and of the associated years for each flight in the data set.

ds = tabularTextDatastore('airlinesmall.csv','TreatAsMissing','NA');
ds.SelectedVariableNames = {'ArrDelay' 'DepDelay' 'Year'};
tt = tall(ds);
tt = rmmissing(tt);

%
% Use matlab.tall.reduce to apply two functions to the tall table. The first function combines the ArrDelay and DepDelay variables to find the total mean delay for each flight. The function determines how many unique years are in each chunk of data, and then cycles through each year and calculates the average total delay for flights in that year. The result is a two-variable table containing the year and mean total delay. This intermediate data needs to be reduced further to arrive at the mean delay per year. Save this function in your current folder as transform_fcn.m.

type transform_fcn

%
% The second function uses the results from the first function to calculate the mean total delay for each year. The output from reduce_fcn is compatible with the output from transform_fcn, so that blocks of data can be concatenated in any order and continually reduced until only one row remains for each year.

type reduce_fcn

%
% Apply the transform and reduce functions to the tall vectors. Since the inputs (type double) and outputs (type table) have different data types, use the 'OutputsLike' name-value pair to specify that the output is a table. A simple way to specify the type of the output is to call the transform function with dummy inputs.

a = tt.ArrDelay;
b = tt.DepDelay;
c = tt.Year;
d1 = matlab.tall.reduce(@transform_fcn, @reduce_fcn, a, b, c, 'OutputsLike',{transform_fcn(0,0,0)})

%
% Gather the results into memory to see the mean total flight delay per year.

d1 = gather(d1)

%
%% Alternative Approach
% Another way to calculate the same statistics by group is to use splitapply to call matlab.tall.reduce (rather than using matlab.tall.reduce to call splitapply).
% Using this approach, you call findgroups and splitapply directly on the data. The function mySplitFcn that operates on each group of data includes a call to matlab.tall.reduce. The transform and reduce functions employed by matlab.tall.reduce do not need to group the data, so those functions just perform calculations on the pregrouped data that splitapply passes to them.

type mySplitFcn

%
% Call findgroups and splitapply to operate on the data and apply mySplitFcn to each group of data.

groups = findgroups(c);
d2 = splitapply(@mySplitFcn, a, b, c, groups);
d2 = gather(d2)

%
% Copyright 2018 The MathWorks, Inc.
%