%% File Ensemble Datastore Using Data in Text Files
% In predictive maintenance algorithm design, you frequently have system data in a plain text format such as comma-separated values (CSV). This example shows how to create and use a fileEnsembleDatastore object to manage an ensemble of data stored in such a format.
%% Ensemble Data
% Extract the compressed data for the example.

unzip fleetdata.zip  % extract compressed files

%
% The ensemble consists of ten files, fleetdata_01.txt, ..., fleetdata_10.txt, each containing data for one car in a fleet of cars. Each file contains five unlabeled columns of data, corresponding to daily readings of the following values:
% Odometer reading at the end of the day, in miles
% Fuel consumed that day, in gallons
% Maximum rpm for the day
% Maximum engine temperature for the day, in degrees Celsius
% Engine light status at the end of the day (0 = off, 1 = on)
% Each file contains data for between about 80 and about 120 days of operation. The data sets were artificially manufactured for this example and do not correspond to real fleet data.
%
%% Configure the Ensemble Datastore
% Create a fileEnsembleDatastore object to manage the data.

location = pwd;
extension = '.txt';
fensemble = fileEnsembleDatastore(location,extension);

%
% Configure the ensemble datastore to use the provided function readFleetData.m to read data from the files.

addpath(fullfile(matlabroot,'examples','predmaint','main')) % Make sure functions are on path
fensemble.ReadFcn = @readFleetData;

%
% Because the columns in the data files are unlabeled, the function readFleetData attaches a predefined label to the corresponding data. Configure the ensemble data variables to match the labels defined in readFleetData.

fensemble.DataVariables = ["Odometer";"FuelConsump";"MaxRPM";"MaxTemp";"EngineLight"];

%
% The function readFleetData also parses the file name to return an ID of the car from which the data was collected, a number from 1 through 10. This ID is the ensemble independent variable.

fensemble.IndependentVariables = "ID";

% Specify all data variables and the independent variable as selected variables for reading from the ensemble datastore.

fensemble.SelectedVariables = [fensemble.IndependentVariables;fensemble.DataVariables];
fensemble

%
%% Read Ensemble Data
% When you call read on the ensemble datastore, it uses readFleetData to read the selected variables from the first ensemble member.

data1 = read(fensemble)

%
% Examine and plot the odometer data.

odo1 = data1.Odometer{1}
plot(odo1.Time,odo1.Var1)

%
% Compute the average gas mileage for this member of the fleet. This value is the odometer reading on the last day, divided by the total fuel consumed.

fuelConsump1 = data1.FuelConsump{1}.Var1;
totalConsump1 = sum(fuelConsump1);
totalMiles1 = odo1.Var1(end);
mpg1 = totalMiles1/totalConsump1

%
%% Batch-Process Data from All Ensemble Members
% If you call read again, it reads data from the next ensemble member and advances the LastMemberRead property of fensemble to reflect the file name of that ensemble. You can repeat the processing steps to compute the average gas mileage for that member. In practice, it is more useful to automate the process of reading and processing the data. To do so, reset the ensemble datastore to a state in which no data has been read. Then loop through the ensemble and perform the read and process steps for each member, returning a table that contains each car's ID and average gas mileage. (If you have Parallel Computing Toolbox™, you can use it to speed up the processing of larger data ensembles.)

reset(fensemble)
mpgData = zeros(10,2);    % preallocate array for 10 ensemble members
ct = 1;
while hasdata(fensemble)
    data = read(fensemble);
    odo = data.Odometer{1}.Var1;
    fuelConsump = data.FuelConsump{1}.Var1;
    totalConsump = sum(fuelConsump);
    mpg = odo(end)/totalConsump1;
    ID = data.ID;
    mpgData(ct,:) = [ID,mpg];
    ct = ct + 1;
end
mpgTable = array2table(mpgData,'VariableNames',{'ID','mpg'})

%
%
% Copyright 2018 The MathWorks, Inc.