%% Missing Data in MATLAB
% Working with missing data is a common task in data preprocessing. Although sometimes missing values signify a meaningful event in the data, they often represent unreliable or unusable data points. In either case, MATLAB® has many options for handling missing data.
%
%% Create and Organize Missing Data
% The form that missing values take in MATLAB depends on the data type. For example, numeric data types such as double use NaN (not a number) to represent missing values.

x = [NaN 1 2 3 4];

% You can also use the missing value to represent missing numeric data or data of other types, such as datetime, string, and categorical. MATLAB automatically converts the missing value to the data's native type.

xDouble = [missing 1 2 3 4]
xDatetime = [missing datetime(2014,1:4,1)]
xString = [missing "a" "b" "c" "d"]
xCategorical = [missing categorical({'cat1' 'cat2' 'cat3' 'cat4'})]

%
% A data set might contain values that you want to treat as missing data, but are not standard MATLAB missing values in MATLAB such as NaN. You can use the standardizeMissing function to convert those values to the standard missing value for that data type. For example, treat 4 as a missing double value in addition to NaN.

xStandard = standardizeMissing(xDouble,[4 NaN])

%
% Suppose you want to keep missing values as part of your data set but segregate them from the rest of the data. Several MATLAB functions enable you to control the placement of missing values before further processing. For example, use the 'MissingPlacement' option with the sort function to move NaNs to the end of the data.

xSort = sort(xStandard,'MissingPlacement','last')

%
%% Find, Replace, and Ignore Missing Data
% Even if you do not explicitly create missing values in MATLAB, they can appear when importing existing data or computing with the data. If you are not aware of missing values in your data, subsequent computation or analysis can be misleading.
% For example, if you unknowingly plot a vector containing a NaN value, the NaN does not appear because the plot function ignores it and plots the remaining points normally.

nanData = [1:9 NaN];
plot(1:10,nanData)

%
% However, if you compute the average of the data, the result is NaN. In this case, it is more helpful to know in advance that the data contains a NaN, and then choose to ignore or remove it before computing the average.

meanData = mean(nanData)

%
% One way to find NaNs in data is by using the isnan function, which returns a logical array indicating the location of any NaN value.

TF = isnan(nanData)

%
% Similarly, the ismissing function returns the location of missing values in data for multiple data types.

TFdouble = ismissing(xDouble)
TFdatetime = ismissing(xDatetime)

%
% Suppose you are working with a table or timetable made up of variables with multiple data types. You can find all of the missing values with one call to ismissing, regardless of their type.

xTable = table(xDouble',xDatetime',xString',xCategorical')
TF = ismissing(xTable)

%
% Missing values can represent unusable data for processing or analysis. Use fillmissing to replace missing values with another value, or use rmmissing to remove missing values altogether.

xFill = fillmissing(xStandard,'constant',0)
xRemove = rmmissing(xStandard)

%
% Many MATLAB functions enable you to ignore missing values, without having to explicitly locate, fill, or remove them first. For example, if you compute the sum of a vector containing NaN values, the result is NaN.  However, you can directly ignore NaNs in the sum by using the 'omitnan' option with the sum function.

sumNan = sum(xDouble)
sumOmitnan = sum(xDouble,'omitnan')

% Copyright 2012 The MathWorks, Inc.