%% Create Simple Text Model for Classification
% This example shows how to train a simple text classifier on word frequency counts using a bag-of-words model.
% You can create a simple classification model which uses word frequency counts as predictors. This example trains a simple classification model to predict the category of factory reports using text descriptions.
%% Load and Extract Text Data
% Load the example data. The file factoryReports.csv contains factory reports, including a text description and categorical labels for each report.

filename = "factoryReports.csv";
data = readtable(filename,'TextType','string');
head(data)

% Convert the labels in the Category column of the table to categorical and view the distribution of the classes in the data using a histogram.

data.Category = categorical(data.Category);
figure
histogram(data.Category)
xlabel("Class")
ylabel("Frequency")
title("Class Distribution")

% Partition the data into a training partition and a held-out test set. Specify the holdout percentage to be 10%.

cvp = cvpartition(data.Category,'Holdout',0.1);
dataTrain = data(cvp.training,:);
dataTest = data(cvp.test,:);

% Extract the text data and labels from the tables.

textDataTrain = dataTrain.Description;
textDataTest = dataTest.Description;
YTrain = dataTrain.Category;
YTest = dataTest.Category;

%
%% Prepare Text Data for Analysis
% Create a function which tokenizes and preprocesses the text data so it can be used for analysis. The function preprocessText, performs the following steps in order:
% Tokenize the text using tokenizedDocument.
% Remove a list of stop words (such as "and", "of", and "the") using removeStopWords.
% Lemmatize the words using normalizeWords.
% Erase punctuation using erasePunctuation.
% Remove words with 2 or fewer characters using removeShortWords.
% Remove words with 15 or more characters using removeLongWords.
% Use the example preprocessing function preprocessText to prepare the text data.

documents = preprocessText(textDataTrain);
documents(1:5)

% Create a bag-of-words model from the tokenized documents.

bag = bagOfWords(documents)

% Remove words from the bag-of-words model that do not appear more than two times in total. Remove any documents containing no words from the bag-of-words model, and remove the corresponding entries in labels.

bag = removeInfrequentWords(bag,2);
[bag,idx] = removeEmptyDocuments(bag);
YTrain(idx) = [];
bag

%
%% Train Supervised Classifier
% Train a supervised classification model using the word frequency counts from the bag-of-words model and the labels.
% Train a multiclass linear classification model using fitcecoc. Specify the Counts property of the bag-of-words model to be the predictors, and the event type labels to be the response. Specify the learners to be linear. These learners support sparse data input.

XTrain = bag.Counts;
mdl = fitcecoc(XTrain,YTrain,'Learners','linear')

% For a better fit, you can try specifying different parameters of the linear learners. For more information on linear classification learner templates, see templateLinear.
%
%% Test Classifier
% Predict the labels of the test data using the trained model and calculate the classification accuracy. The classification accuracy is the proportion of the labels that the model predicts correctly.
% Preprocess the test data using the same preprocessing steps as the training data. Encode the resulting test documents as a matrix of word frequency counts according to the bag-of-words model.

documentsTest = preprocessText(textDataTest);
XTest = encode(bag,documentsTest);

% Predict the labels of the test data using the trained model and calculate the classification accuracy.

YPred = predict(mdl,XTest);
acc = sum(YPred == YTest)/numel(YTest)

%
%% Predict Using New Data
% Classify the event type of new factory reports. Create a string array containing the new factory reports.

str = [
    "Coolant is pooling underneath sorter."
    "Sorter blows fuses at start up."
    "There are some very loud rattling sounds coming from the assembler."];
documentsNew = preprocessText(str);
XNew = encode(bag,documentsNew);
labelsNew = predict(mdl,XNew)

%
%% Example Preprocessing Function
% The function preprocessText, performs the following steps in order:
% Tokenize the text using tokenizedDocument.
% Remove a list of stop words (such as "and", "of", and "the") using removeStopWords.
% Lemmatize the words using normalizeWords.
% Erase punctuation using erasePunctuation.
% Remove words with 2 or fewer characters using removeShortWords.
% Remove words with 15 or more characters using removeLongWords.

function documents = preprocessText(textData)

% Tokenize the text.
documents = tokenizedDocument(textData);

% Remove a list of stop words then lemmatize the words. To improve
% lemmatization, first use addPartOfSpeechDetails.
documents = addPartOfSpeechDetails(documents);
documents = removeStopWords(documents);
documents = normalizeWords(documents,'Style','lemma');

% Erase punctuation.
documents = erasePunctuation(documents);

% Remove words with 2 or fewer characters, and words with 15 or more
% characters.
documents = removeShortWords(documents,2);
documents = removeLongWords(documents,15);

end

% Copyright 2018 The MathWorks, Inc.