%% Prepare Text Data for Analysis
% This example shows how to create a function which cleans and preprocesses text data for analysis.
% Text data can be large and can contain lots of noise which negatively affects statistical analysis. For example, text data can contain the following:
% Variations in case, for example "new" and "New"
% Variations in word forms, for example "walk" and "walking"
% Words which add noise, for example stop words such as "the" and "of"
% Punctuation and special characters
% HTML and XML tags
% These word clouds illustrate word frequency analysis applied to some raw text data from factory reports, and a preprocessed version of the same text data.
%
%% Load and Extract Text Data
% Load the example data. The file factoryReports.csv contains factory reports, including a text description and categorical labels for each event.

filename = "factoryReports.csv";
data = readtable(filename,'TextType','string');

% Extract the text data from the field Description, and the label data from the field Category.

textData = data.Description;
labels = data.Category;
textData(1:10)

%
%% Create Tokenized Documents
% Create an array of tokenized documents.

cleanedDocuments = tokenizedDocument(textData);
cleanedDocuments(1:10)

% To improve lemmatization, add part of speech details to the documents using addPartOfSpeechDetails. Use the addPartOfSpeech function before removing stop words and lemmatizing.

cleanedDocuments = addPartOfSpeechDetails(cleanedDocuments);

% Words like "a", "and", "to", and "the" (known as stop words) can add noise to data. Remove a list of stop words using the removeStopWords function. Use the removeStopWords function before using the normalizeWords function.

cleanedDocuments = removeStopWords(cleanedDocuments);
cleanedDocuments(1:10)

% Lemmatize the words using normalizeWords.

cleanedDocuments = normalizeWords(cleanedDocuments,'Style','lemma');
cleanedDocuments(1:10)

% Erase the punctuation from the documents.

cleanedDocuments = erasePunctuation(cleanedDocuments);
cleanedDocuments(1:10)

%
% Remove words with 2 or fewer characters, and words with 15 or greater characters.

cleanedDocuments = removeShortWords(cleanedDocuments,2);
cleanedDocuments = removeLongWords(cleanedDocuments,15);
cleanedDocuments(1:10)

%
%% Create Bag-of-Words Model
% Create a bag-of-words model.

cleanedBag = bagOfWords(cleanedDocuments)

% Remove words that do not appear more than two times in the bag-of-words model.

cleanedBag = removeInfrequentWords(cleanedBag,2)

% Some preprocessing steps such as removeInfrequentWords leaves empty documents in the bag-of-words model. To ensure that no empty documents remain in the bag-of-words model after preprocessing, use removeEmptyDocuments as the last step.
% Remove empty documents from the bag-of-words model and the corresponding labels from labels.

[cleanedBag,idx] = removeEmptyDocuments(cleanedBag);
labels(idx) = [];
cleanedBag

%
%% Create a Preprocessing Function
% It can be useful to create a function which performs preprocessing so you can prepare different collections of text data in the same way. For example, you can use a function so that you can preprocess new data using the same steps as the training data.
% Create a function which tokenizes and preprocesses the text data so it can be used for analysis. The function preprocessText, performs the following steps:
% Tokenize the text using tokenizedDocument.
% Remove a list of stop words (such as "and", "of", and "the") using removeStopWords.
% Lemmatize the words using normalizeWords.
% Erase punctuation using erasePunctuation.
% Remove words with 2 or fewer characters using removeShortWords.
% Remove words with 15 or more characters using removeLongWords.
% Use the example preprocessing function preprocessText to prepare the text data.

newText = "The sorting machine is making lots of loud noises.";
newDocuments = preprocessText(newText)

%
%% Compare with Raw Data
% Compare the preprocessed data with the raw data.

rawDocuments = tokenizedDocument(textData);
rawBag = bagOfWords(rawDocuments)

% Calculate the reduction in data.

numWordsCleaned = cleanedBag.NumWords;
numWordsRaw = rawBag.NumWords;
reduction = 1 - numWordsCleaned/numWordsRaw

% Compare the raw data and the cleaned data by visualizing the two bag-of-words models using word clouds.

figure
subplot(1,2,1)
wordcloud(rawBag);
title("Raw Data")
subplot(1,2,2)
wordcloud(cleanedBag);
title("Cleaned Data")

%
%% Preprocessing Function
% The function preprocessText, performs the following steps in order:
% Tokenize the text using tokenizedDocument.
% Remove a list of stop words (such as "and", "of", and "the") using removeStopWords.
% Lemmatize the words using normalizeWords.
% Erase punctuation using erasePunctuation.
% Remove words with 2 or fewer characters using removeShortWords.
% Remove words with 15 or more characters using removeLongWords.

function documents = preprocessText(textData)

% Tokenize the text.
documents = tokenizedDocument(textData);

% Remove a list of stop words then lemmatize the words. To improve
% lemmatization, first use addPartOfSpeechDetails.
documents = addPartOfSpeechDetails(documents);
documents = removeStopWords(documents);
documents = normalizeWords(documents,'Style','lemma');

% Erase punctuation.
documents = erasePunctuation(documents);

% Remove words with 2 or fewer characters, and words with 15 or more
% characters.
documents = removeShortWords(documents,2);
documents = removeLongWords(documents,15);

end

% Copyright 2018 The MathWorks, Inc.