%% Analyze Text Data Using Topic Models
% This example shows how to use the Latent Dirichlet Allocation (LDA) topic model to analyze text data.
% A Latent Dirichlet Allocation (LDA) model is a topic model which discovers underlying topics in a collection of documents and infers the word probabilities in topics.
%% Load and Extract Text Data
% Load the example data. The file factoryReports.csv contains factory reports, including a text description and categorical labels for each event.

data = readtable("factoryReports.csv",TextType="string");
head(data)

% Extract the text data from the field Description.

textData = data.Description;
textData(1:10)

%
%% Prepare Text Data for Analysis
% Create a function which tokenizes and preprocesses the text data so it can be used for analysis. The function preprocessText, listed in the Preprocessing Function section of the example, performs the following steps in order:
% Tokenize the text using tokenizedDocument.
% Lemmatize the words using normalizeWords.
% Erase punctuation using erasePunctuation.
% Remove a list of stop words (such as "and", "of", and "the") using removeStopWords.
% Remove words with 2 or fewer characters using removeShortWords.
% Remove words with 15 or more characters using removeLongWords.
% Prepare the text data for analysis using the preprocessText function.

documents = preprocessText(textData);
documents(1:5)

% Create a bag-of-words model from the tokenized documents.

bag = bagOfWords(documents)

% Remove words from the bag-of-words model that have do not appear more than two times in total. Remove any documents containing no words from the bag-of-words model.

bag = removeInfrequentWords(bag,2);
bag = removeEmptyDocuments(bag)

%
%% Fit LDA Model
% Fit an LDA model with 7 topics. For an example showing how to choose the number of topics, see Choose Number of Topics for LDA Model. To suppress verbose output, set the Verbose option to 0. For reproducibility, use the rng function with the "default" option.

rng("default")
numTopics = 7;
mdl = fitlda(bag,numTopics,Verbose=0);

% If you have a large dataset, then the stochastic approximate variational Bayes solver is usually better suited as it can fit a good model in fewer passes of the data. The default solver for fitlda (collapsed Gibbs sampling) can be more accurate at the cost of taking longer to run. To use stochastic approximate variational Bayes, set the Solver option to "savb". For an example showing how to compare LDA solvers, see Compare LDA Solvers.
%
%% Visualize Topics Using Word Clouds
% You can use word clouds to view the words with the highest probabilities in each topic. Visualize the topics using word clouds.

figure
t = tiledlayout("flow");
title(t,"LDA Topics")

for i = 1:numTopics
    nexttile
    wordcloud(mdl,i);
    title("Topic " + i)
end

%
%% View Mixtures of Topics in Documents
% Create an array of tokenized documents for a set of previously unseen documents using the same preprocessing function as the training data.

str = [
    "Coolant is pooling underneath assembler."
    "Sorter blows fuses at start up."
    "There are some very loud rattling sounds coming from the assembler."];

newDocuments = preprocessText(str);

% Use the transform function to transform the documents into vectors of topic probabilities. Note that for very short documents, the topic mixtures may not be a strong representation of the document content.

topicMixtures = transform(mdl,newDocuments);

% Plot the document topic probabilities of the first document in a bar chart. To label the topics, use the top three words of the corresponding topic.

for i = 1:numTopics
    top = topkwords(mdl,3,i);
    topWords(i) = join(top.Word,", ");
end

figure
bar(topicMixtures(1,:))

xlabel("Topic")
xticklabels(topWords);
ylabel("Probability")
title("Document Topic Probabilities")

% Visualize multiple topic mixtures using stacked bar charts. Visualize the topic mixtures of the documents.

figure
barh(topicMixtures,"stacked")
xlim([0 1])

title("Topic Mixtures")
xlabel("Topic Probability")
ylabel("Document")

legend(topWords, ...
    Location="southoutside", ...
    NumColumns=2)

%
%% Preprocessing Function
% The function preprocessText, performs the following steps in order:
% Tokenize the text using tokenizedDocument.
% Lemmatize the words using normalizeWords.
% Erase punctuation using erasePunctuation.
% Remove a list of stop words (such as "and", "of", and "the") using removeStopWords.
% Remove words with 2 or fewer characters using removeShortWords.
% Remove words with 15 or more characters using removeLongWords.

function documents = preprocessText(textData)

% Tokenize the text.
documents = tokenizedDocument(textData);

% Lemmatize the words.
documents = addPartOfSpeechDetails(documents);
documents = normalizeWords(documents,Style="lemma");

% Erase punctuation.
documents = erasePunctuation(documents);

% Remove a list of stop words.
documents = removeStopWords(documents);

% Remove words with 2 or fewer characters, and words with 15 or greater
% characters.
documents = removeShortWords(documents,2);
documents = removeLongWords(documents,15);

end

% Copyright 2018 The MathWorks, Inc.