%% Analyze German Text Data
% This example shows how to import, prepare, and analyze German text data using a topic model.
% German text data can be large and can contain lots of noise that negatively affects statistical analysis. For example, the text data can contain the following:
% Variations in word forms. For example, „rot“, „rote“, and „roten“.
% Words that add noise. For example, stop words such as „der“, „die“, and „das“.
% Punctuation and special characters.
% These word clouds illustrate word frequency analysis applied to some raw text data and a preprocessed version of the same text data.
%
% This example first shows how to import and prepare German text data, and then it shows how to analyze the text data using a Latent Dirichlet Allocation (LDA) model. An LDA model is a topic model that discovers underlying topics in a collection of documents and infers the word probabilities in topics. Use these steps in preparing the text data and fitting the model:
% Import the text data from a CSV file and extract the relevant data.
% Prepare the text data for analysis using standard preprocessing techniques.
% Fit a topic model and visualize the results.
%
%% Import Data
% Download the data vorhaben.csv from https://opendata.bonn.de/dataset/vorhabenliste-b%C3%BCrgerbeteiligungen-planungen-und-projekte. This file can change over time, so the results in the example can vary.
% Use detectImportOptions to determine the format of the CSV file and set the text type to string. Set the 'Encoding' option to  'ISO-8859-15'. Read the data using the readtable function and view the first few rows.

filename = "vorhaben.csv";
options = detectImportOptions(filename,'TextType','string','Encoding','ISO-8859-15');
data = readtable(filename,options);
head(data)

% Extract the text data from the variable InhaltlicheBeschreibungUndZielsetzung (the description of the content and the goal).

textData = data.InhaltlicheBeschreibungUndZielsetzung;

% Visualize the text data in a word cloud.

figure
wordcloud(textData);

%
%% Tokenize Text Data
% Create an array of tokenized documents using the tokenizedDocument function.

documents = tokenizedDocument(textData);
documents(1:10)

%
%% Get Part-of-Speech Tags
% Add the part of speech details using the addPartOfSpeechDetails function.

documents = addPartOfSpeechDetails(documents);

% Get the token details and then view the details of the first few tokens.

tdetails = tokenDetails(documents);
head(tdetails)

% The PartOfSpeech variable in the table contains the part-of-speech tags of the tokens. Create word clouds of all the nouns and adjectives, respectively.

figure
idx = tdetails.PartOfSpeech == "noun";
tokens = tdetails.Token(idx);
subplot(1,2,1)
wordcloud(tokens);
title("Nouns")

idx = tdetails.PartOfSpeech == "adjective";
tokens = tdetails.Token(idx);
subplot(1,2,2)
wordcloud(tokens);
title("Adjectives")

%
%% Prepare Text Data for Analysis
% Tokenize the text using tokenizedDocument and view the first few documents.

documentsRaw = tokenizedDocument(textData);
documents = documentsRaw;
documents(1:10)

% Replace common phrases (n-grams) with a single token and remove the stop words.

old = ["Bad" "Godesberg"];
new = "Bad Godesberg";
documents = replaceNgrams(documents,old,new);
documents = removeStopWords(documents);
documents(1:10)

% Normalize the text using the normalizeWords function.

documents = normalizeWords(documents);
documents(1:10)

% Erase the punctuation using the erasePunctuation function.

documents = erasePunctuation(documents);
documents(1:10)

% Visualize the raw and cleaned data in word clouds.

figure
subplot(1,2,1)
wordcloud(documentsRaw);
title("Raw Data")

subplot(1,2,2)
wordcloud(documents);
title("Cleaned Data")

%
%% Create Preprocessing Function
% Creating a function that performs preprocessing can be useful to prepare different collections of text data in the same way. For example, you can use a function to preprocess new data using the same steps as the training data.
% Create a function which tokenizes and preprocesses the text data to use for analysis. The function preprocessGermanText, listed at the end of the example, performs these steps:
% Tokenize the text using tokenizedDocument.
% Replace the multiword phrase ["Bad" "Godesberg"] with "Bad Godesberg".
% Remove a list of stop words (such as „der“, „die“, and „das“) using removeStopWords.
% Normalize the words using normalizeWords.
% Erase punctuation using erasePunctuation.
% Remove the empty documents after preprocessing using the removeEmptyDocuments function. Removing documents after using a preprocessing function makes it easier to remove corresponding data such as labels from other sources.
% In this example, use the preprocessing function preprocessGermanText, listed at the end of the example, to prepare the text data.

documents = preprocessGermanText(textData);
documents(1:5)

% Remove the empty documents using the removeEmptyDocuments function.

documents = removeEmptyDocuments(documents);

%
%% Fit Topic Model
% Fit a latent Dirichlet allocation (LDA) topic model to the data. An LDA model discovers underlying topics in a collection of documents and infers word probabilities in topics.
% To fit an LDA model to the data, you first must create a bag-of-words model. A bag-of-words model (also known as a term-frequency counter) records the number of times that words appear in each document of a collection. Create a bag-of-words model using bagOfWords.

bag = bagOfWords(documents);

% Remove the empty documents from the bag-of-words model.

bag = removeEmptyDocuments(bag);

% Fit an LDA model with seven topics using fitlda. To suppress the verbose output, set 'Verbose' to 0.

numTopics = 7;
mdl = fitlda(bag,numTopics,'Verbose',0);

% Visualize the first four topics using word clouds.

figure
for i = 1:4
    subplot(2,2,i)
    wordcloud(mdl,i);
    title("Topic " + i)
end

% Visualize multiple topic mixtures using stacked bar charts. View five input documents at random and visualize the corresponding topic mixtures.

numDocuments = numel(documents);
idx = randperm(numDocuments,5);
documents(idx)
topicMixtures = transform(mdl,documents(idx));

figure
barh(topicMixtures(1:5,:),'stacked')
xlim([0 1])
title("Topic Mixtures")
xlabel("Topic Probability")
ylabel("Document")
legend("Topic " + string(1:numTopics),'Location','northeastoutside')

%
%% Example Preprocessing Function
% The function preprocessGermanText, performs these steps:
% Tokenize the text using tokenizedDocument.
% Replace the multiword phrase ["Bad" "Godesberg"] with "Bad Godesberg".
% Remove a list of stop words (such as „der“, „die“, and „das“) using removeStopWords.
% Normalize the words using normalizeWords.
% Erase punctuation using erasePunctuation.

function documents = preprocessGermanText(textData)

% Tokenize the text.
documents = tokenizedDocument(textData);

% Replace multiword phrases
old = ["Bad" "Godesberg"];
new = "Bad Godesberg";
documents = replaceNgrams(documents,old,new);

% Remove a list of stop words.
documents = removeStopWords(documents);

% Normalize the words.
documents = normalizeWords(documents);

% Erase the punctuation.
documents = erasePunctuation(documents);

end

% Copyright 2018 The MathWorks, Inc.