<p>This project deals with applying various techniques from Natural Language Processing to the analysis and visualization of data in text form. We will use some of the most popular models: Word2Vec (CBOW, Skip-Gram) and FastText to process and explore text and further extract extremely crucial insights such as its word similarities and analogies. The objective is to understand how machine learning models capture relations between words.</p>

Understand how CBOW, Skip-Gram, and FastText models capture word meanings, visualize embeddings, and evaluate model performance for various NLP tasks.

Word2Vec and FastText Word Embedding with Gensim in Python

<h2>Project Overview</h2><p>This study uses Natural Language Processing (NLP) methods for text-document analysis via word embedding models. First, the text data is pre-processed, cleansing, tokenizing, and removing stopwords. Thereafter, three popular models- CBOW, Skip-Gram, and FastText-are trained on the dataset, creating vector representations capable of defining the relationship of words with each other.</p><p>Afterward, it will explore the following: word similarity, analogical reasoning- for instance, 'doctor + medicine' - hospital and outlier detection of groups of words. To better understand the performance of these models, we implement diminution techniques PCA and t-SNE to illustrate how devices work on arranging words in a 2D space. The project is geared towards comparing how different embeddings capture word meanings and relations: CBOW, Skip-Gram, and FastText.</p><h2>Prerequisites</h2><ul><li><strong>Python Programming</strong>: Basic knowledge of Python is relevant to this work.  </li><li><strong>NLP Basics</strong>: Familiarity with typologies like tokenization and stopword removal along with word embeddings would be helpful.  </li><li><strong>Machine Learning</strong>: Knowledge of training models and validating text data features will be useful.  </li><li><strong>Libraries</strong>: Knowledge of Python libraries like NumPy, Pandas, NLTK, Gensim, and scikit-learn is necessary.  </li><li><strong>Basic Linear Algebra</strong>: Understanding vectors, matrices, and cosine similarity is fundamental to making sense of word embeddings.</li></ul><h2>Approach</h2><p>It begins with the preprocessing of text data cleaning, tokenization, and stop word removal. Then three different word embedding models, CBOW, Skip-Gram, and FastText are trained using processed text to capture the relationship of words in a high-dimensional vector space. Comparing these models will be based on word similarity checking, a test of analogical reasoning—for example, 'doctor + medicine - hospital' as well as the detection of outliers within a group of words. Finally, dimensionality reduction techniques like PCA and t-SNE will be applied to make sense of these high-dimensional word vectors in a 2D space to visualize their relationships. Ultimately, all these will lead to comparing each model's performance in capturing semantic relationships between words, thereby giving insights into how different word embeddings represent language.</p><h3>Workflow</h3><ul><li><strong>Data Collection:</strong> Collect and load the dataset that contains text information (titles and abstracts').  </li><li><strong>Preprocessing</strong>: Clean, tokenize, and stop word removal.  </li><li><strong>Word Embedding Models</strong>: Build <strong>CBOW, Skip-Gram, and Fasttext-word</strong> embedding models over the processed text.  </li><li><strong>Evaluate Models:</strong> Evaluate word embedding through checking of similarity, analogy, and anomalies.  </li><li><strong>Dimensionality Reduction</strong>: Perform the PCA and t-SNE to visualize the word embeddings in 2D.   </li><li><strong>Visualization</strong>: Results will be visualized using Matplotlib and Plotly for better interpretation.   </li><li><strong>Analysis</strong>: Comparisons and analyses of how each model's performance would be based as a result of the embeddings it generated would be done.</li></ul><h3>Methodology</h3><ul><li><strong>Text Preprocessing</strong>: Use NLTK to clean and preprocess the text data (tokenization, stopword removal).   </li><li><strong>Word2Vec and FastText:</strong> Train CBOW, Skip-Gram, and FastText models using Gensim to produce word embeddings.   </li><li><strong>Similarity Metrics</strong>: Compute similarity between words by Cosine Similarity of Word Vectors.   </li><li><strong>Analogical Reasoning</strong>: Perform tasks of analogical reasoning such as doing "Doctor + medicine - Hospital"; to assess the models  </li><li><strong>Anomaly Detection:</strong> Use the doesn't match method to identify the outlier word in any group.   </li><li><strong>Dimensionality Reduction</strong>: Reduction of dimensional high word embedding using PCA and t-SNE for its visualization.  </li><li><strong>Visualization</strong>: Plots the word embeddings in 2D through scatter plots for comparison purposes between models visually.</li></ul><h2>Data Collection and Preparation</h2><p><strong>Data Collection:</strong><br />In this project, we collected the dataset from a public repository. If you are looking to work on a real-world problem, you can get these kinds of datasets from publicly available repositories such as Kaggle, UCI Machine Learning Repository, or company-specific data. We will provide the dataset in this project so that you can work on the same dataset.</p><p><strong>Data Preparation Workflow:</strong></p><ul><li><strong>Loading Dataset:</strong> The dataset will be imported which contains text data in the form of titles and abstracts.  </li><li><strong>Missing Values Checking</strong>: Missing Values can then be detected and treated in the dataset.  </li><li><strong>Select Relevant Columns:</strong> The text columns "title" and "abstract" are extracted for analysis.  </li><li><strong>Drop Missing Data</strong>: Remove all the rows where no values are present in the selected text columns.  </li><li><strong>Merge the Text:</strong> Now merge these two columns "title" and "abstract" into a single column for analysis.  </li><li><strong>Text Preprocessing:</strong> Cleaning and tokenizing the joint text: Lowercase, remove punctuation, and filter out stopwords.  </li><li><strong>Tokenization:</strong> The disintegration of the text into separate words or tokens to carry out subsequent processing.  </li><li><strong>Lemmatization:</strong> Lemmatization is the process of reducing the word to its base form; e.g., "running" becomes "run.".  </li><li><strong>Preparation of Modeling</strong>: The preprocessed text is stored in the appropriate format to train word embedding models.</li></ul>

<h2>Code Explanation</h2><h4>Mounting Google Drive</h4><p>First, mount Google Drive to access the data stored in the cloud.</p>


<h4>Library setup and installation</h4><p>This installs the required libraries, textblob for text processing, streamlit for web app development, and wordcloud for making word clouds. It also downloads the necessary corpora for textblob to work properly.</p>

<h4>Importing Libraries for NLP and Visualization</h4><p>This code imports libraries for text preprocessing, natural language processing (NLP), and machine learning. Some of the key tools being used are nltk to perform tokenization and lemmatization activities on the text corpus, gensim to obtain word embeddings, matplotlib and plotly for visualizing the data, and sklearn for machine learning activities like dimensionality reduction and feature extraction.</p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.5rem;">Loading Data</span></h3><p>This code loads the CSV file from the specified path</p>


<h4>Previewing Data</h4><p>This block of code displays the first 10 rows of the dataset to give a quick overview of its structure.</p>

<h4>Checking Null Values</h4><p>This code mainly checks whether the null data is present, which may be required to be treated before processing for further analysis.</p>

<h4>Dataset Overview</h4><p>The code summaries along with the number of entries and column names, data types, and memory usage of the dataset df. This is useful for easily understanding the topology and composition of the dataset.</p>

<h4>Inspecting Text Columns and Missing Values</h4><p>This piece of code selects Title and Abstract columns from the dataset, looking for possible missing values in these text-rich columns, and showing the initial few rows of them along with the missing counts to evaluate data quality.</p>

<h4>Dropping Rows with Null Values</h4><p>This code drops all rows from the text_columns Dataframe that miss data in either the Title or Abstract columns, producing a neat dataset to conduct further analyses.</p>

<h4>Checking Null Values</h4><p>This code checks whether the null data is present after removing rows from the dataset.</p>

<h4>Shape-checking of Data after Cleaning</h4><p>The code returns the shape of the text_columns_cleaned DataFrame in terms of dimension (number of rows and columns). It provides an understanding of how many entries exist after dropping rows with missing data in it.</p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.5rem;">Merged Title and Abstract</span></h3><p>This piece of code creates a new column, <strong>Combined_Text</strong>, by concatenation of columns Title and Abstract in text_columns_cleaned DataFrame.</p>


<h4>WordNetLemmatizer and Stopwords Initializer</h4><p>The code initializes a WordNetLemmatizer for lemmatization and loads a set of English stopwords using nltk. These will be used for text preprocessing, such as reducing words to their root forms and eliminating common insignificant terms.</p>

<h4>Text Preprocessing Function</h4><p>The function processes input in lower case, without punctuation and special characters, tokenizes into words, cleanses stopwords, and completes the lemmatization of each token. Returns the cleaned, processed tokens for further analysis.</p>

<h4>Preprocessing Combined Text</h4><p>The following code calls the preprocess_text function for the Combined_Text column of the <strong>text_columns_cleaned</strong> DataFrame and creates a new column <strong>Processed_Text</strong> that captures the cleaned and tokenized version of the original text. It then displays the first few rows of the original versus the transformed text for inspection.</p>

<h4>Word Cloud Generation and Display</h4><p>This code will combine them and convert the resulting string to a word cloud from the text using WordCloud. The image will then be displayed, visualizing again in terms of word frequency using larger words that represent greater frequency in the dataset.</p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.5rem;">Summary Statistics for Text Lengths</span></h3><p>The code computes and provides summary statistics on the counts of words in Title, Abstract, and Combined_Text columns. It employs describe() to get these insights on how the distribution of text lengths looks like for example average numbers of words, minimum and maximum lengths</p>


<h4>Display text statistics by length</h4><p>This code helps in analyzing the distribution of text lengths in each of those columns by showing parameters like mean, standard deviation, and range of word counts.</p>

<h4>Finding Most Common Words</h4><p>\This code flattens the processed token list, counts the fatness of each token using Counter, and gets the twenty most common words. It helps find the most bona fide common words existing in the dataset, post-preprocessing. </p>

<h4>Sentiment Analysis</h4><p>This code creates a function get_sentiment which will give the score of an input text using TextBlob. This function is then applied to the Combined_Text field of text_columns_cleaned DataFrame, resulting in a new Sentiment column with the corresponding scores for positive and negative polarities of the input text. Positive sentiment is scored positively while negative sentiment is scored negatively.</p>

<h4>Plotting Distribution of Sentiments</h4><p>This code plots the histogram for the sentiment polarity scores in the Sentiment column. This gives an understanding of how the entire sentiment (positive, negative, or neutral) in the dataset represents the frequency distribution of different sentiment scores.</p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.5rem;">Extracts n-grams</span></h3><p>This function extracts n-grams-from a corpus given in text form and uses CountVectorizer to get their frequency. It introduces the top-n most common ngrams for use in identifying frequent word combinations across the dataset.</p>


<h4>Extracting Top Bigrams and Trigrams</h4><p>This program extracts the top 20 bigrams (2-word combinations) and trigrams (3-word combinations) found in the Combined_Text column, using the get_ngrams function. Using it gives us the possibility of extracting common words in pairs and triplets that co-occur frequently into the dataset.</p>

<p>It will display the top 20 most frequent bigrams (2-word combinations) from the <strong>Combined_Text</strong> column. </p>

<p>It will display the top 20 most frequent trigrams (3-word combinations) from the <strong>Combined_Text</strong> column. </p>

<h4>List of sentences from Processed Text</h4><p>This code converts the column Processed_Text of the DataFrame text_columns_cleaned into a list of sentences, each being a tokenized and processed version of the text, which can be used further in analysis or modeling.</p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.5rem;">Train a CBOW word2vec model</span></h3><p>The code above uses processed sentences to train a Continuous Bag of Words (CBOW) Word2Vec model. The model is configured to have a vector size of 100, a window size of 5, and count words of at least 5. It trains itself over 4 worker threads and 10 epochs. The produced model is saved as word2vec_cbow.model.</p>


<h4>Skip-Gram Word2Vec Model Training</h4><p>This code is for training a Skip-Gram Word2Vec model using already processed sentences. Like the CBOW model, it employs vector size-100, window size-5 and minimum word count-5. It runs for 10 epochs and with 4-worker threads. The model will be saved into word2vec_skipgram.model for future use. The only difference here with the CBOW is the use of sg=1 to represent the Skip-Gram.</p>

<h4>Training FastText Model</h4><p>This code takes the input sentences as data to train a FastText model. The model is configured with a vector size of a hundred, a window size of 5, and a minimum word occurrence of 5. It runs at 10 epochs in a 4 worker thread mode. The resulting model is stored in a file named fasttext.model. In contrast to Word2Vec, FastText can create vectors for out-of-vocabulary words with the help of their subword information.</p>

<h4>Selecting Common Vocabulary and Extracting Vectors</h4><p>The code identifies the common vocabulary shared by these three models (CBOW, Skip-Gram, and FastText) by finding the intersection across all their vocabularies. The next step consists of selecting the first hundred words from this joint common vocabulary and extracting their word vectors from each model. The collected vectors are stored in cbow_vectors, skipgram_vectors, and fasttext_vectors. It then prints to visualize the number of selected words.</p>

<h4>Using PCA for Reduction in Dimensions</h4><p>This code implements PCA (Principal Component Analysis) with 2 components for the projected CBOW, Skip-Gram, and FastText word vectors into 2D space, taking into account the explained variance ratio that indicates how much of the total variance is captured in the first two principal components. It helps understand how effective dimensionality reduction is.</p>

<h4>Function to Plot Word Embeddings</h4><p>This function visualizes the word embeddings in a 2D space by plotting PCA-reduced word vectors. It accepts the embeddings (2D vectors), words, and titles as inputs and generates a scatter plot where each point represents each of the words. The function then labels each point by each corresponding word in addition to the axes' labels and grid lines to enhance clarity.</p>

<h4>Visualization of Word Embeddings</h4><p>The code uses the function '<strong>plot_embeddings</strong>' to visualize the word embeddings generated from the CBOW, Skip-Gram, and FastText models in 2D spatial coordinates after PCA dimensionality reduction. Each plot shows how the different models depict the relationships between the words chosen in 2D space</p>

<h4>Using t-SNE for Dimensionality Reduction</h4><p>The code below uses t-SNE (t-distributed stochastic neighbor embedding) with 2 components to reduce the CBOW, skip-gram, and fasttext word vectors in two dimensions. It can be defined as a technique that facilitates the visualization of high-dimensional data such that similar items are clustered. Thus, it is a method well suited for the observation of word embeddings.</p>

<h4>Visualization t-SNE of Word Embeddings.</h4><p>The code visualizes 2D plots of word embedding points from the CBOW, Skip-Gram, and FastText models using t-SNE. It then generates plots of the selected words to see how they work with each other. The plots will display how two or more words are grouped, allowing the comparison of separation by context across words through different models.</p>

<h4>Count Word Frequencies</h4><p>The Counter counts the number of each token after flattening the text tokenized and processed from the <strong>Processed_Text</strong> column in a single list of words. Thus, word_freq would be defined as a dictionary where the words are keys and their respective frequencies in the dataset are values.</p>

<h3><span style="color: inherit; font-family: inherit; font-size: 1.5rem;">The Evaluation Word Selection Code</span></h3><p>It takes the top 50 domain-related words from the top 50 most frequent words in the dataset and cross-matches them against the CBOW Word2Vec model's vocabulary. After that, it randomly chooses another 50 words among the top 1000 words available in the CBOW model. The two collections are finally combined into a list called evaluation_words, which has the domain-specific and random words combined. The evaluation or analysis of the two sets will proceed with this combined list.</p>


<h4>Generation of word pairs and word groups.</h4><p>This program produces five random word pairs from the evaluation_words list, creating pairs from adjacent words. This selects one of the words (test_word) in the list and then randomly chooses 4 other words to form a word_group. These can be used in analogy or similarity tests.</p>

<h4>Calculate Word Pairs' Pairwise Similarity</h4><p>This code calculates and prints the similarity between two words in a pair from word pairs using the CBOW, Skip-Gram, and FastText models. It uses the similarity method for each model to calculate the cosine similarity of the word vectors into which the two words are converted. The output is generated, describing how similar those models perceive the two words to be.</p>

<h4>Finding words most related to a specific word</h4><p>In this code, the 5 nearest words are found and printed and given to the random test_word using the CBOW, Skip-Gram, and FastText models. Each model has a method called most_similar, that returns the most similar words as per their vector created for each word. Thus, the output shall show how all models relate the test_word with other words in the vocabulary.</p>

<h4>Analogical Reasoning: Relationship of Words</h4><p>This code implements the analogical reasoning to find out what the third word is to complete the analogy "doctor + medicine - hospital". This is achieved through the most_similar method specifying positive and negative parameters with respect to each model (CBOW, Skip-Gram, and FastText). The output is printed as results for each model showing which word each model has predicted as the answer to the analogy.</p>

<h4>Outlier Detection</h4><p>The code implements outlying detection by determining which of the words does not belong to the word_group and using the doesnt_match method from each model (CBOW, Skip-Gram, and FastText). After calling the method, the models will return the word which does not match mostly with the others of the group. The output will show which odd word has been indicated by each model.</p>

<h2>Conclusion</h2><p>This project showcases how powerful word embedding models such as CBOW, Skip-Gram, and FastText can capture semantic relationships between words, through preprocessing, training, and evaluation tasks like word similarity, analogical reasoning, and outlier detection. Apart from this, we gain an insight into how different models represent word meanings at the end. Furthermore, visualizing word embeddings using PCA and t-SNE will contribute to developing a better understanding of spatial relations and the clustering behavior of words in the models. Finally, the project emphasizes the importance of embedding models, depending on the task involved, and provides the reader with a deeper perception of NLP techniques and their applications.</p><h2>Challenges New Coders Might Face</h2><ul><li><p><strong><em>Challenge</em></strong>:  <strong>Handling noisy or unstructured text data.</strong><br /><strong><em>Solution</em></strong>: Utilize text cleaning methods, which may include the exclusion of special symbols, figures, and extra spaces.  </p></li><li><p><strong><em>Challenge</em></strong>:  <strong>Preprocessing Large Text Data</strong><br /><strong>*Solution***</strong>:** Enhance text cleaning processes by employing better libraries such as NLTK and adopting batch processing for the data.</p></li><li><p><strong><em>Challenge</em></strong>: <strong>Curse of Dimensionality in the high dimensional text datasets affecting clustering and classification results.</strong><br /><strong>*Solution***</strong>:** Use TF-IDF vectorization and reduction techniques like (PCA) to control dimensionality.  </p></li><li><p><strong><em>Challenge</em></strong>: <strong>Inaccessibility of GPU</strong><br /><strong><em>Solution</em></strong>: For debugging or initial testing, consider using smaller datasets or incorporating GPU-based cloud platforms for quick turnaround times.</p></li><li><p><strong><em>Challenge</em></strong>: <strong>Training Time is High due to the Large Vocabulary Size</strong><br /><strong><em>Solution</em></strong>: Use subsampling techniques or restrict vocabulary to use only the N most frequent words for efficiency purposes.</p></li></ul><h2>Frequently Asked Questions (FAQs)</h2><p><strong>Question 1: What is Word Embeddings?</strong><br /><strong>Answer</strong>: Word embedding is the vector representation of the words in a continuous and dense vector space in addition to capturing the semantic relationships between words, thereby enabling the models to practically understand meanings based on the context rather than individual words.</p><p><strong>Question 2: What is the difference between CBOW and Skip-Gram in Word2Vec?</strong><br /><strong>Answer</strong>: </p><ul><li>CBOW (Continuous Bag of Words): Predicts a target word from a context (surrounding words).  </li><li>Skip-Gram: Predicts surrounding words from a target word. Skip-Gram typically works better with smaller datasets.</li></ul><p><strong>Question 3: What is FastText, and how does it differ from Word2Vec?</strong><br /><strong>Answer:</strong> FastText uses bags of character n-grams and treats words as these bags: for this reason, it is capable of generating embeddings for out-of-vocabulary words. As a result, FastText does a better job at handling rare or unseen words.</p><p><strong>Question 4: What are the preprocessing stages of Skip-Gram projects?</strong><br /><strong>Answer:</strong> These include text cleaning and normalization, tokenization, and other processes to remove noise, such as special characters and numbers, for uniformity of the input data.</p><p><strong>Question 5: What libraries are needed to implement the Word embedding project?</strong><br /><strong>Answer:</strong> Building and training the model cannot be done without Pytorch whilst NLTK, NumPy, and Matplotlib are useful for preprocessing, and numerical and graphical manipulations respectively.</p>