<p>This project deals with human action recognition from images through deep learning models. We use datasets of annotated images that show various human interactions, such as sitting, standing, laughing and etc. The main objective is to classify these images into predefined action classes. Several state-of-the-art models, like ResNet50 and InceptionV3, are used to predict highly accurate results.</p>

Learn how to classify human actions from images using deep learning models like ResNet50 and InceptionV3 for security, healthcare and smart home applications.

Human Action Recognition Using Image Preprocessing

<h2>Project Overview</h2><p>This project focuses on deep learning modules to develop a human action recognition system. Thus, actions like sitting, standing, walking or laughing are distinct categories that will serve as inputs for the classification of images. The dataset contains images capturing different human activities tagged with their corresponding categories.</p><p>The data preprocessing stage begins with resizing all images to 160 x 160 pixels, normalization of pixel values and contrast enhancement if required. The stages that follow include the conversion of categorical labels to numerical format via LabelEncoder and one hot encoding of the labels to prepare them for the model.</p><p>We have the architecture, composed of fine-tuned leads of supremely potent pre-trained deep learning models such as ResNet50 and InceptionV3, to our specific task. During training, early stopping is utilized with the best-selected model based on validation performance. However, during the training process itself, accuracy and loss are monitored.</p><p>To evaluate the performance of the trained models, we test them on test data in which we predict labels and compute accuracy. We also produce confusion matrices to visualize class performances. What is achieved in the end is a robust action recognition system that could classify human activity from images accurately.</p><h2>Prerequisite</h2><ul><li>Programming Basics in Python and Data Manipulation Techniques.  </li><li>Basic Knowledge of Machine Learning and Deep Learning.  </li><li>Basics about Images Preprocessing-Resizing, Normalizing.  </li><li>Basics of keras and tensorflow to build deep learning models.  </li><li>Experience working with Jupyter Notebooks, if not Google Colab.   </li><li>Data Visualization with Matplotlib and Plotly.  </li><li>The important evaluation metrics of the model are Accuracy and Confusion Matrix.  </li><li>Knows how to work with models like ResNet50 and InceptionV3 that are already trained.</li></ul><h2>Approach</h2><p>The approach starts with preprocessing the image data, specifically resizing and normalizing the images thus ensuring uniformity throughout the data. The categorical action labels are first subjected to label encoding which is eventually followed by one hot encoding to make it compatible with the model. For the model, we use powerful pre-trained architectures such as ResNet50 and InceptionV3 that are capable of learning complex features given an image, then fine-tuning them with the training data to improve accuracy for the task at hand which is human action recognition. We also utilize early stopping during training, to avoid the occurrence of overfitting, while progress is monitored using accuracy and loss metrics. The model's efficiency is evaluated against testing data after successful training by making use of accuracy scores and a confusion matrix to visualize how well the model classifies the various human actions.</p><h2>Workflow and Methodology</h2><h3>Workflow</h3><ul><li>Data Collection and Organization: Collect and organize the dataset of images with corresponding action labels.  </li><li>Preprocessing of Data: Resize the images to 160 x 160 pixels and normalize pixel values.  </li><li>Encoding Labels: Convert the action labels to numbers with LabelEncoder and use a one-hot encoding.  </li><li>Model construction: Train deep learning models based on pre-trained architectures, for example, ResNet50 and InceptionV3.  </li><li>Model training: Train models on the preprocessed dataset using early stopping to avoid overfitting.  </li><li>Model Evaluation: Evaluate model performance using accuracy and confusion matrices.  </li><li>Visualization of Results: Visualizing the results through graphs and performance metrics.</li></ul><h3>Methodology</h3><ul><li>Transfer learning has been used with pre-trained models such as ResNet50 and InceptionV3 for the learned features by ImageNet.  </li><li>Fine-tune these models with the human action recognition data and obtain increases in accuracy for specific action recognition tasks.  </li><li>Process the images in resizing, normalizing and other data preparation steps.  </li><li>Convert these class labels into a machine-readable format using LabelEncoder and one-hot encoding.  </li><li>Divide the data into training and validation data sets where early stopping occurs during training to avoid overfitting.  </li><li>Finally, evaluate the model using performance metrics like accuracy and confusion matrix for complete assessment.</li></ul><h2>Data collection</h2><p><a href="https://www.kaggle.com/datasets/meetnagadia/human-action-recognition-har-dataset" target="_blank"><strong>Human action</strong></a> dataset is available in Kaggle. It is possible to conveniently and securely access a Kaggle dataset from within Google Colab after configuring your Kaggle credentials to prevent compromising sensitive information. It brings in the user’s data to collect securely the Kaggle API key and username and assigns them as environment variables. This enables the use of Kaggle’s CLI command (!kaggle datasets download -d  meetnagadia/human-action-recognition-har-dataset) which authenticates the user and downloads the dataset straight into Colab. </p><h2>Data preparation workflow</h2><ul><li>Load the dataset from source folders or CSV files with image paths and labels.  </li><li>Resize all images to a consistent shape (e.g., 160x160 pixels).  </li><li>Normalize pixel values to the range [0, 1].  </li><li>Convert categorical labels into numeric values using LabelEncoder.  </li><li>Apply one-hot encoding to the numeric labels.  </li><li>Organize data into batches for model training.</li></ul>

<h2>Code Explanation</h2><h3><span style="color: inherit; font-family: inherit; font-size: 1.5rem;">Mount Google Drive</span></h3><p>Mount your Google Drive to access and save datasets, models and other resources.</p>


<p>Import Libraries for Image Processing and Modeling</p><p>This code imports libraries such as OpenCV, NumPy and Matplotlib for image processing; machine learning libraries such as pandas and Scikit-learn and even TensorFlow/Keras for deep learning model building. In addition, the code imports several pre-trained models such as ResNet50, VGG16, InceptionV3 and other layers that form part of creating and training neural networks.</p>

<h4>Check out the dataset folder structure.</h4><p>This code sets the path for the dataset folder and lists the files and subdirectories within it. It helps one understand the dataset's folder structure for any further processing it might undergo.</p>

<h4>Import and Preview Training Data</h4><p>The code imports training data from a CSV file to pandas DataFrame objects and shows the first few rows in it. This gives an idea about the dataset's structure and allows an overview of the data.</p>

<h4>Import and Preview Test Data</h4><p>The code imports test data from a CSV file to pandas DataFrame objects and shows the first few rows in it. This gives an idea about the test dataset's structure and allows an overview of the data</p>

<h4>Visualizes the Label Distribution.</h4><p>This uses Plotly to make a pie chart that visualizes how different human activity labels in the training dataset are distributed. In particular, this allows an understanding of the balance of classes in the dataset.</p>

<h4>Display Random Image with Label</h4><p>Selects a random image from a training dataset, loads and displays with the assigned label. If the image isn’t found, it just prints a message and skips that file.</p>

<h4>Loading and Preprocessing Images</h4><p>This code applies iterating through the training dataset files; loading images, resizing them to 160 by 160, normalizing the pixel quality values and aggregating the data and labels as images into two separate lists. This step prepares the training data for model training.</p>

<h4>Transforming image data and labels to arrays</h4><p>This code includes the image data list (img_data) and labels (img_label) as NumPy arrays. The shape of the X array will be (num_samples, 160, 160, 3); conversely, the y array will shape it to (num_samples,) for labels.</p>

<h4>One-Hot Encoding the Labels</h4><p>This code uses LabelEncoder to encode string labels to numerical values and then applies to_categorical to convert these encoded labels into one-hot vectors. The labels are encoded in preparation for the training of a neural network model.</p>

<h4>Building a Model with ResNet50</h4><p>It loads the ResNet50 model with the pre-trained ImageNet weights without the top classification layers. Then it passes the base ResNet50 model to a custom build_model function, which creates a complete model fitting for the task.</p>

<h4>Make early stopping for model training</h4><p>This code defines the early stopping conditions on the validation loss while the training continues. In case the validation loss does not improve for five continuous epochs, training stops and the best weights of the model are restored. </p>

<p>ResNet50 Model Training</p><p>This program has been designed to train a model based on ResNet50 architecture using already prepared image data (X) and one-hot encoded target labels (y_one_hot). It will run for 20 epochs at a batch size of 32 on the training set, validating it with 20% of the dataset. The early stopping callback monitors validation loss to prevent overfitting.</p>

<p>InceptionV3 Model Building</p><p>The code first loads the InceptionV3 model along with its pre-trained ImageNet weights but without its top layers for classification. Then, it gets the base model of InceptionV3 passed to a custom build_model function to provide a complete model for the task.</p>

<p>Inception Model Training</p><p>This program has been designed to train a model based on InceptionV3 architecture using already prepared image data (X) and one-hot encoded target labels (y_one_hot). It will run for 20 epochs at a batch size of 32 on the training set, validating it with 20% of the dataset. The early stopping callback monitors validation loss to prevent overfitting.</p>

<h4>Accuracy and Loss Plotting</h4><p>This function plots the training and validating graphs of accuracy and loss. This helps in understanding how both the model performs during training and how it is doing in catching trends of overfitting or underfitting.</p>

<h4>Visualizing ResNet50 Training Results</h4><p>This code calls the plot_history function to display the accuracy and loss graphs for the ResNet50 model. It helps assess the model's performance throughout the training process.</p>

<p>Visualizing InceptionV3 Training Results</p><p>This code calls the plot_history function to display the accuracy and loss graphs for the InceptionV3 model. It helps assess the model's performance throughout the training process.</p>

<h4>Preprocess the Images for Testing</h4><p>This function resizes test images into 160x160 pixels, normalizes pixel values and contrast enhances images that require it. With that, it takes care that the test images are prepared in the same manner as the training data to evaluate the model.</p>

<h4>Predict and show an image along with labels</h4><p>This function preprocesses an image for testing and predicts the output using the given model, displaying the image with real and predicted labels. The function is designed to visualize and compare the predictions of the model with the truth.</p>

<h4>Predicting on a Single Image</h4><p>This code will define the path to a test image and the respective actual label. It will also create a class labels list (which would typically come from the training data but can also be manually defined). This prepares the data to call the prediction function and display the results.</p>

<h4>Predicting and Displaying the Test Image Result</h4><p>This code calls the predict_and_display function to predict the label for a specific test image using the ResNet model.</p>

<h4>Predicting and Displaying the Test Image Result</h4><p>This code calls the predict_and_display function to predict the label for a specific test image using the Inception V3 model.</p>

<h4>Evaluation of a Model Performance on Accuracy and Confusion Matrix</h4><p>Make predictions using the ResNet model from test data and convert predicted probabilities to class labels. Then, decode the labels back to their original string values and calculate the accuracy. A confusion matrix is displayed for visualizing the model's performance.</p>

<h4>Evaluation of a Model Performance on Accuracy and Confusion Matrix</h4><p>Make predictions using the Inception model from test data and convert predicted probabilities to class labels. Then, decode the labels back to their original string values and calculate the accuracy. A confusion matrix is displayed for visualizing the model's performance.</p>

<h2>Conclusion</h2><p>A successful human action recognition system was created in this project using deep learning models such as ResNet50 and InceptionV3. Following the data preprocessing, model training and evaluation, we obtained a resilient approach to classify human actions from images. We improved performance and accuracy by using pre-trained models and fine-tuning them to our specific needs. The results, in terms of accuracy scores and confusion matrices, show the model's ability to identify different human activities with a high degree of precision. The system can be expanded and employed in many applications such as in real life, for example, in surveillance, robotics and interactive gaming.</p><h2>Challenges New Coders Might Face</h2><ul><li><p><strong>Challenge</strong>: <strong>Large Datasets</strong><br /><strong>Solution:</strong> Processing very large collections of images takes up a lot of resources. It is possible to implement batch processing and use GPUs to speed up the extraction of embeddings.</p></li><li><p><strong>Challenge</strong>:  <strong>Resources for Computing</strong><br /><strong>Solution:</strong> The running of computation models especially those like Inception and Resnet requires heavy computation, however, this can be accomplished through cloud platforms with GPU support like Google Colab.  </p></li><li><p><strong>Challenge: Quality and Labeling of Data</strong><br /><strong>Solution:</strong> Inconsistent faces, such as expressions, angles, or light conditions at the time of acquisition, affect different embeddings. Augmentation-related operations like rotations, flipping, lighting adjustments, etc., all can help in creating strong embeddings.  </p></li><li><p><strong>Challenge: Data Imbalance</strong><br /><strong>Solution: Solution</strong>: Make use of data augmentation strategies such as rotation, flipping and zooming to increase the number of images of a few class samples  </p></li><li><p><strong>Challenge: Overfitting</strong><br /><strong>Solution:</strong> Incorporate regularization techniques such as Dropout layers that enable the removal of some neurons at random during training, whereby training may be stopped when the validation ceases to improve.</p></li></ul><h2>FAQ</h2><p><strong>Question 1: What does human action recognition mean?</strong><br /><strong>Answer</strong>: The task known as human action recognition involves identifying an action, for instance walking, sitting, or laughing, performed by a human using an image or video and then classifying it using machine learning or deep learning models.</p><p><strong>Question 2:  How can I prepare my images before feeding them into an action recognizer model?</strong><br /><strong>Answer</strong>: You need to resize all the images to a common dimension, say 160x160 pixels, normalize the pixel values, one-hot encode the labels and maybe use data augmentation to enhance the performance of the model.</p><p><strong>Question 3:Which deep learning models are suitable for human action recognition?</strong><br /><strong>Answer</strong>: Some common models include ResNet50, InceptionV3 and several other pre-trained architectures. However, the models are customized to be fine-tuned for an action recognition task according to your dataset.</p><p><strong>Question 4: How to increase accuracy in action recognition?</strong><br /><strong>Answer:</strong> Data augmentation, fine-tuning pre-trained models, and using procedures like early stopping to avoid overfitting: can all help bring about increased accuracies. Also, trying various architectures for the models and modifying hyperparameters might do the trick.</p><p><strong>Question 5: What is the role of data augmentation in image segmentation?</strong><br /><strong>Answer</strong>: Early stopping is the early termination of training once validation loss ceases to improve so that the model does not overfit the training data. This reduces training time and ensures a better generalization in models trained using it.</p>