Projects | Ukant Jadia

A Beginner's Guide to DevOps with Terraform and GitHub

Mon, 30 Dec 2024 14:15:59 +0530

A Beginner’s Guide to DevOps with Terraform and GitHub

As a fresh graduate in Computer Science and Engineering, my exploration into the world of DevOps has been nothing short of an enlightening adventure. Coming from a machine learning background, I often found myself compelled to understand how backend systems could be crafted with such elegance. This curiosity fueled my capstone project, where I aimed to create a CI/CD pipeline that elegantly demonstrated the power of tools like Terraform and GitHub.

What is DevOps and Why is it Important?

DevOps blends development and operations, aiming to minimize the distance between these two traditionally siloed groups. The goal is to improve collaboration, increase deployment frequency, and deliver more stable operating environments. The DevOps culture promotes continuous integration and continuous delivery (CI/CD), which significantly enhances the software development lifecycle.

Key Components of DevOps

Continuous Integration (CI): Developers frequently integrate their changes in a shared repository.
Continuous Delivery (CD): Automating the delivery of applications to selected infrastructure environments.
Infrastructure as Code (IaC): Managing infrastructure through code, ensuring that setups are easy to replicate.

A Personal Exploration into DevOps

Reflecting on my academic journey, the decision to delve into a DevOps project stemmed from a fundamental question: How can vast complexities be integrated seamlessly? My project aimed to demystify this by constructing a basic CI/CD pipeline orchestrated through Terraform and GitHub.

Setting the Stage: Understanding My Project

The core of my project revolved around leveraging Terraform for infrastructure automation, along with GitHub for version control and CI/CD capabilities. This combination would provide a straightforward yet powerful example of deploying applications in a cloud environment.

Project Objectives

Demonstrate CI/CD using GitHub Actions: Show how automatic deployment can facilitate rapid product releases.
Use Terraform for Infrastructure Management: Illustrate how infrastructure can be provisioned and managed as code.
Visualize Workflows: Employ tools like Eraser.io to create flowcharts to help visualize the project’s structure.

Getting Started with Terraform

When I first explored Terraform, I was struck by how declarative it is. Terraform allows you to define your infrastructure in configuration files which can then be executed to provision that infrastructure automatically.

Example Terraform Configuration

Here’s a simple example of a Terraform configuration file that provisions an AWS EC2 instance:

provider ""aws"" {
 region = ""us-west-2""
}

resource ""aws_instance"" ""my_instance"" {
 ami = ""ami-0c55b159cbfafe1f0""
 instance_type = ""t2.micro""

 tags = {
 Name = ""MyInstance""
 }
}

Key Features of Terraform

Infrastructure as Code: Treat infrastructure setup as code, making it easier to maintain and version control.
Execution Plan: Terraform generates a clear execution plan which shows what will happen before any changes are applied.

Integrating GitHub Actions into the Pipeline

GitHub Actions provides a robust platform for automating workflows directly in your GitHub repository. For my project, I set up actions to automatically test and deploy the application upon code commit.

GitHub Actions Workflow Configuration

This YAML file illustrates a basic GitHub Actions workflow:

name: CI/CD Pipeline

on:
 push:
 branches: [ main ]

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - name: Check out the code
 uses: actions/checkout@v2

 - name: Set up Node.js
 uses: actions/setup-node@v2
 with:
 node-version: '14'

 - name: Install dependencies
 run: npm install

 - name: Run tests
 run: npm test

 - name: Deploy to production
 run: npm run deploy

Benefits of Using GitHub Actions

Seamless Integration: Works directly with your repositories.
Customizability: Create tailored workflows to fit specific project needs.

Visualizing the Workflow

To visualize how data flows through the different stages of the pipeline, I employed Eraser.io. This tool allowed me to create flowcharts that represented the overall CI/CD process from code push to deployment.

Figure 1: The flow of code and data in the CI/CD pipeline

The Elegance of CI/CD: A Real-World Application

As I progressed through my project, witnessing the pipeline’s operation became one of the most rewarding aspects. It was astounding to see how a simple workflow could articulate the path of requests or data between containers and servers.

Key Takeaways from the CI/CD Pipeline Experience

Visual Learning: The flowcharts provided invaluable insight into how requests and data moved, making the whole process comprehensible.
Hands-on with Real Tools: Working directly with GitHub and Terraform solidified my understanding of cloud infrastructure and automation.

Final Thoughts

Reflecting on this project, I can confidently say that building a CI/CD pipeline using Terraform and GitHub not only quenched my curiosity about backend processes but also equipped me with practical skills for my career ahead. This journey reaffirmed my belief that technology can create elegant solutions to complex problems. I hope my exploration encourages others to dive into the DevOps ocean, where every wave brings new learning opportunities.

References

Feel free to explore these resources further to expand your knowledge and understanding of DevOps and CI/CD pipelines.

ECG-Lense: Cardiac ECG Classification (IEEE Published)

Thu, 01 Aug 2024 00:00:00 +0000

Published at ETNCC 2024 · IEEE · DOI: 10.1109/ETNCC63262.2024.10767459

Headline numbers

80% classification accuracy
90% ROC-AUC, the best of all six models tested
21,837 ECG records · 18,885 patients · 12-lead · PTB-XL dataset

What it is

ECG-Lense is the deep CNN architecture I built and benchmarked against five other models for classifying cardiac ECG signals. Three classical ML, three deep learning, head-to-head on the same dataset and same metrics. ECG-Lense wins.

Model	Type
Decision Tree	Classical
Random Forest	Classical
Logistic Regression	Classical
Simple CNN	Deep
LSTM	Deep
ECG-Lense (Complex CNN)	Deep (proposed)

What I actually did

Worked off the PTB-XL dataset: clinical 12-lead ECGs, 10 seconds each, sampled at 100 Hz, with hierarchical diagnostic labels (5 superclasses, 24 subclasses)
Preprocessed signals with wfdb and standard ML libraries
Implemented a stationary wavelet transform (SWT) for data augmentation. The dataset is imbalanced and the augmentation increased sample diversity while preserving signal characteristics
Trained the deep models on raw ECG signals so feature extraction would be automatic
Evaluated on accuracy, precision, recall, F1, and ROC-AUC

What broke and what I learned

Class imbalance was the actual problem. First CNN runs hit a wall around 70% accuracy because rare cardiac classes were under-represented. SWT-based augmentation unlocked the next 10 points.
Raw signals beat hand-crafted features. Classical ML on extracted features stalled. The deep models, given raw signals, found patterns the feature engineering had thrown away.
More layers wasn’t the answer. ECG-Lense is “complex” relative to the simple CNN baseline, but each block earns its place. Depth for depth’s sake didn’t help.

Six months of work. A+ thesis grade. Submitted to IEEE during the project, accepted, presented and published at ETNCC 2024.

Stack

Python · TensorFlow · Keras · scikit-learn · MLFlow · wfdb · NumPy · pandas

Acknowledgements

Mentor: Dr. Kamal Kant Hiran (PhD). He pushed me to write the paper, guided the work end-to-end. Supervisors: Alok Kumar, Dr. Chandani Joshi, Harsh Bansal.

Dataset images

The dataset comprises 21,837 clinical 12-lead ECG records of 10 seconds in length from 18,885 patients. Hierarchical diagnostic organization: 5 superclasses, 24 sub-classes.
Metadata in ptbxl_database.csv: each ECG identified by a unique ecg_id with SCP codes for multi-label classification.

Demographic overview of patients in PTB-XL.

The dataset comprises 21837 clinical 12-lead ECG records of 10 seconds in length from 18885 patients. For the diagnostic labels, it provides a hierarchical organization in terms of 5 coarse superclasses and 24 sub-classes for the diagnostic labels
Columns provided in the metadata table ptbxl_database.csv. Each ECG is identified by a unique ID (ecg_id) and comes with several ECG statements (scp_codes) that can be used to train a multi-label classifier that can be evaluated based on the proposed fold assignments (strat_fold).

Demographic overview of patients in PTB-XL.

Super Market Data - A In-Depth EDA

Fri, 01 Mar 2024 00:00:00 +0000

Retail-Data-Analysis

Problem Statement

In today’s tough business world, companies need useful insights to grow and make money. But without data analysis, making decisions can be hard. This project aims to look at sales and profit data to find trends and chances for improvement. Using these insights, businesses can improve strategies and lead the market.

Project Description

Identify the region and city with the highest sales. Link
Ship Mode’s Impact on Profitability and Sales. Link
Analyze the category and subcategory with the highest profit. Link
Identify the segment with the highest profit and sales. Link

Images

Food Image Classification

Mon, 01 Jan 2024 00:00:00 +0000

This project tackles the challenge of applying computer vision to emphasize accurate food image classification. It compares Convolutional Neural Networks (CNN) with Traditional Machine Learning Models, leveraging CNN’s feature-rich capabilities and traditional models’ transparency and computational efficiency. This dual approach enhances food image classification’s precision and efficiency, impacting quality assessment, dietary analysis, and food inventory management

Used the 5K image dataset for classifying the food images and non-food images, Trained the neural network on the images for classification, and developed the model with streamlit application and achieved 80% accuracy. The dataset is taken from https://mmspg.epfl.ch/food-image-datasets and the name of the data set is 5K.

Dataset description: -. This dataset contains 2500 food and 2500 non-food images, for the task of food/non-food classification in our paper “Food/Non-food Image Classification and Food Categorization using Pre-Trained GoogLeNet Model”. The whole dataset is divided into three parts: training, validation and evaluation. The naming convention is as follows: -. {ClassID}_{ImageID}.jpg -. ClassID: 0 or 1; 0 means non-food and 1 means food. -. ImageID: ID of the image within the class

OBE Module for University

Mon, 01 Jan 2024 00:00:00 +0000

This was a team project.

The aim of this project is to provide the university with a web portal for the implementation of Outcome-Based Education (OBE).

OBE can primarily be distinguished from traditional education methods by the way it incorporates three elements: a theory of education, a systematic structure for education, and a specific approach to instructional practice. It organizes the entire educational system towards what is considered essential for learners to successfully achieve at the end of their learning experiences. In this model, the term ‘outcome’ is the core concept and is sometimes used interchangeably with the terms ‘competency,’ ‘standards,’ ‘benchmarks,’ and ‘attainment targets.’ OBE also utilizes the same methodology formally and informally adopted in the actual workplace to achieve outcomes.

The web portal is built with the Laravel framework of PHP for the front end and utilizes MySQL for the database.

My Role: My role is to build the business logic, conduct literature reviews, generate reports, and suggest the next steps based on those reports. I also assist in building the database

Bloom's Taxonomy Level Detection

Wed, 13 Dec 2023 00:00:00 +0000

To know Bloom’s Taxonomy in Detail see this pdf This

This project aims to classify the given statement into accurate Bloom’s taxonomy levels out of six levels. This is an NLP-based project. This project is built with TF-IDF vectorizer and multinomial Naive Bayes techniques.

Bloom’s Taxonomy Levels: Bloom’s Taxonomy is a classification system that arranges learning objectives into six hierarchical levels, each representing a different cognitive skill.

The model is trained on 1800 augmented statements. Initially, I had 100 statements for each class, so a total of 600 statements for six classes. Then, this data is augmented three times using contextual word embedding augmentation, and the model used in the augmentation is BERT-based. An example of the augmentation methodology.

This project follows MLOps techniques well. I have divided this project into 6 different pipelines for easy implementation and scalability. I created a class in the pipeline for each class using different entities and components. In generating a configuration file from the user input in the YAML format.

In the end, I created a Docker image so it can be used by the Higher Education Institute(HEIs) for those who are using the OBE(Object Based Education) new curriculum technique. It just reduced the manual process of identifying each question and its root taxonomy level.

A MLOps Project- Wine Quality Prediction

Fri, 08 Dec 2023 00:00:00 +0000

This project demonstrates my ability to implement a highly scalable MLOPS project by following development techniques and project procedures. In this project, I divided the whole implementation part into six different pipelines and steps, which are as follows:

Data Ingestion: Collecting the data and setting it up for further use.

Data Validation: Checking the data type, format, and other conditions to see the validity and usability of the data.

Data Transformation: Performing preprocessing and splitting the data into test and train datasets.

Model Training: Selecting a suitable machine learning algorithm and creating a model by training.

Model Evaluation: Testing the data and verifying it with measurement/result metrics.

Model Prediction: Making predictions with user input.

Each pipeline step followed a particular workflow in order to complete the pipeline successfully. See the following:

Update config.yaml
Update schema.yaml
Update params.yaml
Update the entity
Update the configuration manager in the src config
Update the components
Update the pipeline
Update the main.py
Update the app.py

Follow the above workflow based on your current pipeline.

Follow this markdown link to generate the base structure of your root repository for your project in order to implement MLOPS.

Uses a mail flow for model registry and the model monitoring and model can be deployed from ML flow only

Here is the tracking URL of my Molde link

Travel Tales AI

Sun, 27 Aug 2023 00:00:00 +0000

This project is the combination of my flutter and prompt engineering skills. In this, I was trying to create an application for travelers or those who want to travel and are confused about where to go. This application can help them decide and explore the places and monuments. This app will work as a bot built on the top of chat GPT.

Smart Loan Model

Tue, 18 Apr 2023 00:00:00 +0000

This model is trained on a loan dataset with parameters like monthly income, gender, marital status, self-employment status, property area, etc. This model uses all the real-life factors and parameters to detect whether a loan should be given to a person or not. It helps banking or loan departments make an informed decision.

In this project, the models used for classification are logistic regression, K-nearest neighbors classifier, support vector machine, and decision tree classifier.

In this project, the following steps were taken: data collection, data preprocessing, data splitting, model implementation, and model evaluation. A detailed description of each step is given below:

Data Collection: I collected the data from Kaggle and read it with the pandas library. I analyzed the data frame using df.info() and df.describe().
Data Preprocessing: For preprocessing, I dropped the duplicate values and removed and filled the null values with their respective methods, like using the median for numerical data and the mode for objective data. For visualization, I drew some plots like line plots, countplots, and bar plots for counting the loan status employment status, and other attributes to get a brief understanding of the dataset. I also generated the categorical and numerical data separately for further splitting.
Data Splitting: In this state, I divided the data into 80% for training and 20% for testing.
Model Implementation: I created a dictionary where the keys are the model names, and the values are the model instances themselves with parameters. I applied functions like loss calculation and model evaluation to this dictionary. Output: Finally, I saved the data in a suitable format using pandas.

Image Manipulation using OpenCV

Fri, 31 Mar 2023 00:00:00 +0000

I implemented the core concept of image handling from my Digital Signal Processing class. This was a streamlit app with functionality of contrast handling, brightness, inverting, and other basic actions on the uploaded image with the help of OpenCV.

Exploratory Data Analysis - COVID-19 Data

Thu, 23 Feb 2023 00:00:00 +0000

This is an EDA Sample Project on the India geographical dataset of COVID-19.

Problem Definition for COVID-19 EDA Sample Project:

Analyze the association between the number of doses taken and the number of deaths in each state.
Calculate the growth rate of confirmed cases and death rate by state.
Calculate the distribution of doses among the population of each state.
Compare the population of each state based on the different doses taken.

I followed complete workflow stages to complete the task in terms of data extraction and visualization for better understanding.

Workflow stages for COVID-19 EDA Project:

Question or Problem Definition.
Acquire training and testing data.
Wrangle, prepare and cleanse the data(preprocessing and cleaning data).
Analyze, identify patterns, and explore the data(Visualization and information extraction).
Conclusion (Extracted information).

Earlier I learned about visualization with Seaborn. Along with that, I try to utilize those skills and

Collab notebook link: Link

Dataset link: Link

My Website link: https://www.ukant.tech

A Twitter Bot- WiseMonkey77

Sun, 07 Aug 2022 00:00:00 +0000

This was an X(Twitter) bot that posted images containing a quote taken from the best quote REST API, every 24 hours, using cron jobs. I utilize the pillow library for adding quotes on the image. It generates images tweets them and also adds them to the local database

The automated posting is done due to the cron jobs in the Linux OS.