DevOps | Ukant Jadia

A Beginner's Guide to DevOps with Terraform and GitHub

Mon, 30 Dec 2024 14:15:59 +0530

A Beginner’s Guide to DevOps with Terraform and GitHub

As a fresh graduate in Computer Science and Engineering, my exploration into the world of DevOps has been nothing short of an enlightening adventure. Coming from a machine learning background, I often found myself compelled to understand how backend systems could be crafted with such elegance. This curiosity fueled my capstone project, where I aimed to create a CI/CD pipeline that elegantly demonstrated the power of tools like Terraform and GitHub.

What is DevOps and Why is it Important?

DevOps blends development and operations, aiming to minimize the distance between these two traditionally siloed groups. The goal is to improve collaboration, increase deployment frequency, and deliver more stable operating environments. The DevOps culture promotes continuous integration and continuous delivery (CI/CD), which significantly enhances the software development lifecycle.

Key Components of DevOps

Continuous Integration (CI): Developers frequently integrate their changes in a shared repository.
Continuous Delivery (CD): Automating the delivery of applications to selected infrastructure environments.
Infrastructure as Code (IaC): Managing infrastructure through code, ensuring that setups are easy to replicate.

A Personal Exploration into DevOps

Reflecting on my academic journey, the decision to delve into a DevOps project stemmed from a fundamental question: How can vast complexities be integrated seamlessly? My project aimed to demystify this by constructing a basic CI/CD pipeline orchestrated through Terraform and GitHub.

Setting the Stage: Understanding My Project

The core of my project revolved around leveraging Terraform for infrastructure automation, along with GitHub for version control and CI/CD capabilities. This combination would provide a straightforward yet powerful example of deploying applications in a cloud environment.

Project Objectives

Demonstrate CI/CD using GitHub Actions: Show how automatic deployment can facilitate rapid product releases.
Use Terraform for Infrastructure Management: Illustrate how infrastructure can be provisioned and managed as code.
Visualize Workflows: Employ tools like Eraser.io to create flowcharts to help visualize the project’s structure.

Getting Started with Terraform

When I first explored Terraform, I was struck by how declarative it is. Terraform allows you to define your infrastructure in configuration files which can then be executed to provision that infrastructure automatically.

Example Terraform Configuration

Here’s a simple example of a Terraform configuration file that provisions an AWS EC2 instance:

provider ""aws"" {
 region = ""us-west-2""
}

resource ""aws_instance"" ""my_instance"" {
 ami = ""ami-0c55b159cbfafe1f0""
 instance_type = ""t2.micro""

 tags = {
 Name = ""MyInstance""
 }
}

Key Features of Terraform

Infrastructure as Code: Treat infrastructure setup as code, making it easier to maintain and version control.
Execution Plan: Terraform generates a clear execution plan which shows what will happen before any changes are applied.

Integrating GitHub Actions into the Pipeline

GitHub Actions provides a robust platform for automating workflows directly in your GitHub repository. For my project, I set up actions to automatically test and deploy the application upon code commit.

GitHub Actions Workflow Configuration

This YAML file illustrates a basic GitHub Actions workflow:

name: CI/CD Pipeline

on:
 push:
 branches: [ main ]

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - name: Check out the code
 uses: actions/checkout@v2

 - name: Set up Node.js
 uses: actions/setup-node@v2
 with:
 node-version: '14'

 - name: Install dependencies
 run: npm install

 - name: Run tests
 run: npm test

 - name: Deploy to production
 run: npm run deploy

Benefits of Using GitHub Actions

Seamless Integration: Works directly with your repositories.
Customizability: Create tailored workflows to fit specific project needs.

Visualizing the Workflow

To visualize how data flows through the different stages of the pipeline, I employed Eraser.io. This tool allowed me to create flowcharts that represented the overall CI/CD process from code push to deployment.

Figure 1: The flow of code and data in the CI/CD pipeline

The Elegance of CI/CD: A Real-World Application

As I progressed through my project, witnessing the pipeline’s operation became one of the most rewarding aspects. It was astounding to see how a simple workflow could articulate the path of requests or data between containers and servers.

Key Takeaways from the CI/CD Pipeline Experience

Visual Learning: The flowcharts provided invaluable insight into how requests and data moved, making the whole process comprehensible.
Hands-on with Real Tools: Working directly with GitHub and Terraform solidified my understanding of cloud infrastructure and automation.

Final Thoughts

Reflecting on this project, I can confidently say that building a CI/CD pipeline using Terraform and GitHub not only quenched my curiosity about backend processes but also equipped me with practical skills for my career ahead. This journey reaffirmed my belief that technology can create elegant solutions to complex problems. I hope my exploration encourages others to dive into the DevOps ocean, where every wave brings new learning opportunities.

References

Feel free to explore these resources further to expand your knowledge and understanding of DevOps and CI/CD pipelines.

Food Image Classification

Mon, 01 Jan 2024 00:00:00 +0000

This project tackles the challenge of applying computer vision to emphasize accurate food image classification. It compares Convolutional Neural Networks (CNN) with Traditional Machine Learning Models, leveraging CNN’s feature-rich capabilities and traditional models’ transparency and computational efficiency. This dual approach enhances food image classification’s precision and efficiency, impacting quality assessment, dietary analysis, and food inventory management

Used the 5K image dataset for classifying the food images and non-food images, Trained the neural network on the images for classification, and developed the model with streamlit application and achieved 80% accuracy. The dataset is taken from https://mmspg.epfl.ch/food-image-datasets and the name of the data set is 5K.

Dataset description: -. This dataset contains 2500 food and 2500 non-food images, for the task of food/non-food classification in our paper “Food/Non-food Image Classification and Food Categorization using Pre-Trained GoogLeNet Model”. The whole dataset is divided into three parts: training, validation and evaluation. The naming convention is as follows: -. {ClassID}_{ImageID}.jpg -. ClassID: 0 or 1; 0 means non-food and 1 means food. -. ImageID: ID of the image within the class

OBE Module for University

Mon, 01 Jan 2024 00:00:00 +0000

This was a team project.

The aim of this project is to provide the university with a web portal for the implementation of Outcome-Based Education (OBE).

OBE can primarily be distinguished from traditional education methods by the way it incorporates three elements: a theory of education, a systematic structure for education, and a specific approach to instructional practice. It organizes the entire educational system towards what is considered essential for learners to successfully achieve at the end of their learning experiences. In this model, the term ‘outcome’ is the core concept and is sometimes used interchangeably with the terms ‘competency,’ ‘standards,’ ‘benchmarks,’ and ‘attainment targets.’ OBE also utilizes the same methodology formally and informally adopted in the actual workplace to achieve outcomes.

The web portal is built with the Laravel framework of PHP for the front end and utilizes MySQL for the database.

My Role: My role is to build the business logic, conduct literature reviews, generate reports, and suggest the next steps based on those reports. I also assist in building the database

Bloom's Taxonomy Level Detection

Wed, 13 Dec 2023 00:00:00 +0000

To know Bloom’s Taxonomy in Detail see this pdf This

This project aims to classify the given statement into accurate Bloom’s taxonomy levels out of six levels. This is an NLP-based project. This project is built with TF-IDF vectorizer and multinomial Naive Bayes techniques.

Bloom’s Taxonomy Levels: Bloom’s Taxonomy is a classification system that arranges learning objectives into six hierarchical levels, each representing a different cognitive skill.

The model is trained on 1800 augmented statements. Initially, I had 100 statements for each class, so a total of 600 statements for six classes. Then, this data is augmented three times using contextual word embedding augmentation, and the model used in the augmentation is BERT-based. An example of the augmentation methodology.

This project follows MLOps techniques well. I have divided this project into 6 different pipelines for easy implementation and scalability. I created a class in the pipeline for each class using different entities and components. In generating a configuration file from the user input in the YAML format.

In the end, I created a Docker image so it can be used by the Higher Education Institute(HEIs) for those who are using the OBE(Object Based Education) new curriculum technique. It just reduced the manual process of identifying each question and its root taxonomy level.

A MLOps Project- Wine Quality Prediction

Fri, 08 Dec 2023 00:00:00 +0000

This project demonstrates my ability to implement a highly scalable MLOPS project by following development techniques and project procedures. In this project, I divided the whole implementation part into six different pipelines and steps, which are as follows:

Data Ingestion: Collecting the data and setting it up for further use.

Data Validation: Checking the data type, format, and other conditions to see the validity and usability of the data.

Data Transformation: Performing preprocessing and splitting the data into test and train datasets.

Model Training: Selecting a suitable machine learning algorithm and creating a model by training.

Model Evaluation: Testing the data and verifying it with measurement/result metrics.

Model Prediction: Making predictions with user input.

Each pipeline step followed a particular workflow in order to complete the pipeline successfully. See the following:

Update config.yaml
Update schema.yaml
Update params.yaml
Update the entity
Update the configuration manager in the src config
Update the components
Update the pipeline
Update the main.py
Update the app.py

Follow the above workflow based on your current pipeline.

Follow this markdown link to generate the base structure of your root repository for your project in order to implement MLOPS.

Uses a mail flow for model registry and the model monitoring and model can be deployed from ML flow only

Here is the tracking URL of my Molde link