Linux | Ukant Jadia

A Beginner's Guide to DevOps with Terraform and GitHub

Mon, 30 Dec 2024 14:15:59 +0530

A Beginner’s Guide to DevOps with Terraform and GitHub

As a fresh graduate in Computer Science and Engineering, my exploration into the world of DevOps has been nothing short of an enlightening adventure. Coming from a machine learning background, I often found myself compelled to understand how backend systems could be crafted with such elegance. This curiosity fueled my capstone project, where I aimed to create a CI/CD pipeline that elegantly demonstrated the power of tools like Terraform and GitHub.

What is DevOps and Why is it Important?

DevOps blends development and operations, aiming to minimize the distance between these two traditionally siloed groups. The goal is to improve collaboration, increase deployment frequency, and deliver more stable operating environments. The DevOps culture promotes continuous integration and continuous delivery (CI/CD), which significantly enhances the software development lifecycle.

Key Components of DevOps

Continuous Integration (CI): Developers frequently integrate their changes in a shared repository.
Continuous Delivery (CD): Automating the delivery of applications to selected infrastructure environments.
Infrastructure as Code (IaC): Managing infrastructure through code, ensuring that setups are easy to replicate.

A Personal Exploration into DevOps

Reflecting on my academic journey, the decision to delve into a DevOps project stemmed from a fundamental question: How can vast complexities be integrated seamlessly? My project aimed to demystify this by constructing a basic CI/CD pipeline orchestrated through Terraform and GitHub.

Setting the Stage: Understanding My Project

The core of my project revolved around leveraging Terraform for infrastructure automation, along with GitHub for version control and CI/CD capabilities. This combination would provide a straightforward yet powerful example of deploying applications in a cloud environment.

Project Objectives

Demonstrate CI/CD using GitHub Actions: Show how automatic deployment can facilitate rapid product releases.
Use Terraform for Infrastructure Management: Illustrate how infrastructure can be provisioned and managed as code.
Visualize Workflows: Employ tools like Eraser.io to create flowcharts to help visualize the project’s structure.

Getting Started with Terraform

When I first explored Terraform, I was struck by how declarative it is. Terraform allows you to define your infrastructure in configuration files which can then be executed to provision that infrastructure automatically.

Example Terraform Configuration

Here’s a simple example of a Terraform configuration file that provisions an AWS EC2 instance:

provider ""aws"" {
 region = ""us-west-2""
}

resource ""aws_instance"" ""my_instance"" {
 ami = ""ami-0c55b159cbfafe1f0""
 instance_type = ""t2.micro""

 tags = {
 Name = ""MyInstance""
 }
}

Key Features of Terraform

Infrastructure as Code: Treat infrastructure setup as code, making it easier to maintain and version control.
Execution Plan: Terraform generates a clear execution plan which shows what will happen before any changes are applied.

Integrating GitHub Actions into the Pipeline

GitHub Actions provides a robust platform for automating workflows directly in your GitHub repository. For my project, I set up actions to automatically test and deploy the application upon code commit.

GitHub Actions Workflow Configuration

This YAML file illustrates a basic GitHub Actions workflow:

name: CI/CD Pipeline

on:
 push:
 branches: [ main ]

jobs:
 build:
 runs-on: ubuntu-latest
 steps:
 - name: Check out the code
 uses: actions/checkout@v2

 - name: Set up Node.js
 uses: actions/setup-node@v2
 with:
 node-version: '14'

 - name: Install dependencies
 run: npm install

 - name: Run tests
 run: npm test

 - name: Deploy to production
 run: npm run deploy

Benefits of Using GitHub Actions

Seamless Integration: Works directly with your repositories.
Customizability: Create tailored workflows to fit specific project needs.

Visualizing the Workflow

To visualize how data flows through the different stages of the pipeline, I employed Eraser.io. This tool allowed me to create flowcharts that represented the overall CI/CD process from code push to deployment.

Figure 1: The flow of code and data in the CI/CD pipeline

The Elegance of CI/CD: A Real-World Application

As I progressed through my project, witnessing the pipeline’s operation became one of the most rewarding aspects. It was astounding to see how a simple workflow could articulate the path of requests or data between containers and servers.

Key Takeaways from the CI/CD Pipeline Experience

Visual Learning: The flowcharts provided invaluable insight into how requests and data moved, making the whole process comprehensible.
Hands-on with Real Tools: Working directly with GitHub and Terraform solidified my understanding of cloud infrastructure and automation.

Final Thoughts

Reflecting on this project, I can confidently say that building a CI/CD pipeline using Terraform and GitHub not only quenched my curiosity about backend processes but also equipped me with practical skills for my career ahead. This journey reaffirmed my belief that technology can create elegant solutions to complex problems. I hope my exploration encourages others to dive into the DevOps ocean, where every wave brings new learning opportunities.

References

Feel free to explore these resources further to expand your knowledge and understanding of DevOps and CI/CD pipelines.

Markdown to Html Conversion

Sat, 30 Dec 2023 14:22:57 +0530

Conversion of Markdown(.md) to html markup language

get the pandoc
Use the following command

 pandoc smaple_readme.md -t html -o sample_readme.html

In shell it would be written like

cat=$(pandoc $1.md -t html -o $1.html )

It will be run the loop with wildcard like

 for i in ~/locat/.md/file/* ; do $(var) ; done

Modification

check extension of each file with if condition and print the non md file like

if [ $1 -ne *.md ]
then
 echo "UNEXPECTED Document format $1 "
 continue
else:
 `further code`
fi

Learn tmux, a Terminal Multiplexer

Sat, 30 Dec 2023 14:22:29 +0530

tmux

Installation

Linux : sudo apt-get install tmux
Fedora : sudo dnf -y install tmux
Cent-OS : sudo yum -y install tmux
Arch Linux : sudo pacman -S tmux
to start : tmux
NOTE: Prefix key : Ctrl+b
github

What is tmux ??

It is terminal multiplexer.
It acts as a window manager in your terminal.
It also allows to create multiple window and panes in a single terminal window.

What it can do

Tmux keeps the sessions , windows and panes in a server located at /tmp/tmux
If your terminal window is crash or killed(detach) then still all your process and session is alive, until you kill the tmux server(reboot).
Again you can pick the session from where you left by simply attaching to the session.
tmux is also helpful while working with remote machine.

tmux key binds

NOTE: Prefix key C-b == Ctrl+b

to start : tmux
Splitting Pane :
- for left and right : C-b+%
- for top and bottom : C-b+"
Navigates through panes :
- C-b+<arrow_keys
- to move left : C-b ⬅
- to move right : C-b ➡
- to move up : C-b ⬆
- to move down : C-b ⬇
Closing pane: type exit or C-b x
Creating windows : C-b c
- rename windows : C-b ,
- to kill window : C-b &
Navigates through windows :
- for previous window : C-b p
- for next window : C-b n
- for required : C-b <number> where number is number in front of the window’s name in your status bar
Session handling :
- to detach your current session : C-b d
- for list of session to detach : C-b D
  - for list of session : C-b s
  - for list of session : tmux ls it worked without a tmux session
  - to attach to session from list : tmux attach -t 0 -t 0 -t for specific target and 0 name of the session
  - attach to a new session : tmux new -s database new -s database new for new , -s for session , database as name of session new -w __ -w for new window
  - rename session : C-b $ rename before starting the session : tmux rename-session -t 0 database
to enter command prompt mode : C-b :

NOTE : to see the more key bindings use C-b ?

It will show the all binding available in it

Links 🔗

Information Gathering, the First Step for Exploitation

Sat, 30 Dec 2023 14:20:17 +0530

Recon

There are two type of Reconnaissance

Passive Reconnaissance :– Doing the recon without coming in direct contact of the target. \
Active Reconnaissance :– Doing the active notecible action on the target site.

Passive Reconnaissance includes

local/Geo Information
- Satellite Images
- Building layout
- Cabin location, fence, count of head in building, ways to exit, break areas, security fencing.
Job information
Senior manager name, meeting routine, job title.
Image from social sites like badge photo, cubical, desktop.

Web and Network Information

Targe Validation

First thing you have to do is, validate your target.

Weather the description of given target is having the relating with the given ip

Finding Subdomain

Testing only one domain or some domain is decrease the chances of finding the flaw.

Testing the all in-scope sub-domain is a good practice.

Tool can be use for finding the subdomain are

Google fu, dig, Nmap, sublist3r, bluto, crt.sh etc.

Fingerprinting

We need to know which website running what on front or back.

What’s running on the host.

What’s web server, ssh, ftp, services.

Data Breaches

Just check if the credentials or account is pawned or not

Use this to check

havebeenpawned, Breach-prose, weleakinfo.

NOTE: Better Recon, enumeration makes you better pentester.

Email Address gathering with Hunter.io

You can use Hunter.io to find the people email address working or related to the domain you enter.
In free plan it only provide the 20 searches as free, so do not abuse the searches.
It also tell the where it get all emails written in source.
You have to login with valid email id to have account.
It also provide the category of email address, the department of the email owner.

Breach-Phrase from github

It is tool on the GitHub, it checks for the leaked credentials in its own in-build data of credentials
This in build data is gather from the dark web store.

NOTE: the gather email address are plays important role in further recon and enumeration.

Utilizing the Harvester

It searches with the domain.
It can gather the subdomain name, e-mail addresses, virtual hosts, open ports/banner, and employee name from different public sources like search engine, social site

Hunting Subdomain

We need see/identify what are the available or working subdomain out there. use this

Sublist3r
- Sometimes it only give the 3 level subdomain
crt.sh
- It give the all registered subdomain that can be 3 level or 4 level
OWASP Amass is more efficient tool than above mentioned
Sublist3r is slow during to fast up increase the threads NOTE: Found subdomain may or may not alive.

Identifying the Web Technologies

To Fingerprinting the web page or to get the services running in the background
wappalyzer is browser extension which show the frontend and backend services.
Terminal tool is whatweb
- use whatweb -h for more details of tool

Basic Git Commands

Sat, 30 Dec 2023 14:15:59 +0530

what you should do in git

git init -b main
git remote add origin <repo_link>
git remote -v

cd to the root file of the project
echo "+ Project " >> README.md
git init
git add .
git commit -m "README"
git branch -M Main
git remote add origin <repo_link>
git push -u origin Main

pushing an existing repo from cli git remote add origin https://github.com/UserCtf777/Inkdrop.git git branch -M Main git push -u origin Main

after clonning any repo it have the address of origin, simply you cannot add your repo url for backup/project/ICR for that you have to change the origin url follow the following command

to check the origin url try the remote verbose git remote -v
to change the remote origin try git remote set-url origin <repo_url>

new error found
sometimes when we clone the repo we use depth=1 to remove it’s history
this give the error during pushing it to other origin
error looks like

 ![remote rejected] Main -> Main (shallow update not allowed)

```
 error: failed to push some refs to
```
to solve this try git remote set-url <old_url> git fetch –unshallow old

git init -b main
git remote add origin <repo_link>
git remote -v

cd to the root file of the project echo “+ Project " » README.md git init git add . git commit -m “README” git branch -M Main git remote add origin <repo_link> git push -u origin Main

pushing an existing repo from cli git remote add origin https://github.com/UserCtf777/Inkdrop.git git branch -M Main git push -u origin Main

after clonning any repo it have the address of origin, simply you cannot add your repo url for backup/project/ICR for that you have to change the origin url follow the following command

to check the origin url try the remote verbose git remote -v
to change the remote origin try git remote set-url origin <repo_url>

new error found
sometimes when we clone the repo we use depth=1 to remove it’s history
this give the error during pushing it to other origin
error looks like

 ![remote rejected] Main -> Main (shallow update not allowed)

```
 error: failed to push some refs to
```
to solve this try git remote set-url <old_url> git fetch –unshallow old +!/bin/bash

cd ~/work/obs/Obsidian

git add . git commit -m ‘daily update’ git push

+!/bin/bash

to make dir work as git repo git init
to check the changes git status
to sort/simplify the changes git add .
git log shows the log of all commit done git log
restore to discard the changes git restore .
to commit the changes [with ‘-m’ for successful commit/{-m => metadata of changes}] git commit -m “daily update”
git push to add new file to the repo git push
command to add remote repo / it skip the authentication part during git push git remote add origin git@github.com:UserCtf777/Obsidian.git git branch -M master git push -m origin master
To solve the conflicts during rebasing git rebase –continue + fix conflicts and run this git rebase –skip + to skip this patch/changes git rebase –abort + to checkout the original branch
When we have unresolved conflicts before merging or having error related to add git fetch + to gain the latest changes from server git merge origin master + to give a try to automatice merging
it will only work if branch has to merge something– it can be checked with git status git merge –abort + the unresolved conflicts will be cleared off
to edit the conflicts git rebase –edit-todo git rebase –continue
if conflicts dosen’t resolve then git rebase –abort
it will give the path of unresolved files
if you know how to edit then resolve them
OR you can remove that confict by git rm
if nothing is ahppening you can skip the commit wiht git rebase –skip

Enumeration Second Step for Exploitation

Sat, 30 Dec 2023 14:15:39 +0530

Enumeration

Kioptrix level 1

Now as beginner we should a have vulnerable machine to test
Use kioptrix level a from vulnhub.
This is used for beginner to develop there skill by testing the vulnerable machine.
Download it, and set up it on your virtual software.
Make sure the network connection is set to NAT
Open the root folder(which contain all configuration files) and open configuration file and change Bridge to NAT
I Coppied select this option form pop dialogue
Once the setup is done do the arp-scan -l
On your system it will discover the available host in your ip CHIR
Ignore the x.x.x.1 and x.x.x.254 because this are your network identity and broadcast ip, Rest all machine are lying in your ip range.

Nmap stealthy scan

What normal two-way connection looks like

SYN <-> SYNACK <-> ACK

Stealthy scan doesn’t establish the connection it ask for the open port on the serve as server reply the status it will take the step back without establishing the connection.

SYN <-> SYNACK <-> RST

Use this Namp flags for better enumeration nmap -T4 -p- -A <ip_address>
Go through the scan properly
Visit ip/webpage, check weather there you can do anything like directory busting.
Get all services running, also check the header and behaviour.
Do whatever you can do.
Then use nikto web vulnerability tool.

nikto -host <url>

Directory busting

dirbuster / birb / gobuster

Dirbuster

start dirbuster from terminal
fill the slot with information of your target
for word file use /usr/share/wordlist/dirbuster
File extension depends on you or the time you have.
Received/common url can be a information disclosure of anything like stats, user-data, server header, response.

Dwmbar vs Dwmblocks, a Task Manager

Sat, 30 Dec 2023 14:15:02 +0530

2022-05-17

problem statement

with dwmbar the cpu uses is touching its picks and that thing i don’t like

solution

to solve this i am trying to get the dwmblocks of Bugswriter

error or problem faced

get repo
run make and you may get this error 🔽
Pasted image 20220518015013.png
solving this we get this

Solution

okh now the error is fixed

So first know what is error about

error we are trying to automatically compile the dwmblocks.c file and in the code the library which is ariseing the error that is -lX11 The path to solve this is we have to do the maual compilation of an C language file and that is very easy

command for compilation is gcc dwmblocks.c -lX11 -o dwmblocks ▶ gcc as in-build compiler, dwmblocks.c file to compile, -lX11 is lib adding manually during compilation, -o to get output with, dwmblocks as name of output file

edit the config file according to required blocks

NOTE📌 : cp the dwmblocks file to the /usr/local/bin so it can run from any where

take the dwmbloks script form lukesmith voidrice build use this

add bar scripts in the /usr/local/bin this path is given in the documentation of dwmblocks

now you are ready to run

Command

idconfig –> configure dynamic linker run-time bindings.

Used to create links and cache to the most recent shared libraries .

gcc <-lib-to-add> -o <output-file-name>

gcc dwmblocks.c -lX11 -o dwmblocks

Networking Command Netstat

Sat, 30 Dec 2023 14:14:11 +0530

Learn Netstat

A basic Linux network command

NETSTAT

Prints the network connection and routing tables.
Like an CCTV monitor for every connection come in and go out from your machine.
MiniManual

netstat -ant : a for all connection, n for numeric, t for all TCP connection//u for the UDP connection

netstat -rnec :r for the route information, n numeric, e extend, c continous prints the result

netstat -stuw : s for statistics summary,t/u for tcp/udp, w prints raw data

netstat -tulpn : it shows the connectio with PID numbers

lsof -i : it establish the connection

A Twitter Bot- WiseMonkey77

Sun, 07 Aug 2022 00:00:00 +0000

This was an X(Twitter) bot that posted images containing a quote taken from the best quote REST API, every 24 hours, using cron jobs. I utilize the pillow library for adding quotes on the image. It generates images tweets them and also adds them to the local database

The automated posting is done due to the cron jobs in the Linux OS.