Building a Robust CI/CD Pipeline with Jenkins, Docker, Kubernetes, and ArgoCD

Prerequisites:

1. Docker Installation: Follow the official Docker documentation to install Docker on your Ubuntu machine. Refer to Docker’s official documentation.
2. Kubernetes Setup with Minikube: Utilize the step-by-step guide I’ve crafted for setting up Minikube Kubernetes cluster with KVM driver on Ubuntu. You can access it here.
3. Jenkins Installation: Install Jenkins on your Ubuntu machine following the instructions outlined in the official Jenkins documentation.
4. ArgoCD Installation: Set up ArgoCD on Ubuntu using the instructions provided here.

Introduction:

This project integrates Jenkins, Docker, Kubernetes, and ArgoCD to establish a robust CI/CD pipeline. We’ll be working with two repositories:

1. jenkins-gitops-docker: Responsible for building a Flask application, creating a Docker image from it, tagging the image with the build number, and pushing it to Docker Hub.
2. jenkins-gitops-k8s: Manages Kubernetes deployment and service manifests.

jenkins-gitops-docker Repository:

This repository contains the following files:

• Dockerfile
• Jenkinsfile
• app.py
• Requirements.txt

Dockerfile:

# Use an official Python runtime as a parent image
FROM python:alpine

# Set the working directory in the container
WORKDIR /app

# Copy the current directory contents into the container at /app
COPY . /app

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Make port 5000 available to the world outside this container
EXPOSE 5000

# Run app.py when the container launches
CMD ["python", "app.py"]

Jenkinsfile:

node {
    def app

    stage('Clone repository') {
        checkout scm
    }

    stage('Build image') {
       app = docker.build("devopswithsam/jenkins-flask")
    }

    stage('Test image') {
        app.inside {
            sh 'echo "Tests passed"'
        }
    }

    stage('Push image') {
        docker.withRegistry('https://registry.hub.docker.com', 'dockerhub') {
            app.push("${env.BUILD_NUMBER}")
        }
    }
    
    stage('Trigger ManifestUpdate') {
        echo "triggering updatemanifestjob"
        build job: 'updatemanifest', parameters: [string(name: 'DOCKERTAG', value: env.BUILD_NUMBER)]
    }
}

app.py:

from flask import Flask

app = Flask(__name__)

@app.route('/')
def hello():
    return 'Hello Sam'

if __name__ == '__main__':
    app.run(debug=True, host='0.0.0.0')

Requirements.txt:

Flask

jenkins-gitops-k8s Repository:

This repository includes the following files:

• Deployment.yml
• Jenkinsfile

Deployment.yml:

apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: flaskdemo
  name: flaskdemo
spec:
  replicas: 2
  selector:
    matchLabels:
      app: flaskdemo
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: flaskdemo
    spec:
      containers:
      - image: devopswithsam/jenkins-flask:4
        name: flaskdemo
        resources: {}
status: {}

---
apiVersion: v1
kind: Service
metadata:
  name: lb-service
spec:
  type: NodePort
  selector:
    app: flaskdemo
  ports:
  - port: 80
    targetPort: 5000
    nodePort: 30007

Jenkinsfile:

node {
    def app

    stage('Clone repository') {
        checkout scm
    }

    stage('Update GIT') {
        script {
            catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
                withCredentials([usernamePassword(credentialsId: 'github', passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
                    sh "git config user.email devopssameera@gmail.com"
                    sh "git config user.name Sameera Dissanayaka"
                    sh "cat deployment.yml"
                    sh "sed -i 's+devopswithsam/jenkins-flask.*+devopswithsam/jenkins-flask:${DOCKERTAG}+g' deployment.yml"
                    sh "cat deployment.yml"
                    sh "git add ."
                    sh "git commit -m 'Done by Jenkins Job changemanifest: ${env.BUILD_NUMBER}'"
                    sh "git push @github.com/${GIT_USERNAME}/jenkins-gitops-k8s.git">https://${GIT_USERNAME}:${GIT_PASSWORD}@github.com/${GIT_USERNAME}/jenkins-gitops-k8s.git HEAD:main"
                }
            }
        }
    }
}

Setting Up Credentials in Jenkins:

1. Go to Jenkins Dashboard -> Manage Jenkins -> Credentials.

2.Under “Stores scoped to Jenkins,” select “Global credentials (unrestricted)” -> Add Credentials.

3. Choose “Username with password” as the type.

4. Provide your GitHub username as the username and your Personal Access Token (PAT) as the password.

• To generate a PAT, go to your GitHub profile settings -> Developer settings -> Personal access tokens -> Generate new token. Ensure to grant ‘repo’ and ‘notification’ scopes.

5. Use the generated token as the password and assign ‘github’ as the ID.

Creating Jenkins Pipelines:

1. Build Docker Image Pipeline:

• Create a new item -> Select “Pipeline” -> Define pipeline script from SCM.

• Repository URL: URL of the jenkins-gitops-docker repository.
• Credentials: Select Docker Hub credentials.
• Branches to build: Change from “master” to “main”.

• Script Path: Jenkinsfile.
• Apply and save.

2. Update Manifest Pipeline (updatemanifest):

• Create a new item -> Select “Pipeline” -> Check “This project is parameterized” -> Add a string parameter named “DOCKERTAG” with a default value of “latest”.

• Define pipeline script from SCM.

• Repository URL: URL of the jenkins-gitops-k8s repository.
• Credentials: GitHub credentials.
• Branches to build: Change from “master” to “main”.

• Script Path: Jenkinsfile.
• Apply and save.

Starting the Jenkins Pipeline Manually:

1. Navigate to Jenkins Dashboard:

• Open your web browser and go to the Jenkins dashboard.

2. Locate the Pipeline Job:

• On the Jenkins dashboard, find the pipeline job named “buildimage.”

3. Initiate the Build:

• Click on the “buildimage” pipeline job to access its details.

4. Start the Build Process:

• Inside the pipeline job details, locate and click the “Build Now” option.

5. Monitor Build Progress:

• Jenkins will now initiate the pipeline build process. You can monitor the progress in real-time on the job’s page.

6. Review Build Logs:

• During the build process, Jenkins will execute the stages defined in the Jenkinsfile. You can review the build logs to track each stage’s execution.

7. Check Pipeline Status:

• Once the build process completes, Jenkins will display the status of the pipeline build. Ensure that all stages are executed successfully without any errors.

8. Verify Docker Image Push:

• After the pipeline build, Jenkins will push the Docker image to Docker Hub as specified in the Jenkinsfile. You can verify the image push by checking your Docker Hub repository.

9. Confirm Update to Deployment.yml:

• Following the successful build and image push, Jenkins will update the deployment.yml file in the jenkins-gitops-k8s repository with the latest Docker image tag. You can verify this by inspecting the repository.

ArgoCD Setup:

1. Start Minikube: Minikube start.
2. Get ArgoCD services: Kubectl get svc -n argocd.

3. Access ArgoCD web UI: Minikube service argocd-server -n argocd.

4. Retrieve ArgoCD admin password: kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath={.data.password} | base64 -d | more.

5. Log in to ArgoCD and create a new app with the desired configurations.

Configuring Application in ArgoCD:

Once your Jenkins pipeline updates the deployment manifests in the jenkins-gitops-k8s repository, you can configure ArgoCD to monitor these changes and automatically deploy the updated application to your Kubernetes cluster.

1. Create a New Application:

• Navigate to the ArgoCD dashboard.
• Click on “New App” to create a new application.

2. General Configuration:

• In the General section, provide the following details:
• Application Name: Enter a name for your application (e.g., “My-flask-app”).
• Project: Leave it as default.
• Sync Policy: Set it to “Automatic” to enable continuous synchronization.

3. Source Configuration:

• In the Source section:
• Repository URL: Specify the URL of your jenkins-gitops-k8s repository.
• Path: As all your manifests are stored in the root directory, leave this field as ./
• Revision: Choose the branch or commit to track for changes (e.g., “HEAD”).

4. Destination Configuration:

• In the Destination section:
• Cluster URL: Select the default cluster URL to deploy your application.
• Namespace: Choose the namespace where you want your application to be deployed (e.g., “default”).

5. Create the Application:

• After configuring the settings, click on “Create” to create the application in ArgoCD.

6. Monitor Application Deployment:

• Once the application is created, ArgoCD will automatically start the synchronization process.
• You can monitor the deployment progress in the ArgoCD dashboard.
• After successful deployment, you can access your application by navigating to the corresponding service.

Viewing Running Pods and Accessing the Application:

After deploying your application using ArgoCD, you can easily monitor the running pods and access your application through a web browser.

1. View Running Pods:

• Once the deployment is complete, navigate to the ArgoCD dashboard.
• Click on your application to view its details.
• In the application details, you can see the list of pods running for your application.

2. Accessing the Application:

• To access your application, you need to expose it as a service to be accessible from outside the Kubernetes cluster.
• Open a terminal window and execute the following command to get the service details:

kubectl get svc

• This command will display a list of services in your Kubernetes cluster.
• Locate the service named lb-service (or the service associated with your application).

3. Create a Service to Minikube:

• Once you’ve identified the service, create a tunnel to the service using Minikube:

minikube service lb-service

This command will open up your default web browser and automatically navigate to the URL where your Flask application is running.

4. Accessing the Application:

In your web browser, you should now see your Flask application running, displaying the content “Hello Sam”

Testing Application Updates:

After modifying the app.py content in the jenkins-gitops-docker repository to display "Hello ArgoCD is Cool" and pushing the changes to the repository, you need to trigger a new build in Jenkins to update the Docker image and deploy the changes to your Kubernetes cluster.Trigger will not automatically start because im using a laptop.Otherwise github webhook will trigger the automation.I will explain solution later

1. Trigger Jenkins Pipeline:

• Navigate back to your Jenkins dashboard.
• Locate the “buildimage” pipeline job and initiate a new build by clicking “Build Now”.

2. Monitor Build Progress:

• Keep an eye on the Jenkins dashboard to track the progress of the build pipeline.
• Jenkins will execute the pipeline stages, including building the Docker image with the updated content and pushing it to Docker Hub.

3. Deployment by ArgoCD:

• After Jenkins completes the build process, ArgoCD will detect the changes in the GitOps repository and initiate a deployment to your Kubernetes cluster.
• ArgoCD will synchronize the deployment manifests, ensuring that the updated Docker image is deployed.

4. Wait for Deployment:

• Allow a few minutes for the deployment to complete. You can monitor the deployment progress in the ArgoCD dashboard.

5. Accessing the Updated Application:

Once the deployment is successful, open your web browser and access the application.

Using Ngrok for GitHub Webhooks (Optional):

If you’re running Jenkins on your local machine instead of AWS EC2, you won’t have a public IP address accessible to GitHub for webhook notifications. In such cases, you can utilize Ngrok, a tool that creates secure tunnels to localhost, enabling external access.

1. Install Ngrok:

• Download and install Ngrok from the official website: Ngrok Website.

2. Start Ngrok Tunnel:

• Open a terminal window and navigate to the directory where Ngrok is installed.
• Run the following command to start an HTTP tunnel on port 8080 (assuming Jenkins runs on port 8080):

ngrok http 8080

3. Note Ngrok URL:

• After starting Ngrok, it will generate a temporary public URL (e.g., http://<random_string>.ngrok.io). Note down this URL as it will be used to receive GitHub webhook payloads.

4. Configure GitHub Webhook:

• Go to your GitHub repository settings and navigate to the “Webhooks” section.
• Click on “Add webhook” and enter the Ngrok URL followed by /github-webhook/ (e.g., http://<random_string>.ngrok.io/github-webhook/) as the payload URL.
• Choose the events you want Jenkins to listen to, typically “push” events.
• Save the webhook configuration.

5. Verify Webhook Delivery:

• GitHub will send a test payload to the Ngrok URL. You should see a successful delivery response.

6.Restart Jenkins:

• Restart Jenkins to ensure that it picks up the changes and starts listening to webhook events.

7. Test Webhook Integration:

• Make a change in your GitHub repository and push it.
• GitHub will trigger the webhook, sending a payload to Ngrok, which forwards it to your local Jenkins instance.
• Jenkins should start a new build based on the webhook payload.

Conclusion:

This comprehensive guide provides a step-by-step walkthrough of setting up a complete end-to-end project using Docker, Kubernetes, Jenkins, and ArgoCD. By following these instructions, you’ve learned how to automate the building, testing, and deployment processes, ensuring efficiency and reliability in your software development workflow.

As you continue your journey in mastering these technologies, don’t hesitate to refer back to this guide for reference. Additionally, exploring other articles and resources can further deepen your understanding and expand your knowledge.

For future updates and more insightful articles, consider following the author’s profile. Stay tuned for more informative content and happy learning!

Feel free to reach out if you have any questions or need further assistance. Happy coding!