Building a Master Pipeline for Microservices
Introduction
Microservice architecture becomes a de facto pattern for
building and deploying large scale applications. Microservice architecture means
breaking down the one big monolithic application into multiple small services
that can be independently developed and deployed. This means we may be having
lots of small services and the volume comes with it own set of challenges to
manage. One such problem is DevOps. Due to its volume, it will be less
efficient to build and deploy these services manually and we need to automate
the whole process. To build and deploy these services we should have a robust
CICD pipeline. This article describes about how to build a one such pipeline
using Azure Devops.
DevOps
In typical software development, the DevOps has the following process,
1.
Coding – code development and review, source
code management tools, code merging.
2.
Building – continuous integration tools, build
status.
3.
Testing – continuous testing tools that provide
quick and timely feedback on business risks.
4.
Packaging – artifact repository, application
pre-deployment staging.
5.
Releasing – change management, release
approvals, release automation.
6.
Configuring – infrastructure configuration and
management, infrastructure as code tools.
7.
Monitoring – applications performance
monitoring, end-user experience.
In the above building, testing, packaging, and releasing are
the process where CICD plays a major role. CI/CD comprises of continuous
integration (CI) and continuous delivery (CD), or continuous deployment (CD).
Used together, the three processes automate build, testing, and deployment so
DevOps teams can ship code changes faster and more reliably.
In the DevSecOps we will be having one more process called
Scanning, where the code or binary are scanned with a set of tools for any
security vulnerabilities.
Master pipeline
Although the DevOps process are clear, implementing these
into many services becomes cumbersome. We may have to repeat the same process
for each microservice probably duplicating the pipeline code. If there is a
change in any of the process it will be tedious task to update each service
pipeline. So, we should abstract the common process irrespective of the
services into a master pipeline. In this article I use the team “Master
Pipeline” that contains the common pipeline code and “Service Pipeline” for
service specific pipeline code.
Advantages of master pipeline,
-
Centrally manage the common pipeline code
-
Avoid code duplication by coping the repeated
pipeline code into each service pipeline
-
Any changes to the master pipeline will
automatically reflected in service pipeline as well
-
Manage security related stuff centrally which is
critical to DevSecOps process
-
Versioning the pipeline becomes easier instead
of versioning each service pipeline
The following diagram shows the process of master and
service pipelines,
Figure 2
Master-Service pipeline architecture
Master pipeline using Azure DevOps
Master pipeline in our project isn’t a pipeline on its own. It is an abstract pipeline, and it contains collections of yaml files, scripts, and configuration files. These files are referenced from the service pipelines. Templates from Azure Pipelines are used to build the master pipeline for our project. Templates let you define reusable content, logic, and parameters. We will have one master pipeline and multiple service pipelines for their respective services.
Figure 3
Master & service pipeline relationship
Many service pipelines use the same master pipeline thus
avoiding the code duplication. The service-pipeline.yaml from the service uses
the main-pipeline.yaml from the master pipeline. The master-pipeline.yaml
contains the abstract flow of stages, steps etc. Where the service-pipeline.yaml
simply invokes the master-pipeline.yaml. Each service contains its own
service-pipeline.yaml.
Setup repositories & Sample Project
In our sample setup we have two repositories in azure devops,
‘Master.Pipeline’ and ‘Service.Pipeline’. We require one repository for Master
and one repository for service to logically create a boundary between two
pipelines. If we have more than one services, we have to place each services
project in a separate repository. The Master.Pipeline repository contains the CI
templates (yaml files), scripts and master-pipeline.yaml. The
‘Service.Pipeline’ repository contains the sample project. Our sample project
is a simple web API project created from microsoft project templates.
Master repository
As mentioned earlier the master pipeline isn’t a pipeline on
its own. We are not going to configure any pipeline for master. The
‘Master.Pipeline’ repository contains the collections of yaml templates and
scripts etc. Each piece of build process is abstracted and separated into a
yaml files and collected as templates in master repository. Similarly, any
script files which assist in the process of build will also be collected within
the master repository.
The below screen grab shows the structure of master repository,
Figure 4
Master repository structure
The ‘Master.Pipeline’ repository contains the
‘service-pipeline’ folder which enclosed the files meant for service pipeline.
The master repository may contain other pipelines for other type of project
like ‘nuget-pipeline’ for building nuget packages. The pipeline files are
versioned in a separate folder ‘v1’. More on versioning the pipeline is
discussed in the below article. There are two folders ‘templates’ contains the
yaml templates and ‘scripts’ folder contains any build scripts such as
powershell scripts, bash scripts, and python scripts etc. Last there is a master-pipeline.yaml
which is the core file for master pipeline that contains the master flow for
the CI process. More on the flow will be discussed below in ‘Master Pipeline’
section.
Service repository
The service repository contains the sample web API project. In
our sample I choose web API project but it may contain any service projects
like web app, web jobs etc.
The below screen grab shows the structure of service repository,
Figure 5
Service repository structure
The ‘Service.Pipeline’ repository contains two folders
‘build’ and ‘src’ and a file ‘service-config.yaml’ at root level. The ‘build’
folder contains the file ‘service-pipeline.yaml’ which is the yaml file to be
configured in azure pipeline. The ‘src’ folder contain the source code for our
sample project. The file ‘service-config.yaml’ file the is the configuration
file for service pipeline. Any configurations that is required for the service
pipeline is included in this file. More on the service-config.yaml is discussed
in the ‘Service Configuration’ section.
Now we have the basic setup of infrastructure, we can take a
closer look at how the templates are abstracted and a generic CI process flow in
master pipeline. We also look at how the service pipeline invokes master
pipeline and how we can pass the configurations to master pipeline through
‘service-config.yaml’ instead of doing any hardcoding the values.
Master Pipeline
The master pipeline contains the collections of template and
master flow. The master flow is defined in ‘master-pipeline.yaml’.
The below code shows the parameters section,
parameters:
- name: checkoutRepos
type: stepList
default: []
The checkoutRepo parameter takes the steps list and the steps
are defined from the child pipeline.
The next section of code shows the different stages in the
flow,
stages:
-
stage: BuildDotNet
displayName: Build
Dot Net
jobs:
- job: RSGBuildDotNet
displayName: Read
Service Configs
steps:
- ${{ each repo in parameters.checkoutRepos }}:
- ${{ each step in repo }}:
${{ step.key }}:
${{ step.value }}
Here we are executing the steps from the checkoutRepos
parameter. Here we are doing two things one, it reads all the service configurations
and check-out the projects from the repository.
The next section has the job to build the dotnet project,
- job: BuildDotNet
dependsOn: RSGBuildDotNet
#condition: eq(
dependencies.RSGBuildDotNet.outputs['sc.pipeline.build'], 'true')
variables:
solution:
$[
dependencies.RSGBuildDotNet.outputs['sc.param.solution'] ]
buildConfiguration:
$[
dependencies.RSGBuildDotNet.outputs['sc.param.buildConfiguration'] ]
displayName: Build
dotnet core project
steps:
- ${{ each repo in
parameters.checkoutRepos }}:
- ${{ each step in repo }}:
${{ step.key }}:
${{ step.value }}
#Prepare dotnet core project
- template:
templates/prepare-dotnet-core.yaml
#Build dotnet core project"
- template:
templates/build-dotnet-core.yaml
parameters:
solution:
$(solution)
buildConfiguration:
$(buildConfiguration)
This section is split into multiple templates,
prepare-dotnet-core.yaml and build-dotnet-core.yaml files, where the actual
build code for dotnet projects exists.
Here the sample project is only dotnet projects and if the
project is multiple technologies the flow can be controlled using conditions. The
master pipeline flow will become more complex if with more parameters like
multiple branches and technologies.
Service Pipeline
The service pipeline contains the service specific pipeline
code and invokes the master pipeline and passed necessary parameter. You can
find the pipeline code at ‘build\service-pipeline.yaml’ in the ‘Service.Pipeline’
repository.
We will now look at each section of code in
service-pipeline.yaml file. In the first part of the code we defined the
resources to be used in our pipeline.
resources:
repositories:
- repository: self
type: git
name: Service.Pipeline
- repository: master
type: git
name: Master.Pipeline
Two repositories ‘Service.Pipeline’ and ‘Master.Pipeline’
is defined in resources section. Here the master repositories contain the reusable
templates and scripts. The service pipeline will make use of it.
In the next section of the code we had included the trigger,
how and when the pipeline should be triggered whenever a code commit event
occurs.
trigger:
branches:
include:
- main
In the above code we have mentioned that the pipeline should
be trigged when there is a commit happened in ‘main’ branch. In a
complex repo there might be multiple branches and also the paths should also be
considered.
We also defined the agent pool to be used for our pipeline
executing,
pool:
vmImage: 'ubuntu-latest'
Here I had used ubuntu virtual image.
In the next section we have defined few variables that will
affect the flow of our master pipeline.
variables:
mainBranchTriggered:
$[eq(variables['Build.SourceBranchName'], 'main')]
serviceConfigFile:
"service-config.yaml"
Here we capture the status if the main branch got trigged.
This code ‘$[eq(variables['Build.SourceBranchName'],
'main')]’ checks if the trigged branch is ‘main’ and assigns the
variable ‘mainBranchTriggered’ to true
or false. With this variable the master pipeline flow is controlled if there is
any task or steps to be performed specific to main branch.
In the next section we have the stages defined. In the below
code the master pipeline is invoked by the service pipeline and passed few
parameters.
stages:
- template: service-pipeline/v1/master-pipeline.yaml@master
parameters:
checkoutRepos:
The parameters are lists and are executed in master
pipeline.
The first parameter is where we create a folder for both the
master and service repositories.
task:
PowerShell@2
displayName:
"Repository folder creation"
inputs:
targetType:
inline
script:
|
New-Item -Path . -Name "$(Build.Repository.Name)"
-ItemType "directory" -Force
New-Item -Path . -Name Master.Pipeline
-ItemType "directory" -Force
- checkout:
self
path:
./s/service-repo
clean:
true
- checkout:
master
path:
./s/master-repo
This step enables to write the consistent folder structure across
the service and master pipelines.
The next parameter is where we read the service
configurations.
- task:
PowerShell@2
inputs:
targetType:
filePath
filePath:
master-repo/service-pipeline/v1/scripts/read-service-config.ps1
arguments:
>
$(serviceConfigFile)
$(mainBranchTriggered)
name:
sc
displayName:
"Read Service Configuration"
The ‘read-service-config.ps1’ script reads all the
configurations defined in ‘service-config.yaml’ file and exported to variables for
the other parts of the code to use.
Versioning Pipeline
Versioning the pipeline is important to avoid any conflict
with the existing service pipelines. There are many service pipeline which are
already referring to the master pipeline. Updating the pipeline code will
create a drastic impact to the existing service pipelines. So, it is wise to
have a pipeline code to be versioned and put the new code in a separate folder.
In our ‘Master.Pipeline’ repository the cod is versioned as
‘service-pipeline/v{N}’, where N is the version number. It can be versioned as
v1, v2 and so on. Any upgradation to the pipeline should be put into a new
versioned folder.
Below is the structure of how the versioned pipeline is invoked from the service,
Figure 6
Pipeline Versioning
One of the disadvantages of this approach is we may have to update
all our services to refer to new version. Any update to the master pipeline may
not readily available to the services. This may be critical if the pipeline
contains the stages that scan the code for security vulnerabilities. Considering
the fact that all the service-pipeline may goes down whenever a wrong code update
was made in master pipeline, it is better to update the service pipeline to
newer master version independently.
Service Configuration
The service configuration is where we pass the dynamic
values to the master pipeline. The master pipeline uses the service
configuration file and controls the flow of the build process. The
service-config.yaml file is shown below,
parameter:
repoName: 'Service.Pipeline'
solution: '**/*.sln'
buildPlatform: 'Any
CPU'
buildConfiguration:
'Release'
projectName: 'Service.Pipeline'
system.Debug: false
The yaml file contains few properties like repository name,
solution path and few build configurations. The pipeline uses ‘read-service-config.ps1’
script to read the parameters and pass on to the master pipeline.
Below power shell script shows reading the yaml file,
$serviceConfigFile=$args[0]
$serviceconfig
= [IO.File]::ReadAllText("service-repo/$serviceConfigFile")
$parsedYAML
= ConvertFrom-Yaml
$serviceconfig
-AllDocuments
and the below code snippet shows, a single value read from
the parsedYaml object and posted to pipeline,
$value
= $parsedYAML.buildParameter.repoName
Write-Host
"##vso[task.setvariable
variable=param.repoName;isOutput=true]$value"
Configuring Service Pipeline in Azure DevOps
Configuring the service pipeline is straight forward. Go the azrue devops portal and select Pipeline -> New Pipeline -> Azure Repos Git -> ‘Service.Pipeline’ (repo where the service pipeline yaml file is located) -> Existing Azure Pipelines YAML file and choose the yaml file ‘build\service-pipeline.yaml’ from the drop down. Below is the final screen shot,
Figure 7
Service pipeline configuration
Note: I
have omitted the screen shot for every step. It can be easily understood, or
you can refer to the Microsoft Azure Pipeline documentation for more
information on how to configure the pipeline.
One the pipeline is configured you can either trigger the
pipeline manually or it will be automatically triggered when ever when there is
a code commit to the repository.
Running and Testing the Pipeline
The below screen shot shows the final execution summary of service pipeline,
Figure 8 Service pipeline execution summary
Nice article with detailed explanation
ReplyDelete