Azure DevOps — Creating Build and Running it on Self Hosted Agents

Title Image

There are multiple way how to create builds. Build processes are unique and aligned with the organizational processes. As the demand to build a software to be highly scalable and available is becoming more common, organizations are moving towards adopting distributed architectures and most popular among them is Microservice Architecture.

Though it has provided a lot of flexibility in achieving the organizational goals but it has introduced a lot of complexity in build processes. To overcome these challenges all the organizations have started adopting to DevOps. This article is not about DevOps processes so will not go in depth of it. To increase the efficiency and effectiveness of DevOps processes lots of paid and free Build and Deployment tools such as Jenkins, Bamboo, TeamCity and Azure DevOps are available.

Though build process can be different based on organizational needs but fundamentally all builds look like:

• Extracting code from source control
• Perform quality checks such as static code analysis or static code security checks
• Transform the code with compilation and/or transpilation
• Post transformation quality operations such as execute unit test and update documentation
• Packaging the output
• Create deployable artifacts

Creating build with Azure DevOps

Before jumping on self hosted agents in Azure DevOps lets start with creating builds in Azure DevOps. Go to dev.azure.com and create a new project in repository. I already have one which is build on .Net Core framework API project:

We can see on the right top there is a button “Set up Build”. We can click on it to setup initial build.

It will take us to a new screen as shown below:

Under configure your pipeline there are various templates available to setup build. If we click Show more we can see more templates available

Based on the application framework we can select pipeline models. In our case we will select ASP.NET Core.

Review the pipeline YAML, This is the default we will get it with Azure DevOps with basic YAML already available in it.

Let me go thorugh the high level understanding of this:

This YAML will trigger on the master branch of the repo. It will use the ubuntu image for the build agent. There is a single build variable configured as release. Finally it will run the step to create the dotnet build using build configuration variable.

This is the default template I have created a multistage build template for demo here:-

As mentioned able this build will execute on master branch. Pool defined here is “ubuntu- latest”. We can create our own pool for self hosted agent. We will discuss it further in the article. Then there are multiple stages defined here starting from restore –> Build –> Test — Publish –> publish artifacts. We have to assign the inputs and required arguments with each steps. Values for the argument is same as we need to run create the build locally with dotnet cli. command within input define what action does each step will perform.

For more details for creating YAML file for .Net Core framework please refer to — .NET Core CLI task

Now lets not waste much time and click Save and run:

Now this is the biggest benefit of using YAML build as we can see it provides an option to either commit directly to master or create a new branch for commit. This helps in creating multiple version of builds as an when requirement changes.

Here builds can be stored as piece of code within repo. It is a common issue we have faced is we need to support multiple versions of a software or have to provide backward compatibility if we are not using YAML based (or some other scripting language) pipelines we end with lots of issues with pipelines.

Here will will commit directly to master and click Save and run.

As we run the build it will trigger the job. A Azure hosted agent will pick up is aligned and it start run the build steps.

Once the all the build steps are executed it will change the job status to success:

We can go back to the repos and can see AzurePipeline.yml file is available there (when we are doing for the first time)

This is all about creating a YAML based build in Azure DevOps.

Set up and use self-hosted agents

What we have done so far is creating a simple build and run it. In real world that’s not case. There are instances when when we need to use different OS to build our application. To support this scenario Azure DevOps provide multiple build agents still lots of time these are not compatible with the kind of build we need. To over come this Azure DevOps provide the support of self hosted agents through which we can literally use our local machine as build agent and run Azure DevOps agent as window service at it. This is used by using Agent Pool in Azure DevOps.

Agent Pools are configured at organization level and not at project level. Go to the Organization settings and look for Agent Pool

when we click on it we will see various Pools. In my case it shows Azure Pipelines and Default:

If we click on Azure Pipelines we can see all the jobs run by ubuntu OS here as configured in pipeline

Now lets select Default agent pool and as we can see there are no jobs or agents configured in it. To start with lets click on New Agent to add a new agent to Agent pool.

As we click on New agent it will give us option to Download agent based on OS. As we can see it will give us steps how to setup agent locally.

We want to run our agent as window service so we will follow following steps

• After download open powershell in administrator mode and create a new folder where we want to maintain builds.

Here I have created a new directory and called agent and moved to that folder.

• Next we will run the following command to unzip the files in new folder

• Now we need to configure the shared access token for the agent to authenticate it from Azure DevOps. So we will go back to the portal and click on small icon up here and click Personal access tokens:

• Now we will click on New Token:

• We will friendly name to it. I have used Agent Token and set the expiration to 90 days.

• We will scroll down to Agent Pools and select read and manage. If we are not able to Agent Pools click on “Show more scopes” and then click “Create“

• Now the token is generated along with the warning that if we missed this step of copying token we will not able to recover it. and this is literally the case Azure DevOps does not keep the token. So copy the token and close the screen

• Now we go back to the powershell window and run “./config.cmd“.

• It will as for server url. It is just url of our organization. In my case it will be “https://dev.azure.com/gagan1983/”

• We will press enter and it will ask for authentication type. We are using Personal Access Token i.e. PAT which is default value so we just click enter.

• Now we will paste our token here.

• Once the token is authenticated it will communicate with the server and ask for Agent pool. If the agent pool is Default as it is in our case will press enter.

• Now it will ask for the name of the agent pool we will keep it as “LocalAgent“. Once the agent name is selected it will ask for working folder by default it is _work which is fine in our case (you can use any other name as well). So we press enter.

• Now it will ask to run the agent as service. This is what we want so we will type Y and press enter.

• Next we will again press enter to select the Network Service account

• Now we will wait for the command to get executed. As we open services.msc and check our agent will be running as a service.

• Now when we get back to portal and see our default agent pool we can see our LocalAgent is “Online” but status as “Idle“

• Now we will go back to _work folder. As we can see its empty because we have not run any build using LocalAgent so far.

• Now we go back to portal and Edit our build:

• As we can see pool is currently pointing to ‘ubuntu-latest‘ image.

• We will update the pool value to default here:

• Now we will save and run it. As wen can see a job is created and assigned to the agent pool

• Before moving ahead it will ask for permission to access the default queue. Click “Permit“

• Once the permission is provided it will trigger the build immediately.

• It will run through all the build steps successfully.

• Once all the steps are completed we can go back to _work folder and see it not empty anymore. Our first build with name “1” is already created in it.

• If we open the “1” folder we see there are “a”, “b” and “s” folders are there.

• Here S folder stand for the source where complete source code is kept.

• If we move the bin folder we can see the output.

• Inside “1” folder, “b” folder stand for binaries and “a” stands for artifacts where final artifacts goes.

Now we are done with creating a self hosted agent, here we have used our local machine to run self hosted agent but we can use multiple VMs available in cloud to do so. Though Azure DevOps provide agents but they are based on availability and we have to wait if they are not immediately available also, if we need incremental builds its better to use self hosted agents and Azure Pipelines.