Azure First Community

loading...
Cover image for 3 Ways to run Automated Tests on Azure DevOps

3 Ways to run Automated Tests on Azure DevOps

preetham profile image Preetham Reddy Updated on ・13 min read

tech-fabric-example-schema
tech-fabric-example-schema
Before we start, let’s imagine the situation where you are talking with your colleagues after a good weekend in your office kitchen and the manager runs into the room shouting ‘BUG, WE HAVE A BUG!”.

Alt Text

‘Bugs are everywhere’ – you think, trying to fill the water bank on your office’s coffee machine, but the situation changes since the manager continues with ‘WE HAVE A BUG IN PRODUCTION!’.

So, what will you do if your team misses some bug in production? What will you do if your manager declares the Bug Hunting as a primary goal for your team in the name of the Client? What will you do, if once you come into the office and see the new poster ‘WANTED Bug Bounty Program’ on your team dashboard?

Alt Text

Well, the last question seems to be off the topic, but we’ll try to answer at least the other ones about quality processes and how we can save a lot of money without hiring dozens of engineers for manual support.

The answer we suggest is – Automation, providing some coverage for your Project with Unit, API, UI Tests allows to ensure that important functionality doesn’t break after the regular code changes. Because in this way you will save the time or/and money in one or another way, that’s why we discuss how can we use pre-scripted tests on an application.

Alt Text

Defining the chapters, you will find how to:

  1. Setup a Demo Project for our Tests
  2. Run the Tests for CI Pipeline [is the first way to run automated tests in a project, and a first barrier of your QA defence which contain Unit Testing coverage].
  3. Run any Tests for CD Pipeline [if the second way, which contains UI or/and API Testing coverage].
  4. Run your Tests from Test Plans on Demand [is the third, and a final way, which gives possibility to run any test you want whenever you are].

Alt Text

Having a little remark, I believe it’s important to say, that Setup Chapter for a Demo Project is made with the idea to create some basis on which we will add our Automated Tests, so you can use your own project instead or download already created project with the following github link:

Now we are ready to start, so let’s the battle begin!


Chapter I – Setup a Demo Project for our Tests

The idea for this chapter is to create a Demo Project for the further needs, as the last warning - if you have your own project, you can continue with the Chapter II, implementing the testing coverage. Otherwise, welcome to the Chapter instructions:

  1. Open your console or terminal
  2. Create a folder for your solution:

      mkdir TechFabricSln
    
  3. Create folders for Main and Test Projects

      mkdir src
      mkdir test
    
  4. Create a Project in the ‘src’ folder & build the Project

      cd src
      dotnet new webapp -n TechFabricSln
      cd TechFabricSln
      dotnet build
    
  5. Create a Test Project in the ‘test’ folder & build the Project

      cd..
      cd..
      cd test
      dotnet new nunit -n TechFabricSln.Test
      cd TechFabr*
      dotnet build
    
  6. Create solution for your projects:

      dotnet new sln --name TechFabricSln
    
  7. Add your Projects to solution:

      dotnet sln TechFabricSln.sln add src/TechFabricSln/TechFabricSln.csproj test/TechFabricSln.Test/TechFabricSln.Test.csproj
    
  8. Add a reference for Test Project:

      dotnet add test/TechFabricSln.Test/TechFabricSln.Test.csproj reference src/TechFabricSln/TechFabricSln.csproj
    

At the end, you will have some structure like this:
Alt Text

The only difference is the name of the Project, the diagram shows you the example of how it can be used. To make an analogue for your project – replace the ‘TechFabric’ as proposed in graph or ‘TechFabricSln’ as in the example with your Project’s name.

So now, lets add a simple function to our project, which we can cover and test with our unit-tests:

  1. Add an additional class for the Main ‘TechFabricSln’ Project, in our example we will add ‘Bought’ class:
    Alt Text

  2. Write some functions which you want to test. The following code defines a new bool variable which verifies who bought something in our shop.
    Alt Text

  3. Now, let's create a quick check for declared variable. To do this, create an additional [Test] in our Test Project, and write down the code which verifies something you would like to test. In the example we verify ‘Bought’ class with a ‘isBoughtBy’ method:

Alt Text

As a final step, all that you need to setup is to create a GIT repository, and push your code. Be a team member and track yor changes in source control tool, but not on your local machine!

Well, the instructions are done, so now is the time to run your Unit tests and verify if everything good or not.

Alt Text

If you see the green lines, you can be pretty sure, that you made the first step in protecting your project from uninvited guests.

Alt Text

The following step is to go to the main parts of our topic, and create your first Azure DevOps Pipeline.


Chapter II – Create a build and run the UNIT Tests on Continuous Integration

Alt Text

Performing an automated testing as a part of building pipeline is a good way of verifying unexpected problems before pushing the build on some Test or Customer environments.

In this chapter you will find how to create a build procedure which will include your test runs and made a good basic fundament for the further quality assurance processes.

No more talks, lets start with creating your first Build pipeline:

  1. Login in Azure DevOps
  2. Go to Pipelines -> Builds
  3. Press on New pipeline: Alt Text
  4. Connect to your source control tool: Alt Text
  5. Select your Project’s repository: Alt Text
  6. Select ASP.NET Core Template: Alt Text
  7. Add the following command to run the Tests from Test Project into the yml file:

      - task: [email protected]
        inputs:
          command: test
          projects: '**/*Test/*.csproj'
          arguments: '--configuration $(buildConfiguration)'
    

    In fact Azure DevOps will read configuration file, and execute the steps as described.
    Alt Text

  8. Press Save and Run button, with commit directly to the master branch.
    After the build procedure is done, we can see the builds results in Logs. Especially, we are interested in our DotNetCoreCLI command, so let’s check it.
    Alt Text
    You can see, that in fact, nothing more than ‘dotnet test’ command is used in our DotNetCoreCLI task.
    Detailed information about our Test Run for the build can be found in ‘Tests’ block of the build:
    Alt Text
    Also, you can check detailed report about all your test runs in Test Plan -> Runs
    Alt Text
    Now, your Build Pipeline contains at least one Unit test, which works every time you run the Build, and verifying if are there any unexpected changes or not.
    And from this point, we can move to the next chapter, and Create Continuous Delivery Pipeline, which includes some UI Automated Tests on Selenium.


Chapter III – Run UI Selenium Tests in Continuous Delivery Pipeline

As soon as the build is done and Unit – Tests are passed on the Build Workflow, it’s a common thing for a quality control or development teams to create and support some functional (UI) tests in release workflow after the app is deployed to some test environment.

Alt Text

That’s why in this chapter, we will implement some UI Tests based on a Selenium framework [Selenium is an open source test framework for web applications which supports any popular browser and can be run on almost every operating system], using the UI Test framework additionally to your Unit or API tests you will be able to detect any changes also on front-end of your application.

Deploy your WEB App

To run Selenium Test for your own project, you need to Deploy it’s in continuous deployment (CD) release pipeline and Publish it on Azure, we won’t spend time on this in the post, and if you’re interested in - you can find the details here: Publish Web App to Azure or write us a message, and we’ll create a separate article with detailed description.

In this chapter, we will run the tests against Mircrosoft.com, and you will find how to:

  1. Create UI Test using Selenium Test Framework.
  2. Create a Release Pipeline using your CI from the Chapter II.
  3. Run created UI Tests on Azure DevOps CD.
  4. Publish and monitor Test Results.

ADD SELENIUM TO THE TEST PROJECT

Till this moment, we already have continuous integration (CI) build pipeline with running Unit Tests, so all we need is to add Selenium References, Driver and Test for existing Project. To do this:

  1. Open the Solution in IDE;
  2. Go to the .Test Project -> Manage NuGet Packages;
  3. Add additional Packages to your .Test Project:

    • Selenium.WebDriver.ChromeDriver;
    • Selenium.Support;
    • Selenium.WebDriver;
    • Microsoft.TestPlatform.TestHost;

    The list of installed packages at this moment is:
    Alt Text

  4. Add new Class for Selenium Tests ‘SeleniumTest.cs’ to the Project

  5. Add Selenium Test in SeleniumTest.cs class.

    As an example, we will add the Test which finds the Microsoft Page, and verifies which page contains the ‘Windows’ menu. The code is:

      using NUnit.Framework;
      using OpenQA.Selenium;
      using OpenQA.Selenium.Chrome;
      using System.Threading;
    
      namespace TechFabricSln.Test
      {
        class SeleniumTest
        {
          [Test]
          [Category("UITests")]
          public void VisitMicrosoft_CheckWindowsMenu()
          {
            IWebDriver driver = new ChromeDriver();
            driver.Navigate().GoToUrl("https://www.microsoft.com/“);
            Thread.Sleep(10000);
            string Windows_text = driver.FindElement(By.Id("shellmenu_1")).Text;
            Assert.AreEqual("Windows", Windows_text);
            driver.Quit();
          }
        }
      }
    
  6. Add Publish chromedriver for your Test Project into .csproj file:
    Alt Text

      <PropertyGroup>
        ...
        ...
        <PublishChromeDriver>true</PublishChromeDriver>
      </PropertyGroup>
    

    Alt Text
    We need this option to get chromedriver into the artefacts after the project is published.

  7. Run Selenium Test locally to check if it works.

    **MAKE CHANGES INTO BUILD PIPELINE FOR UI TESTS

    In some cases on the road to hell automation you can face with obstacles in verifying the corrects versions of driver & packages, version for NuGet, Builder and/or .Net core.

    Alt Text

    That’s why we provide some hints in the form of code pieceswhich you can use in your Build procedure to avoid the obstacles you faced with:

    trigger: 
    - master 
    
    pool: 
      vmImage: 'windows-2019' 
    
    variables: 
      buildConfiguration: 'Release' 
    
    steps: 
    
    - task: [email protected] 
      inputs: 
        packageType: 'sdk' 
        version: '3.0.x' 
        includePreviewVersions: true 
    
    - task: [email protected] 
      inputs: 
        versionSpec: '5.1.0' 
        checkLatest: true 
    
    - task: [email protected] 
      inputs: 
        command: 'restore' 
        restoreSolution: '**/*Test/*.csproj' 
        feedsToUse: 'select' 
        noCache: true 
    
    - task: [email protected] 
      inputs: 
        command: test 
        projects: '**/*Test/*.csproj' 
        arguments: '--configuration $(buildConfiguration) --filter TestCategory=UnitTest' 
         
    - script: dotnet build --configuration $(buildConfiguration) 
      displayName: 'dotnet build $(buildConfiguration)' 
    
    - task: [email protected] 
      displayName: 'dotnet build test' 
      inputs: 
        projects: '**/*Test/*.csproj' 
        arguments: '--runtime win-x64' 
      continueOnError: true 
    
    - task: [email protected] 
      displayName: 'dotnet publish --configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)' 
      inputs: 
        command: publish 
        publishWebProjects: false 
        projects: 'src/TechFabricSln/TechFabricSln.csproj' 
        arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)/TechFabricSln/' 
        zipAfterPublish: false 
    
    - task: [email protected] 
      displayName: 'dotnet publish test --configuration $(buildConfiguration) --output $(Build.ArtifactStagingDirectory)' 
      inputs: 
        command: publish 
        publishWebProjects: false 
        projects: 'test/TechFabricSln.Test/TechFabricSln.Test.csproj' 
        arguments: '-p:PublishChromeDriver=true --runtime win-x64 --output $(Build.ArtifactStagingDirectory)/TechFabricSln.Test/' 
        zipAfterPublish: false 
    
    - task: [email protected] 
      displayName: 'publish artifacts' 
    

    CREATE RELEASE PIPELINE WITH INCLUDED TESTS

    As soon as the preparation is done, and the build is ended successfully, check the chromedriver.exe located in artefacts of your build. It’s an important thing because without the driver, you won’t be able to complete written UI tests successfully.

    Alt Text

    Then make changes into your Release Pipeline or create a new one:

    1. Open the Releases page in the Azure Pipelines section
    2. Click on New pipeline button in Releases block: Alt Text
    3. In opened Templates block click to start with Empty job: Alt Text
    4. Name the stage and click on the Job/Task link: Alt Text
    5. Add the Dotnet Core Task:
      Alt Text
      Complete the added task with the following parameters:

      • Task Version: 2;
      • Display Name: Choose any name you want, the field is just responsible for the name of procedure;
      • Command: There are several command we can find, but the one we need is custom, we’ll specify the command father in Custom command section;
      • Path to project(s): Specify the path to your Test Project’s dll, in our case it’s: **/TechFabricSln.Test/TechFabricSln.Test/TechFabricSln.Test.dll
      • Add the arguments you want to use. To set logs you can use:

          --logger:trx;logfilename=TEST.xml
        

      To run only UI tests on this Release, add:

        /TestCaseFilter:"TestCategory=UITests”
      

      Alt Text
      Also, it’s important to say, that in case one or more tests fail, the procedure will be stopped, to prevent this, we recommend to use continue on error option.

    6. Add the Publish Test Results Task
      Alt Text
      Complete the added task with the following parameters:

      • Task Version: 2;
      • Display Name: Choose any name you want, the field is just responsible for the name of the procedure;
      • Test result format: Format of test result files generated by your choice of test runner. In our case we need VSTest option;
      • Search Folder: Specify the folder path where to search for the test result files. Alt Text

Now we are ready for our first release, save the Release Pipeline and start a new release. You can do this by queuing a new CI build, or by choosing Create release from the Release drop-down list in the release pipeline.
Check results you can do in 2 ways and they are the same as in the Build Pipeline:

  • Visit the Test Logs - > Tests block;
  • Visit the Test Plan - > Runs.

CHAPTER IV – Run your Tests from Test Plans on Demand

Alt Text

The idea of this chapter is to show how to use Azure Test Plan to create linkages between manual and automated tests. Rather than using scheduled tests, running the tests on demand can be useful if you:

  • Don’t want to run all tests on build or release stages (e.g. if you want to save the time and run only some important tests on Release, and afterwards run whatever you want on demand);
  • The changes in your environment are made not by release (e.g. changes in DB);
  • To rerun some individual tests (e.g. the tests can fail on build/release stage because of some infrastructure issues, and you just want to rerun them);
  • To run the tests on a new build without it’s releasing.

You already have almost everything we need, so in this chapter we proceed with:

  • Create a Test Plan with a Manual TC to linkage with our Automated TC;
  • Create separate Release Pipeline for Test Project for running the Tests on Demand;

Create linkage between Manual and Automated TCses

  1. Go to the Test Plans and push on ‘New Test Plan’ button;
  2. Create a Test Plan with any Name and Area;
  3. Go to Test Plan – ‘Define’ block and push on ‘New Test Case’ button: Alt Text
  4. Write some TC Check, no matter which step is it. Alt Text
  5. Open your Project in IDE;
  6. Connect into the Team Services/Azure; Alt Text
  7. Add linkage with right click on the test -> Associate to Test Case, and add a TC by ID Alt Text

Create separate Release Pipeline for Test Project

For management and moderation, it’s much easier to create a different stage with procedure, then add commands to the existed one. So, from this moment, we add additional stage named DEV Test, which we will use for running the tests on Demand, and not to increase the number of commands for our usual Web Project Pipeline as it’s usually not the easiest one.

To Add new Stage, go into All Pipelines and select the one you use for CD of the Project, and start with an Empty Template as we’ve done int the previous Chapter:

Alt Text
Alt Text

Now we need to define 3 tasks:

  • VSTest Platform Installer: You need the Visual Studio Test Platform to be installed on the agent computer, and if it’s not - you must add the Visual Studio Test Platform Installer task to the pipeline definition;
  • VSTest Task: This command allows us to use vstest run command with specifying settings and parameters for our Test Run;
  • Publish Test Results: This task allows us to collect the data from test runs to make some statistical analysis and verify the ‘weakest’ areas of the App. Alt Text

Add all 3 described tasks with the following settings:

  1. VSTest Platform Installer task settings: Alt Text
  2. VSTest Task settings:
    Alt Text

  3. Use the analogue for settings of Publish Test Results command from your Release Pipeline, which has been described in Chapter III.

The common view in your CD after the all work is done, will look like this:
Alt Text

Now we can run the Tests from Test Plans chapter, so let’s try to do it:

  1. Go to Test Plans;
  2. Select ‘Execute’ block;
  3. Select a TC you want to run;
  4. Choose ‘Run with options’ in the menu ‘Run for web applications’: Alt Text
  5. Choose the build and Stage where you want to run your UI Selenium tests. Use the latest build, and choose ‘Dev Tests’ Stage which we just created. Alt Text
  6. Push ‘Run’ button, and let's wait for result. The system will check if only automated tests are selected, validate the stage to ensure the VS Test task is present and has valid settings. After the validation it creates a test run, and then triggers the creation of a release to the selected stage. All these processes you can see as soon as you initialize the ‘Run’: Alt Text
  7. After the Test execution is complete, visit the Runs page and check for the test Results. The Test results page lists the results for each test in the test run, and if everything goes good you will find a prize from Azure saying that there are no test failures. Alt Text Alt Text

Summary

  • Setup a WEB Project with Unit Tests;
  • Create a CI in Azure DevOps;
  • Add Selenium Tests on existing Projects;
    • Run all the tests locally, in your own IDE.
  • Run the Tests in 3 Ways on Azure DevOps:
    • On each Build (for Unit Tests);
    • On each Release (for UI or API tests);
    • Create linkages between your manual and automated tests and run them whenever you want.
  • Check for Test run results.

QA or Developer Teams usually cover Testing and Staged Servers with Autotests, to check for new features, complete regression testing, and to test builds and updates to ensure quality under a production-like environment.

The advantage is once the automated tests created, they can easily be provided in several of your Test Environments.

It’s a good practice for bigger projects so save the money and time for verifying software in this way before the users find something.

Alt Text

And returning to the start, using the automation properly, and covering your code with checks, you will incredibly decrease the number of heart attacks you can have because the customers found the bug in production.

Discussion (0)

pic
Editor guide