Test Management Tool Connectors: VSTS 2017 / TFS 2017

Versions:3.6+ 

Setting up QuerySurge Test Management Tool Connectors with VSTS 2017

The Scheme:

QuerySurge's Test Management Tool Connector works with TFS's Build Agent to enable you to execute QuerySurge tests from TFS, and for results from QuerySurge to flow back to TFS. The following scheme shows the relationships between these components:

Prerequisites:

• QuerySurge installation
• QuerySurge Test Management Tool Connectors Installation
• Visual Studio Team Services Project
• Visual Studio 2017 Client
  
  

Test Management Tool Connector Configuration

The installation of the Test Management Tool Connectors is a prerequisite for the setup. If you haven't installed the Connectors yet, you can find the installation steps here. 

Note: The Connectors are installed using the same QuerySurge installer that you used to install QuerySurge. Use the same installer version as your installed QuerySurge.

Note: The QuerySurge Test Management Tool Connectors must be installed on the same machine as your Visual Studio 2017 Client.

1. Once your Test Management Tool Connectors are installed, open Windows Explorer and navigate to the install directory. In your Connector installation directory, open the qsapi.properties file, which, if you chose the default options while installing QuerySurge, is in the C:\Program Files\QuerySurge\cli folder. You can edit this file with a text editor, however you may need elevated permissions.

2. In the file, insert a line right after # Client API and type the UsernameYouWantToUseFromQuerySurge=PasswordOfUsernameInQuerySurge

Visual Studio Team Project Setup

1. In an existing or new Team project, make sure that Version Control is set to Team Foundation Server Control:

2.  You can close out of the next screen (use the 'X' icon in the upper right).

3. You will then be taken to the Team Project home dashboard. In the bottom right corner, click on the Open in Visual Studio link.

Visual Studio Unit Test Setup

1. Under Solutions, click on the New link to create a new Visual Studio Solution.

2. Select Unit Test Project to create a new Visual Studio Unit Test Project.

Note: Your Unit Test name needs to match the Test Assembly pattern (see Step 8 in the section "Create Visual Studio Team Services Build Verification Test" below). You can either modify your Unit Test naming convention to match the default pattern (see Step 8 in the section "Create Visual Studio Team Services Build Verification Test" for the default pattern) or you can modify the pattern to match your Unit Test naming.

3. On the Visual Studio client machine (where you installed the QuerySurge Test Management Tool Connectors), locate the Connector installation directory. In this directory, you'll find a samples/ subdirectory containing sample .cs files for use with the Connector. In the default installation, the path is: C:\Program Files\QuerySurge\cli\samples. The sample files for this setup are: TFS_ScenarioBySuiteId.cs and TFS_ScenarioBySuiteName.cs. You will need to modify the following string variables in these sample .cs files for your usage: userName, scenarioName, suiteIdList, and outputPath. The following shows what the modified lines of your .cs should look like:

4. Open the CS TFS_QuerySurgeConnection.cs file in your Visual Studio Project and update the QuerySurgeInstallationDirectoryPath, hostName, and port string variables appropriately. 

5. Check In the updated .cs files to your Team Project in Visual Studio.

Configure Visual Studio Test Agent Machine

1. Navigate to your Visual Studio Team Services Project (the project is named "QuerySurge_BVT_Test" in this example). Use the Settings menu item to Navigate to Agent queues.

2. On the Agent Queues page, start the procedure to download, install and configure a VSTS agent. Click on the Download agent link for the instructions.

3. Linked instructions to download, install and configure the VSTS agent are shown:

Note: When you install the agent, it is important that you install the agent as a service.

Note: Part of the agent setup is done with PowerShell. Your PowerShell Version must be 4.0 or greater to complete this process.

4. Once agent set up is finished go to Start -> Services and make sure VSTS Agent is started.

5. In the Services console, right click on VSTS Agent and select Properties. Click on the Log On tab and make sure Local System account is selected.

6. Navigate back to your Visual Studio Team Services Project, and return to the Agent Queues page. You should now see your VSTS agent:

Create Visual Studio Team Services Build Verification Test

1. Navigate to the Build menu and click the New button.

2. Under the Create New Build Definition window select Empty, Start a definition that has no steps. Click the Next button.

3. Under Default agent queue select Default from the dropdown menu to indicate that you wish to use the Visual Studio Test Agent machine was configured earlier. Click the Create button.

4. Click the Add build step… button to open the Task Catalog.

5. Navigate to the Build category and Add the Visual Studio Build task.

6. Navigate to the Test category and Add the Visual Studio Test task and Click the Close button.

7. In the Visual Studio Build build item and apply the appropriate solution information for your project in VSTS.

8. Navigate to the Visual Studio Test build item and apply the appropriate Test Assembly and Test Filter Criteria information for your project in VSTS.

Note: The unit tests that are used in this example have the test method names of QuerySurgeConnectionBySuiteIdTest and QuerySurgeConnectionBySuiteNameTest.

Note: See Step 2 in the section "Visual Studio Unit Test Setup" above. Your Unit Test name in Step 2 of "Visual Studio Unit Test Setup" needs to contain the sub-string 'test' to be recognized by the default Test Assembly pattern in this step (**\*test*.dll;-:**\obj\**). Alternately, you can change the Test Assembly pattern here to match your Unit Test naming convention.

9. Click the Save button to save the Build Definition and apply an appropriate name to the Build. Click the OK button.

10. Once prepared to create a new build and execute associated tests, click the Queue New Build button.

11. Confirm that the Default Queue item is selected from the dropdown menu and click the OK button.

The build process will execute and the following Summary screen will be shown which includes Build Item Status, Run Time, etc.:

Optional: Run Tests from the Test Hub

1. In order to run tests from the Test Hub, follow the setup instructions from this link:

https://www.visualstudio.com/en-us/docs/test/continuous-testing/run-automated-tests-from-test-hub

Troubleshooting

Note: The steps in this section can be done with any PC on you corporate network that has Microsoft Visual Studio installed. Some steps in this section refer to the install directories for the QuerySurge Host, so please make sure that you have access to a QuerySurge Host installation.

A TEST FAILS WHEN RAN FROM VISUAL STUDIO TEAM SERVICES AND RUN STATUS DOES NOT EQUAL "PASS" OR OFFERS NO LOG SHOWING THE FAILED QUERYPAIRS

The generated QuerySurge HTML results attachment will be made available on Build Verification Test failure to provide more details on the current test run.