Skip to content

Test Runners ​

Introduction ​

While Playwright for .NET isn't tied to a particular test runner or testing framework, in our experience the easiest way of getting started is by using the base classes we provide for MSTest, NUnit, xUnit, or xUnit v3. These classes support running tests on multiple browser engines, adjusting launch/context options and getting a Page/BrowserContext instance per test out of the box.

Playwright and Browser instances will be reused between tests for better performance. We recommend running each test case in a new BrowserContext, this way browser state will be isolated between the tests.

Playwright provides base classes to write tests with MSTest via the Microsoft.Playwright.MSTest package.

Check out the installation guide to get started.

Running tests in Parallel ​

By default MSTest will run all classes in parallel, while running tests inside each class sequentially (ExecutionScope.ClassLevel). It will create as many processes as there are cores on the host system. You can adjust this behavior by using the following CLI parameter or using a .runsettings file, see below. Running tests in parallel at the method level (ExecutionScope.MethodLevel) is not supported.

bash
dotnet test --settings:.runsettings -- MSTest.Parallelize.Workers=4

Customizing BrowserContext options ​

To customize context options, you can override the ContextOptions method of your test class derived from Microsoft.Playwright.MSTest.PageTest or Microsoft.Playwright.MSTest.ContextTest. See the following example:

csharp
using System.Threading.Tasks;
using Microsoft.Playwright;
using Microsoft.Playwright.MSTest;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace PlaywrightTests;

[TestClass]
public class ExampleTest : PageTest
{
    [TestMethod]
    public async Task TestWithCustomContextOptions()
    {
        // The following Page (and BrowserContext) instance has the custom colorScheme, viewport and baseURL set:
        await Page.GotoAsync("/login");
    }

    public override BrowserNewContextOptions ContextOptions()
    {
        return new BrowserNewContextOptions()
        {
            ColorScheme = ColorScheme.Light,
            ViewportSize = new()
            {
                Width = 1920,
                Height = 1080
            },
            BaseURL = "https://github.com",
        };
    }
}

Customizing Browser/launch options ​

Browser/launch options can be overridden either using a run settings file or by setting the run settings options directly via the CLI. See the following example:

xml
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <Playwright>
    <BrowserName>chromium</BrowserName>
    <LaunchOptions>
      <Headless>false</Headless>
      <Channel>msedge</Channel>
    </LaunchOptions>
  </Playwright>
</RunSettings>
bash
dotnet test -- Playwright.BrowserName=chromium Playwright.LaunchOptions.Headless=false Playwright.LaunchOptions.Channel=msedge

Retrying failed tests ​

Playwright for .NET does not retry failed tests itself, use the retries of your test runner. NUnit and MSTest have an attribute that retries a single test, and Microsoft.Testing.Platform can retry all failed tests of a run. Every attempt gets a new BrowserContext and Page.

Add the Retry attribute to a test method. It requires MSTest 3.8 or later. Its argument is the number of retries, so [Retry(2)] runs a failing test up to three times. MSTest retries after any failure, including a failed Playwright assertion and a timeout.

csharp
using System.Threading.Tasks;
using Microsoft.Playwright;
using Microsoft.Playwright.MSTest;
using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace PlaywrightTests;

[TestClass]
public class ExampleTest : PageTest
{
    [TestMethod]
    [Retry(2)]
    public async Task TestWithRetries()
    {
        await Page.GotoAsync("https://playwright.dev");
        await Expect(Page.GetByRole(AriaRole.Link, new() { Name = "Get started" })).ToBeVisibleAsync();
    }
}

Retrying with Microsoft.Testing.Platform ​

The retry extension of Microsoft.Testing.Platform reruns the failed tests of a run in a new process, without changes to the tests. It works with MSTest, NUnit and xUnit v3, but not with xUnit v2.

First, run your tests on Microsoft.Testing.Platform. xUnit v3 supports it out of the box. For MSTest, set EnableMSTestRunner in the project file, and for NUnit set EnableNUnitRunner:

xml
<PropertyGroup>
  <OutputType>Exe</OutputType>
  <EnableMSTestRunner>true</EnableMSTestRunner>
</PropertyGroup>

With the .NET 10 SDK, dotnet test also needs the following global.json file:

json
{
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}

Then add the extension to the test project:

bash
dotnet add package Microsoft.Testing.Extensions.Retry

INFO

The major version of the package has to match the version of Microsoft.Testing.Platform that your test runner uses, otherwise no tests run. For example, xUnit v3 3.x needs version 1.x of the package, and xUnit v3 4.x needs version 2.x.

Now you can retry each failed test up to two times:

bash
dotnet test --retry-failed-tests 2

Using Verbose API Logs ​

When you have enabled the verbose API log, via the DEBUG environment variable, you will see the messages in the standard error stream. Within Visual Studio, that will be the Tests pane of the Output window. It will also be displayed in the Test Log for each test.

Using the .runsettings file ​

When running tests from Visual Studio, you can take advantage of the .runsettings file. The following shows a reference of the supported values.

For example, to specify the number of workers, you can use MSTest.Parallelize.Workers. You can also enable DEBUG logs using RunConfiguration.EnvironmentVariables.

xml
<RunSettings>
  <!-- MSTest adapter -->
  <MSTest>
    <Parallelize>
      <Workers>4</Workers>
      <Scope>ClassLevel</Scope>
    </Parallelize>
  </MSTest>
  <!-- General run configuration -->
  <RunConfiguration>
    <EnvironmentVariables>
      <!-- For debugging selectors, it's recommend to set the following environment variable -->
      <DEBUG>pw:api</DEBUG>
    </EnvironmentVariables>
  </RunConfiguration>
  <!-- Playwright -->
  <Playwright>
    <BrowserName>chromium</BrowserName>
    <ExpectTimeout>5000</ExpectTimeout>
    <LaunchOptions>
      <Headless>false</Headless>
      <Channel>msedge</Channel>
    </LaunchOptions>
  </Playwright>
</RunSettings>

Base classes for Playwright ​

There are a few base classes available to you in Microsoft.Playwright.MSTest namespace:

TestDescription
PageTestEach test gets a fresh copy of a web Page created in its own unique BrowserContext. Extending this class is the simplest way of writing a fully-functional Playwright test.

Note: You can override the ContextOptions method in each test file to control context options, the ones typically passed into the Browser.NewContextAsync() method. That way you can specify all kinds of emulation options for your test file individually.
ContextTestEach test will get a fresh copy of a BrowserContext. You can create as many pages in this context as you'd like. Using this test is the easiest way to test multi-page scenarios where you need more than one tab.

Note: You can override the ContextOptions method in each test file to control context options, the ones typically passed into the Browser.NewContextAsync() method. That way you can specify all kinds of emulation options for your test file individually.
BrowserTestEach test will get a browser and can create as many contexts as it likes. Each test is responsible for cleaning up all the contexts it created.
PlaywrightTestThis gives each test a Playwright object so that the test could start and stop as many browsers as it likes.

Released under the Apache 2.0 License.