DEV Community: Hakan GÜL

TestFly Part 11: Network interception and API mocking in Selenium 4 with CDP

Hakan GÜL — Sun, 18 Oct 2026 18:54:14 +0000

Network mocking has long been treated as something only Playwright does. TestFly adds Chrome DevTools Protocol (CDP) interception and network throttling to Selenium 4 in Java. You can simulate offline mode, throttle the connection to slow 3G, stub third-party API responses and fail tests automatically on browser JavaScript errors, all without an external proxy server.

Why frontend tests need network interception

A single page application (SPA) talks to many backends: distributed microservices, third-party payment gateways, analytics trackers. In traditional Selenium tests this caused trouble:

To test an edge case such as a 500 Internal Server Error or a slow payment response, you had to change backend database state or configure a proxy server such as BrowserMob Proxy.
Slow third-party widgets and CDN outages made tests flaky.
JavaScript errors in the browser console went unnoticed unless an explicit assertion failed.

TestFly uses Selenium 4's native CDP bridge to intercept and modify HTTP traffic at the browser process level.

Simulating slow 3G and offline mode

Mobile-first web apps need to be tested on poor connections:

package io.testfly.examples.network;

import io.testfly.network.NetworkSpeed;
import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

public class NetworkThrottlingTest extends BaseTest {

    @Test
    public void testOfflineBannerVisibility() {
        open("/dashboard");

        // 1. Emulate instant network disconnection
        network().emulateOffline();

        // 2. Perform action requiring network
        find("#sync-btn").click();

        // 3. Verify offline warning UI appears immediately
        assertThat(find("#offline-alert")).hasText("You are currently offline");

        // 4. Restore regular network speed
        network().resetThrottling();
    }

    @Test
    public void testSlow3GSpinnerState() {
        // Emulate realistic mobile latency: 2000ms latency, 400kbps download
        network().throttle(NetworkSpeed.SLOW_3G);

        open("/heavy-report");
        assertThat(find(".loading-spinner")).isVisible();
    }
}

Mocking backend API responses (stubbing)

You can simulate server failures or return custom JSON without touching the backend environment:

@Test
public void testPaymentGateway500ErrorHandling() {
    // Intercept outgoing POST requests to payment service and return HTTP 500
    network().intercept(route -> route
        .matchingUrl("**/api/v1/charge")
        .respondWith(500, "{\"error\": \"Payment processor unavailable\"}")
    );

    open("/checkout");
    find("#pay-btn").click();

    // Verify graceful error message shown to user
    assertThat(find(".error-toast")).hasText("Payment processor unavailable, please try again.");
}

Catching silent console errors

An uncaught JavaScript exception can break the page for users even when every DOM assertion passes. TestFly's ConsoleErrorCollector watches the browser logs:

```yaml title="testfly.yml"
browser:
captureConsoleErrors: true
failOnConsoleErrors: true # Fails the test if severe JS errors are detected




When a severe JavaScript error occurs (`Uncaught TypeError`, or a `404 Not Found` for a critical script), TestFly logs the stack trace and fails the test with diagnostic context you can act on.

## FAQ

### Does TestFly require a separate proxy server like BrowserMob?

No. TestFly talks to the browser engine over the Chrome DevTools Protocol (CDP) WebSocket, so there are no extra ports, proxies or certificates to set up.

### Does CDP network interception work with headless Chrome?

Yes. Headful and headless Chrome/Chromium both support CDP network interception in CI/CD environments.

### Can I mock response headers and status codes?

Yes. The fluent `respondWith()` builder lets you override headers (for example `Content-Type: application/json` or CORS headers) and return any HTTP status code.

---

*Originally published on [hakangul.lovable.app](https://hakangul.lovable.app/blog/testfly-part-11-selenium-cdp-network-interception-and-mocking).*

TestFly Part 10: Email and 2FA testing with Mailhog, Mailtrap and IMAP

Hakan GÜL — Sun, 18 Oct 2026 18:52:39 +0000

End-to-end tests for user registration, password resets and two-factor authentication (2FA) often fail because email delivery is awkward to test. TestFly ships mailbox clients for Mailhog, Mailtrap, Outlook Graph and generic IMAP. In one automated test you can fetch incoming email, pull an OTP code out with a regex and follow the verification link.

Why email verification gets in the way of E2E tests

Almost every SaaS platform asks for email validation at signup or before sensitive operations. Automation suites usually have trouble with it:

Public inboxes such as Gmail enforce rate limits, CAPTCHAs and bot detection.
Developers often turn email verification off in test environments, so regressions in that flow reach production unnoticed.
Third-party email libraries need MIME parsing, socket handling and hand-written wait loops.

TestFly puts an email() helper on BaseTest. It hides the protocol details and runs its asynchronous polling through the central WaitEngine.

Connecting to a mailbox

Local mail catchers and enterprise cloud providers are both configured in testfly.yml:

```yaml title="testfly.yml"
email:
provider: mailhog # options: mailhog, mailtrap, outlook, imap
host: localhost
port: 8025
pollIntervalSeconds: 2
timeoutSeconds: 30




## Testing signup with 2FA / OTP in Java

The example below signs up through the web UI, waits for the verification email, extracts the 6-digit OTP code and completes the login:



```java
package io.testfly.examples.email;

import io.testfly.email.EmailMessage;
import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

public class EmailVerificationTest extends BaseTest {

    @Test
    public void testUserRegistrationWithEmailConfirmation() {
        String testEmail = "user_" + System.currentTimeMillis() + "@testfly.local";

        // 1. Fill registration form on WebUI
        open("/register");
        find("#email-input").type(testEmail);
        find("#submit-btn").click();

        // 2. Wait for confirmation email to arrive
        EmailMessage message = email().waitForMessage(query -> 
            query.to(testEmail).subjectContains("Welcome to TestFly")
        );

        // 3. Extract OTP code or activation link via Regex
        String otpCode = message.extractRegex("Your verification code is: (\\d{6})");

        // 4. Enter OTP in browser and confirm
        find("#otp-input").type(otpCode);
        find("#verify-btn").click();

        assertThat(find("#welcome-banner")).isVisible();
    }
}

Supported mail providers

Provider	Protocol	Typical Use Case
Mailhog	HTTP REST API	Local Docker environments, CI/CD runners
Mailtrap	REST API & SMTP	Staging environments, shared team sandboxes
Outlook Graph	Microsoft Graph API	Enterprise Office 365 test mailboxes
Generic IMAP	SSL/TLS IMAP	Custom corporate email servers

FAQ

Does TestFly poll the mailbox asynchronously?

Yes. It checks the inbox at a configurable interval until a message matching your criteria arrives or the timeout runs out.

Can I extract HTML links and click them directly?

Yes. message.extractLink("Confirm Email") returns the verification URL, and you can go to it right away with open(url).

Does this work with temporary throwaway email addresses?

Yes. Generate dynamic inbox aliases (for example user+123@domain.com) and filter for them with query.to(...).

Originally published on hakangul.lovable.app.

TestFly Part 9: In-process load testing in Java with Gatling and K6

Hakan GÜL — Sun, 18 Oct 2026 18:50:58 +0000

TestFly 1.1.0 adds built-in load and performance testing, powered by Gatling and lightweight virtual threads. You can write, parameterize and run high-concurrency load scenarios inside ordinary TestNG or JUnit 5 test classes, without switching to an external tool.

Why keep functional and load tests together

Performance testing often comes last in the development cycle, or it lives in a separate toolchain such as the JMeter GUI or standalone scripts. That split causes a few problems:

Test data and authentication setup get duplicated between the functional and performance repos.
Performance tests run rarely, often right before a production release.
CI/CD quality gates are hard to configure when the tests run on different runners.

In TestFly you declare load simulations with the same test classes, base configuration (testfly.yml) and assertions you already use for WebUI and API automation.

How load testing works in TestFly

There are two ways to write a load test.

1. Declarative mode with `@LoadTest`

Add @LoadTest to a test method to set the number of concurrent virtual users and the ramp-up phase:

package io.testfly.examples.loadtest;

import io.testfly.loadtest.LoadTest;
import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

public class SimpleLoadTest extends BaseTest {

    @Test
    @LoadTest(users = 100, rampUpSeconds = 15, durationSeconds = 60)
    public void testCheckoutUnderSurgeTraffic() {
        apiClient().post("/api/checkout")
            .body("{\"cartId\": 1024}")
            .send()
            .assertStatus(200);
    }
}

2. Fluent programmatic DSL with `load()`

For multi-step user journeys and dynamic feeders, use the fluent LoadScenario DSL:

package io.testfly.examples.loadtest;

import io.testfly.loadtest.LoadScenario;
import io.testfly.loadtest.LoadTestFeeder;
import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

public class AdvancedLoadJourneyTest extends BaseTest {

    @Test
    public void simulateUserBrowsingJourney() {
        LoadTestFeeder users = LoadTestFeeder.csv("testdata/users.csv");

        LoadScenario.scenario("Search and Add to Cart")
            .feed(users)
            .step("Browse Catalog", client -> client.get("/products?q=shoes").send().assertStatus(200))
            .pause(1000)
            .step("Add to Cart", client -> client.post("/cart/items").body("{\"id\": 42}").send().assertStatus(201))
            .users(500)
            .rampUp(20)
            .run();
    }
}

Checking performance SLOs in code

Metrics only help if something checks them automatically. TestFly records the timing of every request with millisecond precision and computes the standard percentiles (P50, P90, P95, P99):

Percentile Metric	Meaning	Typical SLO Target
P50 (Median)	50% of requests completed faster than this time	< 150 ms
P90	90% of requests completed faster than this time	< 350 ms
P95	95% of requests completed faster than this time	< 500 ms
P99	Tail latency affecting 1% of users	< 1200 ms

Write your service-level objectives as assertions and the build fails on its own when performance regresses:

import static io.testfly.loadtest.LoadTestAssert.*;

assertP95Below(Duration.ofMillis(450));
assertErrorRateBelow(0.01); // Less than 1% failures

Reporting

TestFly produces an interactive HTML performance report with:

live latency distribution curves and percentile breakdowns
HTTP status code distributions (2xx, 4xx, 5xx)
direct export to Allure Reports and ReportPortal, so load results show up in the same dashboards as the rest of your tests

FAQ

Does TestFly require JMeter or an external binary?

No. Simulations run in-process on Gatling's engine or on native Java virtual threads, so there is no external runtime binary to install.

Can I run load tests in CI/CD without overwhelming local machines?

Yes. Load tests run headless in CI (GitHub Actions, Bitbucket, Jenkins). For very large loads (100k+ RPS) you can schedule containerized runner pods on Kubernetes.

Can I use API authentication tokens in load tests?

Yes. TestFly's ApiClient passes Bearer tokens and session headers to the load test steps.

Originally published on hakangul.lovable.app.

TestFly Part 13: Actionability engine, auto-wait and verified click/fill/clear assertions

Hakan GÜL — Mon, 12 Oct 2026 18:53:03 +0000

TestFly 1.1.0 adds an actionability engine. Every click, hover, type, clear and append call goes through it before touching the DOM. The engine checks that the element is visible, enabled where that matters, geometrically stable and not covered by another element, and scrolls it into view if needed. On top of that, assertThat(x).click() / .fill(text) / .clear() make the action itself an assertion: they verify that the click event reached the element and that the typed text stayed in the field. When an element never becomes ready, TestFly throws NotActionableException with the element and the exact blocker in the message.

How TestFly decides an element is ready for an action

Every interaction is tagged with an ActionKind, which decides which checks apply:

public enum ActionKind {
    CLICK(true, true),   // enabled, and not covered by another element
    HOVER(false, true),  // not covered; a disabled element can still be hovered
    EDIT(true, false);   // enabled; being covered doesn't matter for keyboard input
}

Every kind requires the element to be visible and stable, whatever its flags say. WaitEngine.waitForActionable(element, kind) polls until all required checks pass. If the element drifts offscreen between polls, it scrolls it back into view:

public static WebElement waitForActionable(WebElement element, ActionKind kind) {
    TestFlyConfig.Actionability cfg = actionabilityConfig();
    if (!cfg.isEnabled()) {
        return kind == ActionKind.CLICK ? waitForClickable(element) : waitForVisible(element);
    }
    // visible, enabled (if required), stable bounding box, not covered (if required)...
}

A stale element surfaces as StaleElementReferenceException. The caller (the Locator chain) then re-resolves it from scratch instead of failing on a detached node.

What gets checked, and in what order

Check	Applies to	Detail
Visible	all kinds	element has a non-empty bounding box and isn't `display:none`/`visibility:hidden`
Enabled	`CLICK`, `EDIT`	skipped for `HOVER`; a disabled button can still be hovered
Stable	all kinds	bounding box unchanged between two consecutive polls (default poll interval: 100ms)
Not covered	`CLICK`, `HOVER`	nothing else receives a hit-test at the element's centre; skipped for `EDIT`
Auto-scroll	all kinds	scrolls the element to the viewport centre if it's offscreen when checked

The stabilityIntervalMs setting (default 100ms) is both the polling interval and the minimum time the bounding box must stay put. Playwright uses the same idea to wait out animations; TestFly adapts it to Selenium's WebElement.

What happens when an element never becomes actionable

public class NotActionableException extends TimeoutException {
    public NotActionableException(String message, Throwable cause) {
        super(message, cause);
    }
}

It extends TimeoutException, so existing catch (TimeoutException e) handlers keep working. The message names the element and the specific blocker, for example the overlay covering the centre point, or a bounding box that kept moving. A test never silently clicks the wrong spot or types into a field that is still animating.

Configuring or bypassing the checks

# testfly.yml
actionability:
  enabled: true           # false restores the pre-1.1.0 waits
  stabilityIntervalMs: 100
  hitTargetCheck: true
  autoScroll: true

public static final class Actionability {
    private boolean enabled = true;
    private int stabilityIntervalMs = 100;
    private boolean hitTargetCheck = true;
    private boolean autoScroll = true;
}

With actionability.enabled: false, TestFly falls back to the legacy waits: waitForClickable for clicks and waitForVisible for everything else. This helps during a migration, or when a page really needs the old behaviour.

For a single locator, Locator.force() skips only the stability and not-covered checks (like Playwright's force: true). The element still has to be visible, and enabled where the action needs it:

find("#banner-close").force().click();

BasePage.doubleClick, rightClick, selectBy* and BaseConditions.click/type all go through the same waitForActionable call, so the behaviour is the same across the whole API.

How `assertThat(x).click()` verifies that a click landed

A plain Locator.click() runs the actionability wait and clicks, as before. assertThat(x).click() also turns the action into an assertion:

assertThat(loginButton).click();

It resolves the element the same way Locator.click() does, including self-healing, and waits until it is visible and enabled.
It scrolls the element to the centre of the viewport and moves the mouse onto it with Actions, as a real user would.
It waits until nothing else covers the element's centre point, such as a cookie banner, a modal or a sticky header.
It clicks, then asserts that the click event reached the element or one of its descendants.

On success it logs a passed Clicked: ... step. On failure you get a regular AssertionError (or a recorded soft failure in soft mode) that names the step that failed. If the click navigates away or opens an alert before the event listener can report back, the click still counts as delivered, so a page that moves on does not cause a failure. Locator.click() itself has not changed; this is an additional, stricter method on LocatorAssert.

How `assertThat(x).fill(text)` and `.clear()` verify the field changed

assertThat(userNameField).fill("admin");
assertThat(passwordField).fill(password);
assertThat(searchField).clear();

Both use the same path (fillAndVerify):

Resolve the element as Locator.fill(String) would (waiting, with self-healing) and wait until it is visible and enabled.
Wait until the field is not readonly, then scroll it to the viewport centre.
Clear it and type the text exactly as Locator.type(String) does.
Wait until the field's value equals the text and stays that way. This catches cases where maxlength, an input mask or framework state (React, Vue…) silently changed or reset what was typed.

clear() is the same path with an empty string. For type="password" fields, the value is never logged or put in a failure message; only its length is. Fields that reformat input while you type, such as phone masks, will fail this stricter assertion. For those, use Locator.fill(text) followed by assertThat(x).hasValue(formattedValue).

FAQ

Does turning on actionability checks slow every test down?

Each action waits about one extra polling interval (100ms by default) compared with the pre-1.1.0 waits, because the engine needs at least two consecutive stable polls to confirm the bounding box has stopped moving. When JavaScript is unavailable the checks fail open, so the session does not have to support JS.

Does `assertThat(x).click()` replace `Locator.click()`?

No. Locator.click() is unchanged and still only runs the actionability wait and clicks. assertThat(x).click() is a separate, verified method. It also asserts that the click event reached the element, logs a Clicked: ... step, and supports as(...), within(...) and soft mode.

Can I combine `not()` with the verified `click()`/`fill()`/`clear()`?

No. These three methods call rejectNegation(...) and throw if not() was chained before them. They assert that an action happened and changed the page in a way that can be checked, and a "negated click" has no clear meaning.

What's the difference between `force()` and `actionability.enabled: false`?

force() works per locator and skips only the stability and not-covered checks; the element must still be visible, and enabled where the action requires it. actionability.enabled: false is global. It reverts every action in the suite to the pre-1.1.0 waits (waitForClickable/waitForVisible) and drops all the new checks at once.

TestFly Part 14: Step traces with per-step ARIA/DOM snapshots for visual debugging

Hakan GÜL — Mon, 12 Oct 2026 18:50:44 +0000

TestFly 1.1.0 adds per-step snapshots to the trace. Each step records the accessibility tree as it looked when the step started (or, with dom/both, the pruned page HTML). A failed test also records the page after its last step and shows a before/after diff. Three new config keys, tracing.snapshots, tracing.snapshotMaxKb and tracing.snapshotOnFailureAfter, control what is captured. StepRecord carries the new fields, and TraceRecorder.saveIfEnabled writes the trace from TestNG, JUnit 5 and now Cucumber. The HTML report links to the trace from every test.

What a step trace is and why every step gets a snapshot

A trace is one self-contained HTML file, target/traces/{ClassName}/{testMethod}-trace.html, that replays a test's step timeline: step names, offsets, screenshots, the final failure screenshot, the error and stack trace, and the video if recording is on. It opens in any browser without a server.

Screenshots show what the page looked like. They don't show what was in the DOM: whether a button was disabled, whether an element had aria-hidden, or whether the markup you expected was there at all. Per-step snapshots fill that gap. Next to each step's screenshot, the trace now embeds a text snapshot of the page exactly as StepSnapshotCapture saw it when the step began.

# testfly.yml
tracing:
  enabled: true

With tracing on, every step gets a snapshot by default, and the test needs no extra code.

What each step's snapshot contains

StepSnapshotCapture.captureForStep() runs once per step. It checks tracing.enabled, reads the tracing.snapshots mode, and touches the browser only if a driver session is alive. API and database tests, and steps logged before a browser exists, get no snapshot:

// StepSnapshotCapture.java (simplified)
public static Snapshots captureForStep() {
    TestFlyConfig.Tracing tracing = tracing();
    if (tracing == null || !tracing.isEnabled()) {
        return Snapshots.NONE;
    }
    String mode = tracing.getSnapshots();
    boolean aria = mode.equals("aria") || mode.equals("both");
    boolean dom  = mode.equals("dom")  || mode.equals("both");
    WebDriver driver = liveDriver();
    if (driver == null) {
        return Snapshots.NONE;
    }
    int maxKb = tracing.getSnapshotMaxKb();
    return new Snapshots(aria ? captureAria(driver, maxKb) : null,
                          dom  ? captureDom(driver, maxKb)  : null);
}

StepRecord already held a step's name, offset, status and screenshot. It now has two more fields, plus a third that is filled in later with a failed test's final state:

// StepRecord.java
public StepRecord(String name, long offsetMs, String status, String screenshotBase64,
                   String ariaSnapshot, String domSnapshot) { ... }

public String getAriaSnapshot()      { return ariaSnapshot; }
public String getDomSnapshot()       { return domSnapshot; }
public String getAfterAriaSnapshot() { return afterAriaSnapshot; } // set only on failure

aria captures the same YAML text that ariaSnapshot() prints. It is small, readable and unaffected by changing CSS class names. dom captures the page HTML after DomPruner strips scripts, styles and decoration. It is bigger, but it shows markup the accessibility tree leaves out, such as a CSS class you're debugging.

Configuring what gets captured

# testfly.yml
tracing:
  enabled: true
  snapshots: both             # none | aria | dom | both — default: aria
  snapshotMaxKb: 256           # longest snapshot kept per step, in KB — default: 256
  snapshotOnFailureAfter: true # also snapshot the page after the last step of a failed test

TestFlyConfig validates tracing.snapshots when the config loads. Any value other than none, aria, dom or both throws IllegalArgumentException at load time instead of halfway through a run. A snapshot longer than snapshotMaxKb is cut and ends with … truncated (N more characters), so one verbose page can't bloat the whole trace file.

What the trace HTML shows

Clicking a step in the timeline opens the screenshot panel with two tabs, ARIA (before step) and DOM (before step). On the last step of a failed test there are two more: ARIA (after failure) and a computed Diff tab. The diff is a plain line-based set difference between the before and after YAML, computed client-side by the trace's inlined JavaScript (lineDiff() in TraceRecorder). Lines only in the "before" snapshot are marked -, and lines only in "after" are marked +. There is no external diff library and no network call, so the trace stays a single file you can email or drop in Slack.

Step blocks in the timeline show a small 🌳 marker when they carry a snapshot, next to the 📷 marker for a screenshot, so you can see which steps have something to open.

Performance and failure risk

Capturing a snapshot takes one script call per step, a few milliseconds on a typical page, and it only runs when tracing.enabled is on. StepSnapshotCapture wraps every capture in a try/catch that returns null on failure instead of propagating the exception. A page that can't be read, a closed driver or a flaky script call leaves that step without a snapshot and does not fail the test. TraceRecorder.saveIfEnabled is wrapped the same way:

// TraceRecorder.java
@TestFlyApi(since = "1.1.0")
public static void saveIfEnabled(String testId, String testName, boolean isPassing) {
    try {
        TestFlyConfig.Tracing tracing = TestFlyContext.getConfig().getTracing();
        if (tracing == null || !tracing.isEnabled()) return;
        if (isPassing && !tracing.isCaptureOnPass()) return;
        save(testId, testName);
    } catch (Exception ignored) {
    }
}

Cucumber support

As of 1.1.0, CucumberHooks calls the same trace code that TestNG and JUnit 5 use. Gherkin scenarios get a trace file with snapshots, and your feature files and step definitions stay as they are. The HTML test report links to each test's trace from its detail panel, so one click takes you from a red row in the report to the step and snapshot that show what went wrong.

FAQ

Do I need to add code to my tests to get per-step snapshots?

No. With tracing.enabled: true, every step captures a snapshot automatically according to tracing.snapshots (default aria). There is no per-step API to call.

What happens if a test has no browser, like an API test?

Nothing is captured. StepSnapshotCapture checks for a live WebDriver session before taking a snapshot and skips the capture if there is none. It never creates or recreates a driver just to take a snapshot.

Can a snapshot capture failure cause my test to fail?

No. Every capture path is wrapped, so an exception is swallowed and logged at FINE level instead of being rethrown. Snapshots are best-effort and never affect the pass/fail result.

Is the diff tab a real structural diff of the accessibility tree?

No. It is a line-based set difference between the "before" and "after" ARIA YAML text, computed in the trace's own inline JavaScript. That is enough to spot what appeared or disappeared, but it does not compare the trees semantically.

TestFly Part 16: Smart triage and fuzzy self-healing, flaky-test diagnosis without tokens

Hakan GÜL — Mon, 12 Oct 2026 18:50:22 +0000

TestFly 1.0.6 adds a local heuristic layer that runs before any AI is consulted and costs no LLM tokens. SmartTriageEngine inspects the exception hierarchy and classifies a failure as SYSTEM_FLAKY, APPLICATION_BUG or NEEDS_INVESTIGATION in microseconds. FuzzyHealingEngine repairs a broken locator in under 5ms using plain Levenshtein distance against DOM candidates. Neither makes a network call or needs tokens or an API key.

This layer is separate from the LLM-based self-healing in Part 4 (testfly-mcp). That path is the last resort for truly ambiguous locators. The heuristic layer described here is the first resort, and it handles the 80% of healing and triage cases that don't need a model at all. The PageKnowledge / Auto-POM system connects the pieces: TestFly learns a page's elements once and reuses that knowledge locally on every later run.

Why flaky-test triage needs a heuristic layer

Every CI pipeline eventually collects red builds that nobody trusts. Re-running the test is cheap. Working out why it failed takes the time: an infrastructure hiccup, a real regression, or something a person needs to look at more closely. Before TestFly 1.0.6, triage was either manual or meant sending every failure's stack trace to an LLM and paying tokens to hear "this looks like a timeout."

SmartTriageEngine is wired into TestExecutionListener and does this triage synchronously, at no cost, as part of the failure-reporting path:

package io.testfly.flakiness;

public final class SmartTriageEngine {

    public enum TriageResult {
        APPLICATION_BUG,
        SYSTEM_FLAKY,
        NEEDS_INVESTIGATION
    }

    public static TriageResult triageFailure(Throwable exception, boolean wasRetriedAndPassed) {
        if (exception == null) return TriageResult.NEEDS_INVESTIGATION;

        // Rule 1: Immediate indicator of flakiness
        if (wasRetriedAndPassed) {
            return TriageResult.SYSTEM_FLAKY;
        }

        int flakyScore = 0;
        Throwable rootCause = getRootCause(exception);

        // Rule 2: Exception heuristics
        flakyScore += switch (rootCause) {
            case TimeoutException t -> 80;
            case StaleElementReferenceException s -> 80;
            case ConnectException c -> 90;
            case AssertionError a -> -50; // almost always an application bug
            case WebDriverException w -> 40;
            case null, default -> 0;
        };

        if (flakyScore >= 70) return TriageResult.SYSTEM_FLAKY;
        else if (flakyScore <= 0) return TriageResult.APPLICATION_BUG;
        else return TriageResult.NEEDS_INVESTIGATION;
    }
}

There is no model call, prompt or round trip. A Java 21 pattern-matching switch runs over the root cause of the exception chain, which may be wrapped several layers deep. A ConnectException scores 90 and is flagged SYSTEM_FLAKY right away. An AssertionError scores -50 and is flagged APPLICATION_BUG, because a failing assertion almost always points to a real product bug and not to infrastructure noise. Anything that clears neither threshold goes to NEEDS_INVESTIGATION, which means "a person should look at this" instead of a guess that only sounds confident.

TestExecutionListener calls this for every failed test and writes the verdict into the report:

if (result.getThrowable() != null) {
    SmartTriageEngine.TriageResult triage =
        SmartTriageEngine.triageFailure(result.getThrowable(), false);

    String triageMsg = "[Triage] " + triage;
    if (triage == SmartTriageEngine.TriageResult.SYSTEM_FLAKY) {
        System.out.println("⚠️ " + triageMsg + " -> This is likely an infrastructure/flaky issue.");
    } else if (triage == SmartTriageEngine.TriageResult.APPLICATION_BUG) {
        System.out.println("🐞 " + triageMsg + " -> This is likely an application bug (Assertion).");
    }
    StepLogger.step(triageMsg, StepStatus.WARN);
}

A test that passed on retry is flagged SYSTEM_FLAKY immediately. Rule 1 skips the exception scoring entirely, because passing on retry is a stronger flakiness signal than any exception type.

How fast fuzzy self-healing is

FuzzyHealingEngine is the fallback that runs before TestFly asks an LLM to find an element again. It is plain Java with few allocations. It computes the Levenshtein distance between clues from the locator and the attributes and text of interactive DOM candidates, scores each candidate, and picks the best match above a threshold.

package io.testfly.healing;

public final class FuzzyHealingEngine {

    private static final String INTERACTIVE_ELEMENTS =
        "button, a, input, [role='button'], [role='link'], [tabindex='0']";
    private static final int THRESHOLD_SCORE = 5;

    public static WebElement tryFuzzyHeal(WebDriver driver, By originalLocator) {
        String desc = originalLocator.toString().toLowerCase();
        List<String> clues = extractClues(desc);
        if (clues.isEmpty()) return null;

        List<WebElement> candidates = driver.findElements(By.cssSelector(INTERACTIVE_ELEMENTS));

        WebElement bestMatch = null;
        int highestScore = 0;
        for (WebElement el : candidates) {
            int score = scoreElement(el, clues);
            if (score > highestScore) {
                highestScore = score;
                bestMatch = el;
            }
        }
        return highestScore >= THRESHOLD_SCORE ? bestMatch : null;
    }
}

Clue extraction removes locator syntax noise (cssselector, xpath, contains, text, data, testid) and keeps alphanumeric tokens of 3 or more characters, all in a single regex pass. Scoring weights exact attribute matches above fuzzy text matches: an id match is worth 5 points, a placeholder or value match 4, and a CSS class match 3. When an element's text doesn't contain the clue outright, the engine allows one typo for short words (5 characters or fewer) and two for longer ones. It uses the same getLevenshteinDistance() implementation (classic single-array DP, O(n·m)) that the attribute scoring relies on:

public static int getLevenshteinDistance(String a, String b) {
    int[] costs = new int[b.length() + 1];
    for (int j = 0; j < costs.length; j++) costs[j] = j;
    for (int i = 1; i <= a.length(); i++) {
        costs[0] = i;
        int nw = i - 1;
        for (int j = 1; j <= b.length(); j++) {
            int cj = Math.min(1 + Math.min(costs[j], costs[j - 1]),
                    a.charAt(i - 1) == b.charAt(j - 1) ? nw : nw + 1);
            nw = costs[j];
            costs[j] = cj;
        }
    }
    return costs[b.length()];
}

Only elements that score at least THRESHOLD_SCORE = 5 can replace the broken locator, so one weak signal, such as a single 3-point class match, is not enough. The pass stays under 5ms on a typical DOM because it takes no screenshot, serializes no markup and makes no network hop. It only compares strings over the candidate list that findElements() already returned. TestFly escalates to the LLM-based self-healing from Part 4 only when this engine returns null.

What Auto-POM / PageKnowledge does

The third piece is PageKnowledgeStore. It saves a LearnedPageModel per URL path to .testfly/page-knowledge.json. Once TestFly, or a person through recorded runs, discovers an element on a page, every later run can reuse it instead of discovering or healing it again. That includes CI runs, as long as the file is restored.

public static Optional<LearnedPageModel> getPageModel(String url) {
    load();
    String key = normalizeUrl(url);
    return Optional.ofNullable(MODELS.get(key));
}

public static void recordElement(String url, LearnedElement element) {
    if (element == null || element.name().isBlank()) return;
    load();
    LearnedPageModel model = getOrCreate(url, null);
    model.putElement(element);
    save();
}

normalizeUrl reduces URLs to their path, so /checkout?order=123 and /checkout?order=456 share one learned model. If the store is empty on first load, KnowledgeLearner.learnFromActionCache() fills it from the existing action history, so the cache has data even on a fresh checkout.

LocalIntentResolver uses this knowledge. Given a natural-language goal and the current URL, it tries to resolve the goal from PageKnowledge alone before it considers an LLM call:

public static ActionPlan resolve(String url, String goal) {
    Optional<LearnedPageModel> modelOpt = PageKnowledgeStore.getPageModel(url);
    if (modelOpt.isEmpty()) return null;

    LearnedPageModel model = modelOpt.get();

    ActionPlan formPlan = tryResolveFormInput(model, url, goal);
    if (formPlan != null) return formPlan;

    return tryResolveSingleElement(model, url, goal);
}

It also handles a simple bilingual login pattern locally. It matches the keywords "user"/"kullanıcı"/"email"/"posta" and "pass"/"şifre"/"parola" in the goal text, along with quoted literal values, and builds a TYPE → TYPE → CLICK action plan directly from the learned username, password and submit elements without model inference. Only goals that the local resolver can't match against the learned model go on to the LLM agent path.

How the token-free layer fits with Part 4's LLM self-healing

The fallbacks form three nested rings, cheapest first:

PageKnowledge / LocalIntentResolver: if the page has been learned and the goal matches a known element or form pattern, it is resolved locally. There is no DOM re-scan beyond the cache and no LLM.
FuzzyHealingEngine: if a known locator breaks (a renamed class or a shifted DOM), live interactive elements are scored by Levenshtein distance and attribute overlap. This takes under 5ms and uses no LLM.
LLM-based self-healing (testfly-mcp, Part 4): reached only when steps 1 and 2 both fail to produce a confident match. Token cost and network latency appear only here, and this step is kept for genuinely ambiguous cases.

SmartTriageEngine works next to these rings. It classifies a failure after it happens and does not try to prevent it. It tells you quickly, and at no cost, whether the failure deserves a retry, an investigation or a bug ticket.

FAQ

Does SmartTriageEngine call an LLM to classify failures?

No. It is plain Java. It follows the exception's cause chain to the root, matches the root's type in a fixed switch expression, and adds or subtracts fixed point values. The only outside signal it uses is whether the test passed on retry.

What score does FuzzyHealingEngine require before it heals a locator?

A total of at least THRESHOLD_SCORE = 5 across all extracted clues. Exact id matches add 5, placeholder/value matches 4, class matches 3, and fuzzy text matches (within one or two Levenshtein edits, depending on clue length) add 4. Below 5, the engine returns null and the caller escalates.

Does Auto-POM require every page to be mapped by hand first?

No. PageKnowledgeStore starts empty and grows through recordElement() as elements are discovered or healed during real runs. When the store is empty, KnowledgeLearner.learnFromActionCache() also seeds it from previously recorded actions, so there is no separate "page object generation" step.

If FuzzyHealingEngine fails, does TestFly fail the test right away?

No. A null return from tryFuzzyHeal tells TestFly to escalate to the LLM-based self-healing described in Part 4. The heuristic layer is a cheaper first attempt and does not replace the AI fallback.

TestFly Part 15: Enterprise API auth with RFC 7616 Digest, HMAC signing and OAuth2 refresh tokens

Hakan GÜL — Mon, 12 Oct 2026 18:49:20 +0000

TestFly 1.1.0 replaces its old "Digest" auth, which never worked, with a real RFC 7616 challenge/response implementation (MD5, SHA-256, SHA-512-256, -sess, qop=auth/auth-int). It adds ApiAuth.hmacSigner(), which signs the method, path+query, timestamp and body hash of every request, and ApiAuth.oauth2Refresh(...) for automatic refresh-token grants. A new ApiAuth.Intercepting interface lets these strategies run as the innermost interceptor, where they see the exact request and response. @UseAuth now accepts digest, hmac-signed and oauth2-refresh directly from testfly.yml.

Why Digest auth had to be rewritten

Before 1.1.0, ApiAuth.digest(username, password) looked plausible but was wrong: it sent Authorization: Digest base64(user:password) on every request. No RFC 7616-compliant server accepts that. Digest auth is a cryptographic challenge/response built from a nonce the server issues with a 401, and renaming the Basic scheme does not produce it. A suite testing a real Digest-protected API with the old code would fail at the auth layer before it ever reached the API.

Fixing it meant implementing the protocol: parse the WWW-Authenticate header, compute HA1/HA2 per RFC 7616, track nonce counts, and detect when a nonce has gone stale.

How TestFly implements RFC 7616 Digest authentication

ApiAuth.digest(username, password) returns an ApiAuth.Intercepting implementation (DigestAuth) that is installed as the innermost interceptor. That position lets it see the outgoing request and the incoming response, so it can resend a request after a challenge:

ApiClient.setGlobalAuth(ApiAuth.digest("qa-user", System.getenv("DIGEST_PASSWORD")));

apiClient().get("/api/secure/orders")
        .send()
        .assertStatus(200);

The flow on the wire:

The first request to an origin goes out without credentials.
A 401 WWW-Authenticate: Digest ... response is parsed into a Challenge (realm, nonce, opaque, algorithm, qop set, stale flag). When a server advertises several schemes, TestFly picks the strongest supported algorithm (SHA-512-256 > SHA-256 > MD5).
The request is resent once with a computed Authorization: Digest ... header.
The challenge is cached per origin for the current test (ApiClient.challengeCache()) and shared across sendAsync calls. Later requests authenticate preemptively with an incrementing nonce count (nc) instead of going through a 401 every time.
A stale nonce (the server returns stale=true, or a new nonce on a later 401) triggers exactly one re-authentication with the fresh challenge. If the same nonce comes back without the stale flag, TestFly treats it as rejected credentials and returns the 401 to the caller instead of looping.

Supported algorithms are MD5, MD5-sess, SHA-256, SHA-256-sess, SHA-512-256 and SHA-512-256-sess, with qop=auth and qop=auth-int (which folds a hash of the request body into HA2). This is the core of the response hash, taken from the implementation:

String ha1 = hash(digestName, username + ":" + challenge.realm + ":" + password);
if (session) ha1 = hash(digestName, ha1 + ":" + challenge.nonce + ":" + cnonce);

String a2 = request.method() + ":" + digestUri;
if ("auth-int".equals(qop)) a2 += ":" + hash(digestName, request.body());
String ha2 = hash(digestName, a2);

String response = qop == null
        ? hash(digestName, ha1 + ":" + challenge.nonce + ":" + ha2)
        : hash(digestName, ha1 + ":" + challenge.nonce + ":" + nc + ":" + cnonce + ":" + qop + ":" + ha2);

As RFC 7616 specifies, the -sess variants fold the client nonce into HA1 once per challenge instead of recomputing it on every request.

Signing requests with HMAC

Many internal and partner APIs don't use Basic or Bearer auth at all. They want every request signed, so a captured header can't be replayed with a different payload. TestFly 1.1.0 adds ApiAuth.hmacSigner(), a builder for ApiHmacSigner, which is another ApiAuth.Intercepting strategy:

ApiAuth signer = ApiAuth.hmacSigner()
        .apiKey(System.getenv("API_KEY"))
        .secret(System.getenv("API_SECRET"))
        .signatureHeader("X-Signature")
        .encoding(ApiHmacSigner.Encoding.HEX)
        .build();

ApiClient.setGlobalAuth(signer);

By default it signs a canonical string made of the method, path+query, a timestamp and a SHA-256 hash of the body, joined by newlines:

public static String defaultCanonical(SigningContext ctx) {
    return ctx.method() + "\n" + ctx.pathAndQuery() + "\n" + ctx.timestamp() + "\n" + ctx.bodySha256();
}

The builder controls every part of the signature:

algorithm(...): any JCA Mac algorithm, default HmacSHA256
keyHeader(...) / signatureHeader(...) / timestampHeader(...): set any of them to null to omit that header
encoding(...): BASE64 or HEX
timestampFormat(...): default epoch seconds
canonical(...): replaces the whole signing string with a Function<SigningContext, String>, for a server that expects, say, a different line order or an extra nonce

Because the signer runs as the innermost interceptor, it signs the exact request bytes about to go on the wire. That includes headers added earlier in the chain by other ApiAuth strategies. It does not include headers that legacy RequestInterceptor hooks add afterward, because those run outside it.

The old ApiAuth.hmac(apiKey, secret, algorithm) is now deprecated. It only signed the API key string and never the request, so it can't satisfy a server that verifies request signatures. It stays for wire compatibility, but new code should use hmacSigner().

How OAuth2 token refresh works

ApiAuth.oauth2(...) already handled client-credentials tokens with caching. 1.1.0 adds ApiAuth.oauth2Refresh(tokenUrl, clientId, clientSecret, refreshToken) for flows where a long-lived refresh token is exchanged for short-lived access tokens:

ApiClient.setGlobalAuth(ApiAuth.oauth2Refresh(
        "https://auth.example.com/token",
        System.getenv("CLIENT_ID"),
        System.getenv("CLIENT_SECRET"),
        System.getenv("REFRESH_TOKEN")));

All three OAuth2 factories (oauth2, oauth2Password, oauth2Refresh) share the process-wide OAuth2TokenCache. Its keys keep a client-credentials token, a password token and a refresh token for the same tokenUrl/clientId from colliding. When a token expires, the cache first tries a refresh_token grant with the refresh token it holds. If that grant fails outright, it re-runs the original grant instead of failing the test:

if (cached != null && cached.refreshToken != null) {
    try {
        fresh = refresh(tokenUrl, clientId, clientSecret, cached.refreshToken);
    } catch (RuntimeException refreshFailed) {
        fresh = null; // fall back to the original grant
    }
}
if (fresh == null) fresh = grant.get();

Servers that rotate refresh tokens on every use are supported. If the token response includes a new refresh_token, that one is cached for the next refresh. If the server omits it (no rotation), TestFly keeps the one it already has. ApiAuth.clearTokenCache() clears every cached token on demand, and the framework calls it automatically at suite end so tokens never leak between suites in the same JVM.

Declarative setup with @UseAuth

All three strategies are now first-class @UseAuth types. The shared AuthStrategies class, used by both the TestNG listener and the JUnit 5 extension, resolves them from testfly.yml:

# testfly.yml
api:
  auth:
    legacyPartner:
      type: digest
      username: ${DIGEST_USER}
      password: ${DIGEST_PASSWORD}
    signedService:
      type: hmac-signed
      apiKey: ${API_KEY}
      secret: ${API_SECRET}
      algorithm: HmacSHA256
      keyHeader: X-Api-Key
      signatureHeader: X-Signature
      timestampHeader: X-Timestamp
      encoding: base64
    refreshableAccount:
      type: oauth2-refresh
      tokenUrl: https://auth.example.com/token
      clientId: ${CLIENT_ID}
      clientSecret: ${CLIENT_SECRET}
      refreshToken: ${REFRESH_TOKEN}

@Test
@UseAuth("signedService")
public void createOrderIsSigned() {
    apiClient().post("/api/orders").body(order).send().assertStatus(201);
}

keyHeader: none or timestampHeader: none (or a blank header value) omits that header entirely. This is useful when a server only wants the signature header and gets the key from an upstream gateway. The TestNG listener and the JUnit 5 extension resolve @UseAuth through the same code path, so both runners behave identically.

Other API auth changes in 1.1.0

The ApiAuth.Intercepting interface is new infrastructure. A strategy that needs the final request bytes or the actual response, and not only a header to attach before the request is built, implements it. ApiClient then installs it as the innermost ApiInterceptor, closest to the transport. This is what lets DigestAuth resend a request after reading a 401 and lets ApiHmacSigner sign the request exactly as it will leave the process. Both ship as package-private or final implementations, and you reach them only through ApiAuth.digest(...) and ApiAuth.hmacSigner().

Two related fixes landed in the same release. ApiAuth.basicAuth now always encodes credentials as UTF-8 (previously it used the platform default). The OAuth2 token cache is now cleared at suite end instead of living as long as the JVM, so a second suite in the same run can't silently reuse another suite's token.

FAQ

Does `ApiAuth.digest(...)` make two HTTP calls per request?

Only the first request to a given origin does. After the first 401/challenge round trip, the challenge is cached per origin for the current test, and later requests send a valid Authorization: Digest header preemptively with an incremented nonce count. A second request happens only if the nonce goes stale or rotates.

What happens if the Digest credentials are wrong?

TestFly resends once per fresh challenge. If the server returns the same non-stale nonce again, TestFly treats that as rejected credentials and returns the 401 to the caller instead of retrying forever.

Can I use a custom HMAC canonical string instead of the default one?

Yes. ApiAuth.hmacSigner().canonical(ctx -> ...) takes a Function<SigningContext, String> with access to the method, URI, timestamp, raw body bytes and headers. You can match any server-side signing scheme this way; TestFly's default METHOD\npath?query\ntimestamp\nbodyHash is only one option.

Does `oauth2Refresh` break if the server never rotates refresh tokens?

No. If a refresh response omits refresh_token, TestFly keeps reusing the refresh token you supplied. It switches to a new one only when the server returns one.

TestFly is a Java 21 Selenium/API test automation SDK. Previous entries in this series: Part 12: Playwright-style assertions, Part 13: actionability, Part 14: step traces.

TestFly Part 17: Cross-browser reliability with BiDi console capture, Edge/Safari on Grid and guaranteed driver cleanup

Hakan GÜL — Mon, 12 Oct 2026 18:49:15 +0000

TestFly 1.0.6 and 1.1.0 close three cross-browser reliability gaps.

First, browser.bidi opens a real WebDriver BiDi connection to a locally launched Chrome, Edge or Firefox. ConsoleErrorCollector then reads console.error calls and uncaught exceptions from BiDi log events instead of a JS shim, which catches errors Firefox used to hide. ConsoleErrorCollector.isBiDiActive() tells you which capture path ran.

Second, RemoteDriverProvider now resolves Edge and Safari on Selenium Grid and honors browser.matrix. Before, it only knew chrome/firefox and always read browser.name.

Third, JVM shutdown hooks and DriverManager.quitAllTrackedDrivers() quit every tracked browser exactly once when a run is aborted with Ctrl+C or System.exit. That covers browsers on every thread, including threads other than the one running the hook.

How BiDi catches console errors that Firefox used to miss

TestFly's captureConsoleErrors has always read browser console errors. Firefox exposes no browser logs over classic WebDriver, so on Firefox this meant injecting a JavaScript shim that only sees errors after it runs on the current page. The shim could not see an error thrown while the page was still loading, or one on a page reached by following a link.

browser.bidi (default false) solves this at the protocol level. When a Chrome, Edge or Firefox session is launched locally, TestFly requests a real WebDriver BiDi connection and subscribes to BiDi log events from the moment the test gets its browser:

# testfly.yml
browser:
  name: firefox
  bidi: true
  captureConsoleErrors: true

The capability request is one line and is added only when BiDi is turned on:

// BiDiCapability.java
static void applyIfConfigured(MutableCapabilities options, TestFlyConfig config) {
    if (config != null && config.getBrowser() != null && config.getBrowser().isBidi()) {
        options.setCapability("webSocketUrl", true);
    }
}

ConsoleErrorCollector then opens one LogInspector per session over that connection. The inspectors are kept in a WeakHashMap<WebDriver, BiDiCapture>, so a session is not held in memory after its driver is gone:

private static final Map<WebDriver, BiDiCapture> BIDI =
        Collections.synchronizedMap(new WeakHashMap<>());

The subscription starts at beginTest(driver), so errors from a previous test are dropped even when a per-suite browser is reused. There is no JS shim and no race with page load.

Safari and remote drivers (Selenium Grid, BrowserStack, Sauce Labs) don't get a BiDi connection in TestFly today. For those, TestFly logs one warning per session and falls back to the usual capture path without throwing.

To check which path ran without reading logs:

if (ConsoleErrorCollector.isBiDiActive()) {
    // BiDi log events are driving capture for this session
}

// ConsoleErrorCollector.java
@TestFlyApi(since = "1.1.0")
public static boolean isBiDiActive() {
    try {
        return BIDI.containsKey(DriverManager.getDriver());
    } catch (RuntimeException e) {
        return false;
    }
}

Knowing which browser features you lose

A locally started Chrome or Edge gives TestFly the Chrome DevTools Protocol (CDP). CDP powers network mocking, device emulation, geolocation overrides and the screencast-based video recorder. Firefox, Safari, Selenium Grid and cloud sessions don't expose CDP, and neither does a remote Chrome, so these features would degrade silently unless something reported it.

BrowserFeatures, new in 1.1.0, names the four CDP-dependent features and describes the fallback for each:

public enum Feature {
    NETWORK_MOCK("Network interception requires a local Chrome/Edge session... "
            + "routes are ignored and requests reach the real network."),
    CDP_RECORDING("Video recording uses periodic screenshots instead of the "
            + "DevTools screencast, so frames are less frequent."),
    DEVICE_EMULATION("Device emulation only resizes the window and overrides "
            + "navigator.userAgent in the current page..."),
    GEOLOCATION("The location is overridden through JavaScript in the current "
            + "page only and is lost on navigation.");
}

BrowserFeatures.supports(driver, feature) answers the question directly (driver instanceof ChromiumDriver for three of the four features, HasDevTools for recording). warnOnce(driver, feature) logs the fallback exactly once per browser session and feature, and names the browser:

BrowserFeatures.warnOnce(driver, BrowserFeatures.Feature.GEOLOCATION);

GeoLocation already calls this before it falls back to its JS override on Firefox. The video recording fallback used to be silent. Now every degraded feature logs a warning.

Running Edge and Safari on Selenium Grid

Before this release, RemoteDriverProvider only built ChromeOptions or FirefoxOptions and always read browser.name directly. A browser.matrix run on a Grid therefore used whatever browser.name said and ignored the matrix browser, and Edge and Safari on Grid were not supported at all.

Browser resolution now goes through one shared method that both the local and remote paths call. BrowserContext (the matrix override) takes precedence over YAML:

// DriverProviderFactory.java
public static String resolveBrowserName(TestFlyConfig config) {
    String contextBrowser = BrowserContext.get();
    if (contextBrowser != null && !contextBrowser.isBlank()) {
        return contextBrowser;
    }
    String name = config.getBrowser().getName();
    if (name == null || name.isBlank()) {
        throw new IllegalStateException(
                "No browser selected: set browser.name, or run through browser.matrix.");
    }
    return name;
}

RemoteDriverProvider.buildOptions(config, browser) is a public method you can test on its own. It now covers all four browsers that Grid supports:

// RemoteDriverProvider.java
switch (browser.toLowerCase(Locale.ROOT)) {
    case "chrome" -> options = chromium(new ChromeOptions(), browserConfig.isHeadless(), arguments);
    case "edge" -> options = chromium(new EdgeOptions(), browserConfig.isHeadless(), arguments);
    case "firefox" -> { /* FirefoxOptions, -headless */ }
    case "safari" -> {
        if (browserConfig.isHeadless()) {
            System.err.println("[TestFly] Safari does not support headless mode via Selenium; ignoring headless=true.");
        }
        options = new SafariOptions();
    }
    default -> throw new IllegalStateException(
            "Unsupported browser for remote execution: " + browser + ". Supported: chrome, edge, firefox, safari.");
}
options.setCapability("unhandledPromptBehavior", "ignore");

A matrix config that runs the same suite on Chrome, Edge and Safari on a Grid now works without changes:

# testfly.yml
execution:
  mode: remote
  gridUrl: http://grid.local:4444/wd/hub
browser:
  matrix: [chrome, edge, safari]

Two related fixes shipped alongside. The first (do not reuse a per-suite driver for another matrix browser) stops DriverManager from handing a Chrome per-suite driver to a thread that should be testing Edge; DRIVER_BROWSER_CONTEXT is now tracked per thread and checked before reuse. The second (quit the browser when post-launch session setup fails) makes every built-in provider quit the session before rethrowing when timeout configuration fails right after launch, so a setup failure no longer leaves a browser process behind.

Making sure no browser is left running

DriverManager has registered a JVM shutdown hook from early on. Until this release, though, the hook only quit per-suite drivers and whatever driver was bound to the hook's own thread, and that is never the thread that ran a per-test driver. If you aborted a parallel per-test run with Ctrl+C or System.exit, the worker-thread browsers kept running after the JVM exited.

The hook now calls quitAllTrackedDrivers(). It takes every driver that still holds a session permit plus every registered per-suite driver, across all threads, and quits each one exactly once:

static void quitAllTrackedDrivers() {
    java.util.Set<WebDriver> drivers =
            java.util.Collections.newSetFromMap(new java.util.IdentityHashMap<>());
    drivers.addAll(PERMIT_HOLDERS);
    drivers.addAll(SUITE_DRIVERS);
    for (WebDriver driver : drivers) {
        try {
            driver.quit();
        } catch (Throwable ignored) {
        } finally {
            SUITE_DRIVERS.remove(driver);
            releasePermitOf(driver);
        }
    }
}

PERMIT_HOLDERS is the set of drivers that currently hold a session-slot permit. Exactly one code path per driver adds and removes entries, which is also why execution.maxActiveSessions can't leak slots. Normal suite teardown still calls DriverManager.quitAllSuiteDrivers() and then quitDriver() from SuiteExecutionListener, TestFlyExtension (JUnit 5) and CucumberHooks. The shutdown hook is the safety net for abnormal exits and runs the same cleanup logic no matter which thread owned the browser.

This builds on the original "Guaranteed Driver Cleanup" item from 1.0.6 (shutdown hooks plus @AfterAll listener execution). The hook has existed since 1.0.6; this release makes it reach every thread's browser and not only the suite-level ones.

FAQ

Does `browser.bidi` work with Selenium Grid or BrowserStack/Sauce Labs?

No. BiDi only applies to a locally launched Chrome, Edge or Firefox. On Safari or any remote driver, TestFly logs one warning and falls back to the previous capture path (CDP logs on Chrome, the JS shim elsewhere). Nothing fails.

How do I check which console-error capture path is active for the current test?

Call ConsoleErrorCollector.isBiDiActive(). It returns true only when the current thread's driver has an open BiDi log subscription. It never throws; it returns false if there is no driver or the check fails.

Can I run the same suite on Chrome, Edge and Safari against one Selenium Grid?

Yes, with execution.mode: remote, execution.gridUrl and browser.matrix: [chrome, edge, safari]. RemoteDriverProvider resolves the matrix browser before browser.name and builds Grid options for all four supported browsers, including Safari's headless restriction.

If I kill a parallel run with Ctrl+C, are all browser processes gone?

Yes. The JVM shutdown hook now collects every driver that holds a session permit and every per-suite driver, on every thread, and quits each one exactly once, regardless of which thread the hook runs on.

TestFly Part 12: Playwright-style assertions in Selenium with not(), ARIA snapshots and strict mode

Hakan GÜL — Mon, 12 Oct 2026 18:40:51 +0000

TestFly 1.1.0 closes most of the gap with Playwright for Java's assertion API without leaving Selenium. LocatorAssert.not() negates the next check. hasRole(), hasAccessibleName() and hasAccessibleDescription() read the real accessibility tree instead of guessing from DOM attributes. matchesAriaSnapshot() compares a subtree against a Playwright-style YAML template. The opt-in locators.strict setting makes an ambiguous locator fail loudly, where before it would silently click the wrong element.

Why Selenium assertions need not()

In classic Selenium/TestNG code a negative assertion usually means one of two bad options: wrap the positive assertion in a try/catch and invert the result, or write a separate polling loop that waits for a condition to stay false. Either way the intent of the test ends up buried in boilerplate.

TestFly's LocatorAssert now has the same not() as Playwright for Java. It negates only the assertion that comes directly after it in the chain:

import io.testfly.test.BaseTest;
import org.testng.annotations.Test;

import static io.testfly.assertion.Assertions.assertThat;

public class CheckoutErrorTest extends BaseTest {

    @Test
    public void testErrorBannerClearsAfterRetry() {
        open("/checkout");
        find("#pay-btn").click();

        assertThat(find("#error-banner")).hasText("Payment declined");

        find("#retry-btn").click();

        // Only the assertion right after not() is negated
        assertThat(find("#error-banner")).not().isVisible();
        assertThat(find("#spinner")).not().isAttached();
    }
}

As in Playwright, most negated assertions also pass when no matching element exists, so not().isVisible() passes whether the banner is hidden or removed from the DOM. If the element has to exist and only be in a different state, assert presence first: assertThat(status).isVisible().not().hasText("Error"). You cannot combine not() with the verified actions click(), fill() or clear().

Deterministic accessibility assertions

TestFly already had hasAriaLabel-style attribute checks. Version 1.1.0 adds assertions that read the computed accessibility tree the way a screen reader does. They run through one bundled probe script, with no axe-core injection and no LLM call:

@Test
public void testSubmitButtonIsAccessible() {
    open("/signup");

    assertThat(find("#submit"))
        .hasRole(Role.BUTTON)
        .hasAccessibleName("Create account");

    assertThat(find("#email"))
        .hasAccessibleDescription(Pattern.compile("(?i)required"));
}

hasRole(Role | String) checks the explicit role attribute or the implicit role of the HTML tag. hasAccessibleName and hasAccessibleDescription take either an exact String (whitespace collapsed, with the same matching Locator.withName(...) uses) or a Pattern for a partial or case-insensitive match. Only the 36 standard ARIA Role values and visible text are reported. Decorative images and hidden content are skipped, and the probe does not enter Shadow DOM.

How ARIA snapshots work

matchesAriaSnapshot() is TestFly's version of Playwright's toMatchAriaSnapshot(). It captures the accessibility subtree under a locator and compares it with a YAML template. The comparison is partial and ordered, so it does not need a full serialization of the subtree to match.

@Test
public void testNavigationLandmarkStructure() {
    open("/dashboard");

    assertThat(By.tagName("nav")).matchesAriaSnapshot("""
        - link "Home"
        - link "Docs"
        - button "Account" [expanded=false]
        """);
}

The matching rules follow Playwright:

Rule	Behavior
Role	must match exactly
Quoted name	case-insensitive substring match
`/regex/` name	matched with `find()`
`[attribute]` / `[level=2]`	every template attribute must be present on the page; extra page attributes are ignored

A malformed template fails at once with IllegalArgumentException, without waiting for the timeout. When the assertion fails, the error contains both the template and the captured snapshot, so you can see the difference without re-running anything. To write a template in the first place, print what TestFly currently sees:

System.out.println(find("nav").ariaSnapshot());
// or, for the whole page:
System.out.println(ariaSnapshot());

Then paste the printed YAML into the test and trim it down to the nodes that matter.

How strict mode catches silent multi-element bugs

By default, when a locator resolves to several visible elements, TestFly does what classic Selenium does and silently clicks or reads the first one. That is how a renamed CSS class or an A/B-tested duplicate banner produces a test that passes against the wrong element for weeks before anyone notices.

You turn strict mode on per project:

# testfly.yml
locators:
  strict: true

With locators.strict: true, an action on a locator that matches more than one visible element, and that was not narrowed with first(), nth() or last(), throws immediately:

Strict mode violation: css=".card .btn-buy" resolved to 3 visible elements;
use first()/nth()/last() or narrow the locator. Candidates: <button.btn-buy#card-1>, <button.btn-buy#card-2>, <button.btn-buy#card-3>

isVisible() and isEnabled() report the same violation instead of silently checking the first match. count(), elements() and assertThat(...) (for counting or enumerating) are not affected, because multiple matches are expected there. When strict mode flags a locator, narrow it explicitly:

find(".card .btn-buy").nth(1).click();          // second card, explicit
find(".card").filter(withText("Pro Plan"))
             .find(".btn-buy").click();          // narrowed by content instead

Hidden duplicates, such as a closed mobile nav next to the desktop one or a hidden modal clone, are filtered out before the check runs, so responsive layouts don't cause false strict-mode violations.

FAQ

Does `not()` replace the existing Selenium assertion methods?

No. not() is a modifier on LocatorAssert. It negates whichever assertion comes right after it in the same chain, and all existing positive assertions stay as they are.

Do the accessibility assertions require axe-core or an external library?

No. hasRole, hasAccessibleName, hasAccessibleDescription and matchesAriaSnapshot use one bundled probe script that reads the browser's computed accessibility tree directly. There is no axe-core injection and no LLM call.

Will enabling `locators.strict` break existing tests?

It can, and that is intended. Any locator that used to match more than one visible element silently now fails explicitly. The fix is the one Playwright recommends: narrow the locator with first(), nth(), last() or a more specific selector.

Does strict mode affect `count()` or `elements()`?

No. Those methods exist to work with multiple matches, so strict mode does not apply to them. It applies only to actions (click, fill, …) and to the boolean checks isVisible()/isEnabled().

Originally published on hakangul.lovable.app.

Scaling to 100,000 RPS: Deploying Distributed Load Testing on Kubernetes with Helm and Docker

Hakan GÜL — Sun, 30 Aug 2026 07:41:47 +0000

TL;DR: Single-machine load tests inevitably hit CPU and socket descriptor limits when pushing beyond a few thousand requests per second. In LocustPilot, we packaged the entire control plane and distributed worker architecture into Kubernetes Helm charts (helm/locust), allowing engineering teams to scale to 100,000+ RPS across cloud nodes in minutes with automated resource isolation.

When a Single Machine Hits the Wall

A single modern laptop or cloud VM running Locust can comfortably generate between 2,000 and 5,000 requests per second (RPS).

However, when preparing for enterprise-scale traffic events (e.g. streaming sports events, flash sales, or banking cutovers), you need tens of thousands of concurrent users. Attempting this on one machine leads to misleading test results:

CPU Saturation: Python processes hit 100% CPU on single cores, artificially increasing response times on the client side.
Socket Exhaustion (TIME_WAIT): The operating system runs out of ephemeral ports, causing false Connection Refused errors that have nothing to do with your backend.
Bandwidth Throttling: Cloud VM network interfaces bottleneck before your target servers are even stressed.

To scale reliably, you must distribute load generation across a cluster of worker nodes.

The Distributed Master-Worker Architecture

In distributed mode, Locust separates responsibilities across two roles:

Master Node: Runs the LocustPilot web control center, coordinates worker connections, aggregates incoming metrics, and streams telemetry to ReportPortal. It generates zero HTTP traffic itself.
Worker Nodes: Connect to the master, spawn virtual users, and bombard the target backend with requests.

       ┌───────────────────────────────┐
       │   LocustPilot Master (UI)     │
       │   (Streamlit + ReportPortal)  │
       └──────────────┬────────────────┘
                      │ Port 5557 (ZMQ)
         ┌────────────┼────────────┐
         │            │            │
   ┌─────▼────┐ ┌─────▼────┐ ┌─────▼────┐
   │ Worker 1 │ │ Worker 2 │ │ Worker N │
   └─────┬────┘ └─────┬────┘ └─────┬────┘
         │            │            │
         └────────────┼────────────┘
                      ▼
         [ Target Microservices ]

Packaging for Kubernetes with Helm (`helm/locust`)

LocustPilot provides a production-ready Helm chart located in helm/locust/.

1. The Optimized Container (`k8s.Dockerfile`)

FROM python:3.10-slim

WORKDIR /app

RUN apt-get update && apt-get install -y --no-install-recommends \
    curl build-essential && \
    rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

EXPOSE 8501 5557
CMD ["streamlit", "run", "app.py", "--server.port=8501", "--server.address=0.0.0.0"]

Photo by Fotis Fotopoulos on Unsplash

2. Deploying the Cluster with One Command

Using the pre-built Helm chart, deploying the entire distributed load testing cluster with ReportPortal telemetry requires a single command:

helm upgrade --install locust ./helm/locust \
  --namespace loadtest \
  --create-namespace \
  --set image.repository=my-registry.io/locust-pilot \
  --set image.tag=latest \
  --set replicaCount=10 \
  --set extraEnv[0].name=RP_TOKEN --set extraEnv[0].value=$RP_TOKEN \
  --set extraEnv[1].name=RP_ENDPOINT --set extraEnv[1].value=$RP_ENDPOINT \
  --set extraEnv[2].name=RP_PROJECT --set extraEnv[2].value=$RP_PROJECT

3. Tuning OS Limits for Massive Scale

When running high-volume workers inside Kubernetes, ensure your Pod security contexts allow high file descriptors:

# helm/locust/values.yaml
worker:
  resources:
    limits:
      cpu: 2000m
      memory: 2Gi
    requests:
      cpu: 1000m
      memory: 1Gi
  sysctls:
    - name: net.ipv4.tcp_tw_reuse
      value: "1"
    - name: net.core.somaxconn
      value: "65535"

What Is Next?

In the final installment (Part 5), we will connect all the pieces into your continuous delivery pipeline: Shift-Left Performance Testing with Automated Quality Gates in GitHub Actions and Bitbucket Pipelines.

👉 Explore the Helm Charts on GitHub

FAQ

How many workers are needed for 50,000 RPS?

Depending on endpoint payload size and SSL overhead, each CPU core running FastHttpUser can typically generate 1,500–3,000 RPS. A cluster of 20–25 worker pods can comfortably sustain 50,000+ RPS.

Can workers be dynamically auto-scaled?

Yes. Kubernetes Horizontal Pod Autoscaler (HPA) can scale worker pods based on CPU utilization or custom test duration parameters.

Enterprise Observability: Real-Time ReportPortal Integration with Gevent and Percentile Tracking

Hakan GÜL — Sun, 30 Aug 2026 07:41:12 +0000

TL;DR: High-throughput load tests generating thousands of requests per second can easily overwhelm reporting backends with log spam. In LocustPilot, we solved this by implementing an asynchronous gevent listener for ReportPortal that deduplicates error cascades, streams live endpoint metrics every 30 seconds, and calculates comprehensive response time percentiles (P50 to P99) with automated quality gate verdicts.

The Log Flood Dilemma in Load Testing

In standard functional UI or API tests, every failure is unique and warrants a detailed log entry.

In performance testing, the dynamic is completely inverted:

When an endpoint begins to degrade under 5,000 RPS, it does not fail once—it fails 50,000 times with the exact same 504 Gateway Timeout or Connection Refused error.
If your test framework attempts to log every single failure to a centralized telemetry platform like ReportPortal, you will DDoS your own reporting database, exhaust memory, and freeze the test execution.

To provide enterprise-grade observability without degrading load test performance, we built ReportPortalListener (app/core/rp_listener.py) in LocustPilot.

1. Background Telemetry with Gevent Greenlets

Locust relies on gevent for lightweight, non-blocking coroutines. Rather than blocking worker execution during network logging, LocustPilot spawns a background greenlet (_periodic_stats_logger) that flushes aggregated endpoint statistics every 30 seconds:

# app/core/rp_listener.py
import gevent
from reportportal_client.helpers import timestamp

def on_test_start(self, **kwargs):
    # Initialize ReportPortal Launch and Test Item
    self.launch_uuid = self.rp_client.start_launch(
        name=self.rp_launch_name,
        start_time=timestamp(),
        description=self.rp_launch_desc,
    )
    self.test_item_uuid = self.rp_client.start_test_item(
        name="Load Test Execution",
        start_time=timestamp(),
        item_type="STEP",
    )

    # Spawn background stats publisher
    self.running = True
    gevent.spawn(self._periodic_stats_logger)

def _periodic_stats_logger(self):
    """Log live endpoint stats every 30 seconds without blocking load workers"""
    while self.running:
        gevent.sleep(30)
        if self.running and self.rp_client and self.test_item_uuid:
            lines = ["📊 Live Endpoint Telemetry", "=" * 80]
            lines.append(f"{'Method':<8} {'Name':<40} {'Reqs':<8} {'Fails':<8} {'Avg(ms)':<10} {'RPS':<8}")
            lines.append("-" * 80)

            for stat in self.env.stats.entries.values():
                if stat.num_requests > 0:
                    lines.append(
                        f"{stat.method:<8} {stat.name[:40]:<40} "
                        f"{stat.num_requests:<8} {stat.num_failures:<8} "
                        f"{stat.avg_response_time:<10.0f} {stat.current_rps:<8.1f}"
                    )

            self.rp_client.log(
                time=timestamp(),
                message="\n".join(lines),
                level="INFO",
                item_id=self.test_item_uuid,
            )

This ensures team leads and developers watching the ReportPortal dashboard receive live health updates without injecting latency into the simulated users.

2. Smart Error Deduplication (Unique Error Logging)

When a microservice falters, LocustPilot captures the first instance of an error to preserve diagnostic context, while silently aggregating subsequent identical failures into a counter:

def on_request(self, request_type, name, response_time, response_length, exception=None, response=None, **kwargs):
    has_exception = exception is not None
    has_manual_failure = response is not None and getattr(response, "_manual_result", None) is False

    if has_exception or has_manual_failure:
        error_key = f"{request_type}:{name}"

        # Initialize counter on first occurrence
        if error_key not in self.last_stats:
            self.last_stats[error_key] = {"count": 0, "logged": False}

        self.last_stats[error_key]["count"] += 1

        # Only send the first failure payload to ReportPortal
        if not self.last_stats[error_key]["logged"] and self.rp_client:
            error_msg = str(exception) if has_exception else getattr(response, "_manual_result_msg", "Manual failure")
            self.rp_client.log(
                time=timestamp(),
                message=f"❌ First Failure Detected: {request_type} {name}\nDetails: {error_msg}",
                level="ERROR",
                item_id=self.test_item_uuid,
            )
            self.last_stats[error_key]["logged"] = True

Result: Zero log spam. If 100,000 requests fail with the same connection error, ReportPortal receives exactly 1 clean diagnostic entry with a summary count at the end of the test.

Photo by Mika Baumeister on Unsplash

3. Comprehensive Percentiles and Automated Quality Gates

Average response time is a dangerous vanity metric in performance engineering; a 200ms average can hide a 99th percentile spike of 12 seconds.

When a test completes (on_test_stop), LocustPilot computes full percentile distributions and enforces automated quality gates:

def on_test_stop(self, **kwargs):
    total = self.env.stats.total

    # 1. Percentile Distribution Table (50% to 100%)
    for stat in self.env.stats.entries.values():
        if stat.num_requests > 0:
            p50 = stat.get_response_time_percentile(0.50)
            p90 = stat.get_response_time_percentile(0.90)
            p95 = stat.get_response_time_percentile(0.95)
            p99 = stat.get_response_time_percentile(0.99)
            max_rt = stat.max_response_time

    # 2. Automated Quality Gate Rules
    has_issue = False
    if total.avg_response_time > 200:
        log_warning(f"⚠️ Average response time exceeded 200ms! (Actual: {total.avg_response_time:.2f} ms)")
        has_issue = True
    if total.fail_ratio > 0.05:
        log_warning(f"⚠️ Failed request ratio exceeded 5%! (Actual: {total.fail_ratio:.2%})")
        has_issue = True

    # 3. Final Build Verdict
    success_rate = (1 - total.fail_ratio) * 100
    status = "PASSED" if success_rate >= 75 and not has_issue else "FAILED"

    self.rp_client.finish_test_item(item_id=self.test_item_uuid, end_time=timestamp(), status=status)
    self.rp_client.finish_launch(end_time=timestamp())

What Is Next?

In Part 4, we move from single-instance tests to large-scale distributed load: deploying LocustPilot on Kubernetes using Helm charts and Docker worker nodes to simulate over 100,000 requests per second.

👉 Check out the ReportPortal Listener on GitHub

FAQ

Why is ReportPortal preferred over standard HTML reports?

HTML reports are static and isolated on developer machines. ReportPortal centralizes performance history across sprints, enables historical trend comparisons, and automatically flags regression anomalies.

Can custom quality gate thresholds be configured?

Yes. Quality thresholds (such as max average latency or acceptable failure ratio) can be configured via .env variables or custom BaseLocustUser validators.

Under the Hood: Building a Dynamic AST Test Runner and Subprocess Engine in Python

Hakan GÜL — Sun, 30 Aug 2026 07:40:38 +0000

TL;DR: Dynamically discovering test classes at runtime by executing importlib is dangerous because module-level code runs immediately. In LocustPilot, we solved this by building an Abstract Syntax Tree (AST) static scanner that identifies HttpUser and FastHttpUser classes without executing Python code, paired with an unbuffered subprocess.Popen engine for real-time log streaming.

The Hidden Trap of Dynamic Python Discovery

When building a web control plane for load testing, the first requirement is simple: automatically discover and list all test files in the UI dropdown.

Most developers jump straight to dynamic importing:

# ❌ The Naive Way: Dangerous runtime import
import importlib.util

spec = importlib.util.spec_from_file_location("module", file_path)
module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(module) # ⚠️ Executes all top-level code!

Why is this dangerous?

Module-level side effects: If a test file establishes database connections, reads external credentials, or executes setup logic at the module level, simply browsing the UI triggers those actions.
Crash propagation: A syntax error or missing optional dependency in one test file will crash the entire UI server.
High memory footprint: Importing dozens of heavy test modules pollutes the server process memory space.

We needed a way to inspect Python test files statically and safely before execution.

The Solution: Static Analysis via Python's AST

In LocustPilot (app/core/runner.py), we use Python's built-in ast (Abstract Syntax Tree) module. We parse the source code into a syntax tree and walk the nodes without executing a single line of code.

Here is the exact heuristic implemented in LocustPilot:

import ast
import re
from pathlib import Path

def _has_locust_user_class(path: Path) -> bool:
    """Heuristic: returns True if file defines a Locust User class.
    Detects subclasses of HttpUser/FastHttpUser via AST; falls back to regex.
    """
    try:
        src = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        return False

    try:
        tree = ast.parse(src)
        for node in ast.walk(tree):
            if isinstance(node, ast.ClassDef):
                for base in node.bases:
                    # Matches class MyTest(HttpUser)
                    if isinstance(base, ast.Name) and base.id in {"HttpUser", "FastHttpUser"}:
                        return True
                    # Matches class MyTest(locust.HttpUser)
                    if isinstance(base, ast.Attribute) and base.attr in {"HttpUser", "FastHttpUser"}:
                        return True
    except Exception:
        pass

    # Fallback regex check for custom decorators
    return (
        re.search(r"class\s+\w+\(.*?(HttpUser|FastHttpUser).*?\)", src) is not None
        or "@task" in src
    )

Why This Approach Wins:

⚡ Ultra-Fast: Parsing 100 Python files takes less than 15 milliseconds.
🛡️ Zero Side-Effects: No network sockets, database pools, or environment lookups are triggered.
🔍 Smart Filtering: Automatically excludes __pycache__, __init__.py, and helper utility files unless they contain runnable user classes.

Photo by Kevin Ku on Unsplash

Extracting Target Hosts Without Hardcoding

In real-world projects, different test files target different microservices or staging environments. LocustPilot uses an intelligent fallback chain to determine the default target host:

def extract_locustfile_host(path: Path) -> Optional[str]:
    # 1. Environment variable override takes highest precedence
    env_host = os.getenv("LOCUST_TARGET_HOST")
    if env_host:
        return env_host

    # 2. Extract host attribute directly from test file
    try:
        text = path.read_text(encoding="utf-8", errors="ignore")
    except Exception:
        return None

    m = re.search(r"^\s*host\s*=\s*['\"]([^'\"]+)['\"]", text, flags=re.MULTILINE)
    if m:
        return m.group(1).strip()

    # 3. Fallback to centralized BaseUser target host
    if re.search(r"^\s*host\s*=\s*TARGET_HOST", text, flags=re.MULTILINE):
        base_user = BASE_DIR / "locustfiles" / "utils" / "base_user.py"
        if base_user.exists():
            base_text = base_user.read_text(encoding="utf-8", errors="ignore")
            m_target = re.search(r"TARGET_HOST\s*=\s*['\"]([^'\"]+)['\"]", base_text)
            if m_target:
                return m_target.group(1).strip()

    return None

This allows engineers to either set hosts per test file, use a shared BaseLocustUser, or dynamically override them via the Streamlit UI.

Subprocess Execution and Unbuffered Log Streaming

Running load tests inside the main web server process would freeze the UI. LocustPilot spawns an isolated subprocess.Popen running headless Locust while streaming logs in real time.

def run_locust(
    locustfile: Path,
    host: Optional[str],
    users: int,
    spawn_rate: float,
    run_time: str,
    run_dir: Path,
    csv_prefix: str = "stats",
    html_report: bool = True,
    csv_full_history: bool = True,
    stream_logs: bool = True,
) -> Tuple[subprocess.Popen, Path, Path, str, List[str]]:

    cmd = [
        "locust",
        "-f", str(locustfile),
        "--headless",
        "-u", str(users),
        "-r", str(spawn_rate),
        "--run-time", str(run_time),
        "--csv", str(run_dir / csv_prefix),
        "--logfile", str(run_dir / "locust.log"),
        "--only-summary",
    ]
    if host:
        cmd += ["--host", str(host)]
    if html_report:
        cmd += ["--html", str(run_dir / "report.html")]
    if csv_full_history:
        cmd += ["--csv-full-history"]

    # Dynamic PYTHONPATH injection so test libraries import cleanly
    env = os.environ.copy()
    py_paths = [str(LOCUSTFILES_DIR), str(LOCUSTFILES_DIR / "libs"), str(BASE_DIR)]
    env["PYTHONPATH"] = os.pathsep.join(py_paths)

    proc = subprocess.Popen(
        cmd,
        stdout=(subprocess.PIPE if stream_logs else subprocess.DEVNULL),
        stderr=subprocess.STDOUT,
        text=True,
        bufsize=1,  # Line-buffered output for instant streaming
        cwd=str(BASE_DIR),
        env=env,
    )
    return proc, logfile, html_path, start, cmd

Key Subprocess Design Choices:

bufsize=1: Enables line-buffered I/O. Standard Python output buffering waits for hundreds of bytes before flushing; bufsize=1 pushes log lines to Streamlit instantly.
PYTHONPATH Injection: Automatically adds libs/ and utils/ to Python path, allowing test scripts to use shared utilities without complex packaging.
Non-Blocking Polling: The Streamlit UI polls proc.stdout.readline() without blocking the event loop.

Architectural Summary

By decoupling test discovery (AST) from test execution (Subprocess), LocustPilot achieves:

Safety: Malicious or broken test scripts cannot crash the control plane.
Responsiveness: Heavy load generation runs in isolated processes without degrading web UI performance.
Clean Code: Test authors write standard Python without needing framework boilerplate.

What Is Next?

In Part 3, we will dive into Enterprise Observability: how LocustPilot leverages gevent greenlets to stream real-time P95/P99 latency percentiles and deduplicated error alerts to ReportPortal.

👉 Explore the LocustPilot Source Code on GitHub

FAQ

Why not use regex alone to find Locust classes?

Regex can produce false positives on comments or strings (e.g. inside docstrings). AST guarantees that we are inspecting actual Python class inheritance syntax.

Does `subprocess.Popen` limit load test throughput?

No. Headless Locust runs in its own OS process utilizing gevent coroutines. The Streamlit UI only reads lightweight summary lines, keeping CPU overhead below 1%.

Can I run tests with custom Python dependencies?

Yes. By configuring PYTHONPATH dynamically, any custom library inside locustfiles/libs/ or virtualenv is immediately accessible.

DEV Community: Hakan GÜL

TestFly Part 11: Network interception and API mocking in Selenium 4 with CDP

Why frontend tests need network interception

Simulating slow 3G and offline mode

Mocking backend API responses (stubbing)

Catching silent console errors

TestFly Part 10: Email and 2FA testing with Mailhog, Mailtrap and IMAP

Why email verification gets in the way of E2E tests

Connecting to a mailbox

Supported mail providers

FAQ

Does TestFly poll the mailbox asynchronously?

Can I extract HTML links and click them directly?

Does this work with temporary throwaway email addresses?

TestFly Part 9: In-process load testing in Java with Gatling and K6

Why keep functional and load tests together

How load testing works in TestFly

1. Declarative mode with @LoadTest

2. Fluent programmatic DSL with load()

Checking performance SLOs in code

Reporting

FAQ

Does TestFly require JMeter or an external binary?

Can I run load tests in CI/CD without overwhelming local machines?

Can I use API authentication tokens in load tests?

TestFly Part 13: Actionability engine, auto-wait and verified click/fill/clear assertions

How TestFly decides an element is ready for an action

What gets checked, and in what order

What happens when an element never becomes actionable

Configuring or bypassing the checks

How assertThat(x).click() verifies that a click landed

How assertThat(x).fill(text) and .clear() verify the field changed

FAQ

Does turning on actionability checks slow every test down?

Does assertThat(x).click() replace Locator.click()?

Can I combine not() with the verified click()/fill()/clear()?

What's the difference between force() and actionability.enabled: false?

TestFly Part 14: Step traces with per-step ARIA/DOM snapshots for visual debugging

What a step trace is and why every step gets a snapshot

What each step's snapshot contains

Configuring what gets captured

What the trace HTML shows

Performance and failure risk

Cucumber support

FAQ

Do I need to add code to my tests to get per-step snapshots?

What happens if a test has no browser, like an API test?

Can a snapshot capture failure cause my test to fail?

Is the diff tab a real structural diff of the accessibility tree?

TestFly Part 16: Smart triage and fuzzy self-healing, flaky-test diagnosis without tokens

Why flaky-test triage needs a heuristic layer

How fast fuzzy self-healing is

What Auto-POM / PageKnowledge does

How the token-free layer fits with Part 4's LLM self-healing

FAQ

Does SmartTriageEngine call an LLM to classify failures?

What score does FuzzyHealingEngine require before it heals a locator?

Does Auto-POM require every page to be mapped by hand first?

If FuzzyHealingEngine fails, does TestFly fail the test right away?

TestFly Part 15: Enterprise API auth with RFC 7616 Digest, HMAC signing and OAuth2 refresh tokens

Why Digest auth had to be rewritten

How TestFly implements RFC 7616 Digest authentication

Signing requests with HMAC

How OAuth2 token refresh works

Declarative setup with @UseAuth

Other API auth changes in 1.1.0

FAQ

Does ApiAuth.digest(...) make two HTTP calls per request?

What happens if the Digest credentials are wrong?

Can I use a custom HMAC canonical string instead of the default one?

Does oauth2Refresh break if the server never rotates refresh tokens?

TestFly Part 17: Cross-browser reliability with BiDi console capture, Edge/Safari on Grid and guaranteed driver cleanup

How BiDi catches console errors that Firefox used to miss

Knowing which browser features you lose

Running Edge and Safari on Selenium Grid

Making sure no browser is left running

FAQ

Does browser.bidi work with Selenium Grid or BrowserStack/Sauce Labs?

How do I check which console-error capture path is active for the current test?

Can I run the same suite on Chrome, Edge and Safari against one Selenium Grid?

1. Declarative mode with `@LoadTest`

2. Fluent programmatic DSL with `load()`

How `assertThat(x).click()` verifies that a click landed

How `assertThat(x).fill(text)` and `.clear()` verify the field changed

Does `assertThat(x).click()` replace `Locator.click()`?

Can I combine `not()` with the verified `click()`/`fill()`/`clear()`?

What's the difference between `force()` and `actionability.enabled: false`?

Does `ApiAuth.digest(...)` make two HTTP calls per request?

Does `oauth2Refresh` break if the server never rotates refresh tokens?

Does `browser.bidi` work with Selenium Grid or BrowserStack/Sauce Labs?

Does `not()` replace the existing Selenium assertion methods?

Will enabling `locators.strict` break existing tests?

Does strict mode affect `count()` or `elements()`?

Packaging for Kubernetes with Helm (`helm/locust`)

1. The Optimized Container (`k8s.Dockerfile`)

Does `subprocess.Popen` limit load test throughput?