If you have spent any time writing web scrapers or browser automations over the last couple of years, you already know how the cat-and-mouse game goes. You start with Playwright or Puppeteer. Everything works great on a local test page. Then you point it at a real production site behind Cloudflare Turnstile, DataDome, or Akamai, and you immediately hit a wall.

So you reach for puppeteer-extra-plugin-stealth, or maybe you switch to Patchright, or you start passing twenty different Chromium launch flags to hide navigator.webdriver. Sometimes that buys you a week. Eventually, something else gives you away: a subtle Chrome DevTools Protocol (Runtime.enable) leak, a sterile browser profile with zero browsing history, a mismatch in your JA3/JA4 TLS fingerprint, or the simple fact that an automation binary is spawning a child Chromium process with a debugging pipe attached.

While building Browser Automation Framework (BAF)—a .NET 10 library and companion desktop IDE called BAF Studio—the goal was to step off that treadmill entirely. Instead of trying to make a fake browser look human, BAF just uses the browser you already have open.

Flipping the Architecture Inside Out

Every mainstream automation tool works from the outside in: your code launches a browser binary and drives it over WebDriver or the Chrome DevTools Protocol (CDP).

BAF flips that relationship. You keep using your normal, everyday browser—the one with your actual GPU state, extensions, cookies, active logins, and organic trust score. BAF spins up a lightweight local HTTP/WebSocket server in C# (AutomationServer on 127.0.0.1) and talks to a small JavaScript bridge running directly inside the page.

There are two ways that bridge gets into the browser:

1. An Unpacked Manifest V3 Chrome Extension (Recommended for Chromium browsers): BAF generates the extension files (manifest.json, background.js, content_relay.js, bridge.js, popup.html, and popup.js) right out of the box and keeps them synced with your server port. Once loaded into Chrome, Edge, or Brave, the background worker reports every open tab back to your C# app in real time. You can inject the bridge into a tab with one click, switch tabs exclusively, or detach cleanly when you are done.
2. A Tampermonkey Userscript (Required for Firefox): Because BAF doesn't have a native Firefox extension yet, Firefox users (or anyone who prefers userscripts) can generate a ready-to-paste Tampermonkey script from C# or BAF Studio.

Because no external debugger is attached to the browser, anti-bot scripts looking for CDP artifacts come up empty. To the target site, network requests and TLS handshakes are coming from your everyday browser—because they are.

Surviving Strict CSP Rules and Rate Limits

Running inside the page context introduces two classic headaches: Content-Security-Policy (connect-src) rules blocking local WebSockets, and full-page navigations wiping out your script state.

Dual-Transport IPC (WebSocket + HTTP Polling Fallback)

By default, BAF communicates over full-duplex WebSockets (ws://127.0.0.1:{port}) for low-latency command execution, DOM mutation events, and streaming data.

However, strict sites (like GitHub, banking portals, or heavily locked-down SPAs) often set CSP headers that forbid connecting to ws://127.0.0.1. When that happens, BAF's bridge catches the securitypolicyviolation event (or detects the <meta> policy ahead of time) and automatically drops down to HTTP polling via GM_xmlhttpRequest.

In the Chrome Extension, content_relay.js runs in the ISOLATED world and relays those requests to the background service worker, which has host permissions to talk to 127.0.0.1 regardless of the page's CSP. Your C# code doesn't have to change a single line—BrowserClient switches its Mode property from IpcMode.WebSocket to IpcMode.HttpPolling on the fly and keeps running pending commands.

In-Place Navigation and HTTP 429 Recovery

When you navigate to a same-origin page or click a standard link, BAF's safeNavigate function intercepts the transition, fetches the HTML with your current credentials, swaps the DOM in place, and updates window.history.

This does two things:

1. It keeps the WebSocket connection alive across page turns.
2. It lets BAF inspect the raw HTTP status code and headers before the browser renders the page.

If a site throws an HTTP 429 (Too Many Requests) during a multi-page scrape, BAF doesn't crash your run. The bridge displays a small status banner at the top of the tab, extracts the Retry-After, X-Retry-After, or RateLimit-Reset header (whether it's in seconds, a Unix epoch, or an HTTP-date timestamp), and reports it to DataScraper in C#. The scraper pauses until the cooldown expires, bumps up PageDelayMs for future pages so you stop tripping the limiter, and retries the page automatically.

Dealing with React Without Losing Your Mind

Anyone who has tried to automate a modern React form with document.querySelector('input').value = 'foo' knows that React completely ignores it. React overrides the native value setter on the DOM node to track state internally; if you don't bypass that tracker before dispatching an input event, the component's state resets on the next render.

BAF's ReactExtensions handle this in the bridge:

• SetReactInputValueAsync: Grabs the native HTMLInputElement.prototype value setter directly, applies the value (or uses document.execCommand('insertText')), and fires bubbling input and change events.
• ClickReactElementAsync: Looks for React's internal __reactProps$ or __reactEventHandlers$ keys on the element. If it finds an onClick handler, it invokes it with a Proxy around the native MouseEvent that reports isTrusted: true. If it's a standard element or SPA link, it dispatches a full pointerdown -> mousedown -> pointerup -> mouseup -> click sequence.

Writing this in C# looks like standard async code:

using BrowserAutomationFramework.Extensions;
using BrowserAutomationFramework.Models;
using BrowserAutomationFramework.Server;

using var server = new AutomationServer(port: 9999);
server.SyncDefaultChromeExtension(IpcMode.WebSocket);

server.OnClientConnected += async (sender, client) =>
{
    Console.WriteLine($"Hooked into: {client.DisplayName}");

    await client.NavigateToUrlAsync("https://example.com/login");
    await client.SetReactInputValueAsync("#email", "[email protected]", overwrite: true);
    await client.SetReactInputValueAsync("#password", "hunter2", overwrite: true);
    await client.ClickReactElementAsync("button[type='submit']");

    var title = await client.EvaluateJavaScriptAsync("return document.title;");
    Console.WriteLine($"Landed on: {title.Output}");
};

await server.StartAsync();

Structured Scraping: From Point-and-Click to POCOs

Sometimes you just want to pull a quick table (await client.ScrapeTableAsync("table.results")), and other times you need to crawl fifty pages of product cards with pre-scrape login sequences and regex cleanup.

BAF's scraping engine (DataScraper and the fluent ScrapeBuilder) supports 12 extraction types (InnerText, TextContent, InnerHtml, OuterHtml, Value, Attribute, Href, Src, DataAttribute, ComputedStyle, Exists, and Count).

In code, you can chain a multi-page scrape together and bind the output directly to a C# class:

public class PartListing
{
    public string Sku { get; set; } = "";
    public decimal Price { get; set; }
    public string ProductUrl { get; set; } = "";
}

ScrapeResult result = await client.CreateScrape("PartsCatalog")
    .WithContainer("div.part-row")
    .Field("Sku", "span.sku-code", ScrapeExtractionType.InnerText)
    .Field("Price", "span.price", ScrapeExtractionType.InnerText, regexPattern: @"([0-9]+\.[0-9]{2})")
    .Field("ProductUrl", "a.part-link", ScrapeExtractionType.Href)
    .WithPagination(nextPageSelector: "a.pagination-next", maxPages: 10, pageDelayMs: 1500, deduplicateRows: true)
    .ExecuteAsync();

List<PartListing> parts = result.BindTo<PartListing>();
await result.SaveToCsvFileAsync("parts.csv");

If a site doesn't have a "Next" button, you can swap .WithPagination(...) for .WithUrlPagination("https://example.com/parts?page={page}") or .WithPaginationSequence(customScrollSequence) to run an AutomationSequence between pages.

Reverse-Engineering Live Sites with SiteDumper

Modern web apps love to split their styles and logic across external bundles, inline <style> tags, CSSOM constructed stylesheets, document.adoptedStyleSheets, and dynamically fetched JS chunks that don't even appear in the initial HTML source.

BAF includes a dedicated SiteDumper engine that pulls all of these out of the live page:

• HTML (DumpHtmlAsync): Captures the live document HTML (automatically stripping out BAF's own highlight overlays and temporary script tags) plus any accessible same-origin or cross-origin <iframe> documents.
• CSS (DumpCssAsync): Fetches external <link rel="stylesheet"> files, grabs inline <style> blocks, and serializes runtime CSSOM rules from constructed and adopted stylesheets.
• JavaScript (DumpJavaScriptAsync): Collects external <script src> files, inline script/JSON blocks, and queries performance.getEntriesByType('resource') to catch scripts that were lazy-loaded after page startup.

Calling await dump.SaveToDirectoryAsync(@"C:\SiteDump") writes out a clean folder structure (index.html, css/, js/, combined.css, combined.js, and a full manifest.json inventory).

BAF Studio: The Visual IDE Built on the Same Core

Writing selectors by hand from DevTools gets old fast. That's why the repository includes BAF Studio, a dark-mode WinForms IDE built on top of BrowserAutomationFramework.dll.

Notably, 100% of the logic—even the layout math and virtualized tree state—lives in the UI-agnostic BrowserAutomationFramework class library. BAF Studio is just a thin desktop front-end over AutomationStudioSession.

Here is what BAF Studio gives you out of the box:

1. Virtualized 60fps DOM Explorer & Middle-Click Picker

Standard Win32 TreeView controls choke when you feed them a 15,000-node DOM tree. BAF builds an indexed, syntax-tokenized DomTreeSnapshot on a background thread and renders only the visible rows inside a custom virtualized control (DomTreeView).

• Clicking a node in BAF Studio highlights it in your browser.
• Middle-clicking any element in your browser highlights it on the page and immediately expands and scrolls to that exact node in BAF Studio's DOM tree.

2. Automatic Selector Generalization

When you right-click a DOM node in BAF Studio, you don't just get its raw :nth-child() path. Using SelectorHelper, you can:

• Set as Repeating Container: Automatically generalizes the container path (e.g., turning a specific card into a selector that matches all cards on the page) and converts your existing field selectors into :scope-relative paths.
• Add Repeating Field (All Similar): Walks up the DOM snapshot to find the repeating row or structural wrapper and strips :nth-child(N) from the right segment so a single field extracts an entire column of data.
• Set as Pagination Next Button: Resolves inner <span> or icon clicks up to the parent <a> or <button> and prefers resilient attributes like rel="next", aria-label, or id.

3. Workflow & Reverse-Engineering Tabs

From the same window, you can build multi-step AutomationSequence lists, hook them up as Pre-Scrape or Post-Scrape actions, preview scrape results in a live table or JSON view, and run one-click HTML/CSS/JS site dumps with instant code search. Everything saves to a single, portable .json project file that you can either run in the GUI or load headlessly in C# via AutomationProject.LoadFromFile(...).

Extending the Bridge with Custom JS & Streaming

If the built-in actions (Click, SetValue, Navigate, EvaluateScript, Scrape, DumpSite) aren't enough, you can inject your own JavaScript handlers into the bridge via ScriptOptions.CustomScript and stream intermediate results back to C# as they happen:

registerAction('INFINITE_SCROLL', async (payload) => {
    for (let step = 1; step <= payload.maxSteps; step++) {
        window.scrollTo(0, document.body.scrollHeight);
        await new Promise(r => setTimeout(r, 800));
        // Stream progress back to BrowserClient.OnStreamDataReceived
        await window.streamData(payload.taskId, { step, height: document.body.scrollHeight });
    }
    return 'Scroll finished';
});

On the C# side, you can register custom step handlers on AutomationRunner using runner.RegisterCustomActionHandler(...) so your custom logic plugs right into saved project sequences.

Wrapping Up & Where to Get It

Playwright and Selenium are still great for CI/CD testing where you want a clean, disposable browser environment on a build server.

But when you are scraping rate-limited data, automating workflows inside authenticated sessions, reverse-engineering a complex front-end, or dealing with aggressive bot protection that flags headless and CDP-driven browsers on sight, automating your real browser session is a much calmer way to work.

To try it out, clone the repo, run dotnet run --project "BAF Studio/BAF Studio.csproj", load the auto-synced Chrome Extension folder from BrowserAutomationFramework/ChromeExtension (or grab the Tampermonkey script for Firefox), and middle-click an element on any open tab.

You can find the source code, full API reference, and BAF Studio at either of the project repositories:

• Church of Malware Git: https://git.churchofmalware.org/mastercodeon/baf
• GitHub: https://github.com/codeon314/baf