An AI agent can complete a sign-in flow and still miss the security problem behind it. The important question is whether it can establish which account is active, follow a useful workflow, inspect the resulting requests and explain an unexpected permission decision.
HumanBrowser is BreachLine's browser component for that kind of interaction. Its design combines a real browser session, numbered page elements, configurable input behaviour and traffic capture. This article explains the design and provides a practical evaluation workflow for browser-based AI pentesting. The workflow is a proposed test procedure, not a report of benchmark results.
For the broader distinction between automated scanning and an assessment, start with our AI penetration testing guide. For the surrounding product workflow, see Meet Nebula.
What a browser adds to an AI pentest
A browser can establish application state that an isolated HTTP request does not explain: which organisation a user selected, whether a modal changed their role, or what happened between an invitation and its acceptance. Those details help a tester choose the next request to investigate.
The browser view is only part of the evidence. A hidden button does not establish that the corresponding server action is protected. A successful click does not establish that the action was authorised. A useful assessment connects the visible workflow to the server's access decision.
OWASP's authorization testing guidance calls for checking access across identities and privileges, including unauthenticated and logged-out states. Use that as a coverage requirement when evaluating a browser agent. A demonstration in one administrator account cannot answer those questions.
How HumanBrowser grounds a browser action
HumanBrowser collects visible interactive elements and assigns numbers to a page snapshot. The agent can refer to a labelled button or input rather than inventing a selector. A fresh observation matters when navigation, scrolling or a modal changes the page.
The design draws on Set-of-Mark prompting, which adds visual labels to help a model refer to image regions. The original research, published in October 2023, studies visual grounding. It does not establish pentest coverage or guarantee that a model will select the right action.
HumanBrowser's action surface includes navigation, observation, clicking, typing, scrolling and key presses. Its input driver supports curved pointer movement, variable typing delays and pauses. These are interaction settings. They are not evidence that a session will bypass a particular bot defence, and they do not replace an agreed testing arrangement with the application owner.
The session layer also provides proxy configuration, network and console recording, and HAR recording. Its Chromium theater path supports a live view through Chrome's screencast interface. Live viewing helps an operator understand what is happening; request evidence and identity checks establish what the application actually allowed. Latency and frame rate depend on the deployment, so a fixed performance promise would need a measured test setup.
Prepare an authenticated test that can answer a question
Before comparing browser agents, define one small business workflow and its expected permissions. Use accounts and records created for the assessment. Write down the target environment and application version, not just the domain.
For example, consider an application where an owner may invite a member, a viewer may read a shared project and another organisation must remain isolated. This is an illustrative fixture you can adapt, not a claim about a vulnerability found by Nebula.
| Test identity | Intended permission | Evidence to establish before testing |
|---|---|---|
| Organisation A owner | Manage A's test project and invitations | Account identity, organisation and role |
| Organisation A viewer | Read the shared project, without changing membership | A separate authenticated session and the expected role |
| Organisation B member | Access B's records, without reading A's private project | A distinct tenant and its own test record |
| Signed-out session | Reach public pages only | No remaining authenticated state |
Ask the application owner to confirm the intended rules. An undocumented permission is a question to resolve, not automatically a vulnerability.
Keep identities in separate browser contexts and verify them before consequential actions. Playwright documents isolated contexts and reusable authenticated state; it also warns that saved state may contain credentials sufficient to impersonate an account. Store it with the same care as the test credentials and omit secrets from exported evidence.
Run an observe, act, verify loop
Use the following sequence for each agreed workflow:
- Establish identity. Record the active test account, organisation and role. If identity cannot be confirmed, stop that case and mark the access problem.
- Observe the current page. Capture the relevant page state and available controls. Refresh the observation after a state change before selecting another numbered element.
- Take the smallest useful action. Open the test record or use the agreed workflow. Keep potentially destructive actions behind the assessment's scope and approval rules.
- Match the action to traffic. Record the request method, path, relevant parameters and response. Note redirects and session changes.
- Check the outcome. Confirm the expected data or state in the application. A status code by itself can be misleading when a successful HTTP response contains an application error.
- Compare the permitted identities. Repeat the agreed access check using the other test account, keeping its session separate. Record the denied outcome as well as any unexpected access.
Playwright's network APIs support request observation and proxy configuration. A capture still needs review: missing bodies, bounded logs or an incomplete observation window can leave gaps. Record those gaps instead of describing every browser action as fully captured.
For object-level access tests, OWASP recommends using multiple users with distinct owned objects. Its IDOR test procedure is a useful reference for selecting authorised test records. Avoid guessing at unrelated customer data when two prepared accounts can establish the boundary.
What a useful browser finding should contain
A screenshot explains what the tester saw. A reproducible finding also needs the identity, request and application state that made the result possible.
| Evidence field | What the reviewer needs |
|---|---|
| Scope and version | Environment, build or commit, test window and permitted action |
| Preconditions | Role, organisation, relevant record ownership and setup steps |
| Expected result | The permission rule agreed with the application owner |
| Actual result | The precise data or state that crossed the boundary |
| Reproduction | Minimal actions and sanitised requests that reproduce the result |
| Control case | A permitted request that works, plus the corresponding denied case when available |
| Limits | Expired sessions, unavailable routes, missing capture or incomplete checks |
| Retest | The same preconditions exercised against the fixed version |
A finding that relies on another account's cookies is not yet evidence of cross-account access. A test that quietly lost its login is not evidence that authenticated functionality is safe. These are reasons to preserve the identity checks alongside the interesting response.
Our guide to working exploits versus scanner findings explains the evidence standard in more detail.
Evaluate completion, not the smoothness of the demo
During a trial, keep a row for every agreed workflow. Mark it as tested, blocked, excluded or not reached. Require an explanation when the agent cannot pass MFA, loses its session, encounters an unfamiliar control or runs out of its allotted budget.
Measure the work your team must do after the run: time spent restoring access, reproducing findings and correcting unclear reports. Keep that separate from the agent's execution time. A polished live view can help supervision, but it cannot substitute for coverage accounting.
HumanBrowser addresses the interaction layer. The surrounding assessment still needs scope, reasoning, verification and a reviewer who can decide what the result means. Use our AI pentesting tool evaluation worksheet to compare that complete workflow across vendors, and the Nebula product overview to see how it fits BreachLine's offering.
