Browser

mite_browser is mite’s HTML-scraping integration, built on mite_http and BeautifulSoup. Despite the name, it does not drive a real browser engine,there is no JavaScript execution or rendering. It parses HTML responses and gives journeys a page/form/link API on top of plain HTTP requests, which is enough to script form submissions and link navigation without hand-rolling HTML parsing in every journey.

Install: pip install mite[browser] (depends on http; no functional change today. Every mite install already includes it; this will matter starting in mite 3.0).

Basic usage

from mite_browser import browser_decorator

@browser_decorator()
async def journey(ctx):
    async with ctx.transaction("Load homepage"):
        page = await ctx.browser.get("https://example.com")
    page.assert_element_in(name="h1", text="Welcome")

browser_decorator() wraps @mite_http and installs ctx.browser, a Browser instance built on top of ctx.http, for the duration of the journey call. It accepts two optional arguments:

• separation passed to mite.ensure_fixed_separation, for pacing requests within the journey.

• embedded_resources if True, every page fetched also downloads its embedded resources (images, scripts, stylesheets, iframes) recursively. This better simulates a real browser’s network footprint, at the cost of extra requests per page.

The Browser object

ctx.browser exposes one async method per HTTP verb used for page navigation (get, post, options, patch) each returning a Page. Anything else, use ctx.browser.request(method, url, ...) directly. Cookie helpers (erase_all_cookies, erase_session_cookies, get_cookie_list) are also available, forwarding to the underlying ctx.http session.

The Page object

A Page wraps an HTML response and lazily parses it with BeautifulSoup the first time you touch .dom, .find(), or .find_all():

page = await ctx.browser.get("https://example.com")
page.find("a", text="Sign in")
page.find_all("div", {"class": "product"})

• assert_element_in(name=..., attrs=..., text=...) raises ElementNotFoundError if nothing matches; useful as a lightweight assertion that a page loaded correctly.

• click_link(text) finds an <a> by its visible text and follows it via ctx.browser.get.

• get_form(name=None) / get_forms() see Forms, below.

• xhr_request(method, url, ...) / xhr_post(url, ...) issues a request with X-Requested-With: XMLHttpRequest and a Referer set to the current page’s URL, for simulating an AJAX call made from the page.

If embedded_resources=True was passed to the decorator, a page’s .scripts, .stylesheets, .resources and .frames lists are populated with the downloaded sub-resources; otherwise they stay empty.

Forms

form = page.get_form()          # first form on the page
form = page.get_form(name="login")
form["username"] = "test"
form["password"] = "hunter2"
result_page = await form.submit()

Form behaves like a dict of its fields (select, textarea, and input elements, with radio groups collapsed into a single field by name). Assigning to a field name sets its value; assigning to a name that doesn’t exist creates one, so you can inject values a form doesn’t declare. submit() sends the form using its declared method and action.

Important

File upload fields are parsed but not actually submitted: Form._serialize only encodes the non-file fields, with a TODO in source noting file support depends on the underlying HTTP backend (acurl) supporting multipart uploads, which it does not today. A form with a file input submits successfully, just without the file.

Metrics

mite_browser emits no metrics of its own. It has no registered stats processor. Every request it makes goes through ctx.http, so the metrics that show up are mite_http’s (see HTTP), not anything browser-specific.

See also

• HTTP the HTTP integration mite_browser is built on