trading_agent.tools.web_fetch_tool

The web_fetch tool: downloads one web page and returns its main text, title and date.

The page is downloaded with a plain HTTP request and its article text is picked out with trafilatura. That works for most news sites in a fraction of a second. Sites that build the page with JavaScript, or that block automated requests, return no text or HTTP 403; the result then says so and suggests opening the page in the browser instead. Every address, including each redirect, must be on the public internet.

Typical usage example:

tool = WebFetchTool(timeout_seconds=20, maximum_characters=15000)

WebFetchArguments

Bases: BaseModel

The arguments of web_fetch.

Attributes:

Name	Type	Description
url	str	The page's address, starting with http:// or https://.

WebFetchTool

WebFetchTool(timeout_seconds, maximum_characters, transport=None, guard=None)

Bases: Tool

Fetches one page's main text.

Creates the tool.

Parameters:

Name	Type	Description	Default
timeout_seconds	float	How long the download may take.	required
maximum_characters	int	How much of the page's text to return.	required
transport	AsyncBaseTransport | None	What sends the requests; a test passes a fake one, and None means the network.	None
guard	AddressGuard | None	The address check, or None for the normal one.	None

Source code in src/trading_agent/tools/web_fetch_tool.py

def __init__(
    self,
    timeout_seconds: float,
    maximum_characters: int,
    transport: httpx.AsyncBaseTransport | None = None,
    guard: address_guard_module.AddressGuard | None = None,
):
    """Creates the tool.

    Args:
        timeout_seconds (float): How long the download may take.
        maximum_characters (int): How much of the page's text to return.
        transport (httpx.AsyncBaseTransport | None): What sends the requests; a test passes a fake one, and None means the network.
        guard (address_guard_module.AddressGuard | None): The address check, or None for the normal one.
    """
    self._timeout_seconds = timeout_seconds
    self._maximum_characters = maximum_characters
    self._transport = transport
    if guard is None:
        guard = address_guard_module.AddressGuard()
    self._guard = guard

run async

run(arguments)

Downloads the page and extracts its text.

Parameters:

Name	Type	Description	Default
arguments	WebFetchArguments	The page's address.	required

Returns:

Type	Description
ToolResult	The title, date, address and text, or an error saying why the page could not be read.

Source code in src/trading_agent/tools/web_fetch_tool.py

async def run(self, arguments: WebFetchArguments) -> tool.ToolResult:
    """Downloads the page and extracts its text.

    Args:
        arguments (WebFetchArguments): The page's address.

    Returns:
        tool.ToolResult: The title, date, address and text, or an error saying why the page could not be read.
    """
    try:
        response = await self._download(arguments.url)
    except address_guard_module.AddressRefusedError as error:
        return tool.ToolResult.error(str(error))
    except httpx.HTTPError as error:
        return tool.ToolResult.error(
            f'Could not download {arguments.url}: {error!r}'
        )
    if response.status_code in _BLOCKED_STATUSES:
        return tool.ToolResult.error(
            f'{urllib.parse.urlsplit(str(response.url)).hostname} refused the request (HTTP {response.status_code}); it blocks automated downloads. Open it with the browser tools instead, or use another source.'
        )
    if response.is_error:
        return tool.ToolResult.error(
            f'{response.url} answered HTTP {response.status_code}.'
        )
    extracted = await asyncio.to_thread(
        trafilatura.extract,
        response.text,
        url=str(response.url),
        output_format='json',
        with_metadata=True,
        include_tables=True,
        include_links=False,
    )
    if not extracted:
        return tool.ToolResult.error(
            f'{response.url} has no readable article text; it is probably built with JavaScript. Open it with the browser tools instead.'
        )
    document = json.loads(extracted)
    text = document.get('text') or ''
    lines = [
        f'Title: {document.get("title") or "(no title)"}',
        f'Published: {document.get("date") or "date unknown"}',
        f'Address: {response.url}',
        '',
        text[: self._maximum_characters],
    ]
    if len(text) > self._maximum_characters:
        lines.append(
            f'[text cut: {len(text) - self._maximum_characters} more characters]'
        )
    return tool.ToolResult.success(
        untrusted_text.UntrustedText().wrap(
            f'the web page {response.url}',
            '\n'.join(lines),
        )
    )