Package com.tnsai.tools.search

Class WebScrapingTools

java.lang.Object

com.tnsai.tools.search.WebScrapingTools

public class WebScrapingTools
extends Object

Function-shape web-scraping tools — function-method replacements for WebScraperTool (Jsoup, key-free) and FirecrawlTool (Firecrawl Cloud API, requires key).

Two typed @Tool methods on a single POJO; the heavy-weight HeadlessBrowserTool (Playwright) and WebCrawlerTool (recursive multi-page crawl) stay legacy until separate batches — Playwright pulls a large native dependency the average agent doesn't need; recursive crawl is an edge-case workflow better served by a dedicated tool.

Since:

0.5.7 (Phase 2 batch 4d — built-in tool migration to function shape, see #188)

Constructor Summary

Constructors

Constructor	Description
WebScrapingTools()	Default constructor — picks FIRECRAWL_API_KEY from env.
WebScrapingTools(String firecrawlApiKey)	Explicit-key constructor (pass null to skip Firecrawl).
WebScrapingTools(String firecrawlApiKey, okhttp3.OkHttpClient http, com.fasterxml.jackson.databind.ObjectMapper mapper)	Full constructor for tests or shared HTTP/JSON instances.

Method Summary

Modifier and Type	Method	Description
String	firecrawl(String url)
String	webScraper(String url, String selector, String attribute)

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Details

WebScrapingTools

public WebScrapingTools()

Default constructor — picks FIRECRAWL_API_KEY from env.

WebScrapingTools

public WebScrapingTools(String firecrawlApiKey)

Explicit-key constructor (pass null to skip Firecrawl).

WebScrapingTools

public WebScrapingTools(String firecrawlApiKey,
 okhttp3.OkHttpClient http,
 com.fasterxml.jackson.databind.ObjectMapper mapper)

Full constructor for tests or shared HTTP/JSON instances.

Method Details

webScraper

@Tool(name="web_scraper",
      description="Fetch a static web page and extract text content. With no selector, returns the page\'s main text (article body if detected, else the full <body>). With a CSS selector, returns the matching element(s) \u2014 text content by default, or a specific HTML attribute when {attribute} is set (e.g. \"href\" for link URLs). Static HTML only (no JavaScript execution); use a headless-browser tool for SPAs.",
      keywords={"scrape","web","html","jsoup","extract","css-selector"},
      idempotent=true)
public String webScraper(@ToolParam(description="Absolute URL to fetch (http or https)")
 String url,
 @ToolParam(description="Optional CSS selector \u2014 when set, returns matching elements only",required=false,defaultValue="")
 String selector,
 @ToolParam(description="Optional HTML attribute to extract (\"href\", \"src\", \"alt\", \u2026) when selector is set",required=false,defaultValue="")
 String attribute)
                  throws IOException

Throws:

IOException

firecrawl

@Tool(name="firecrawl",
      description="Scrape a single URL via the Firecrawl Cloud API and return clean Markdown \u2014 handles JavaScript-rendered pages, anti-bot bypass, and LLM-optimised output in one call. Requires FIRECRAWL_API_KEY (free tier: 500 credits/month from firecrawl.dev).",
      keywords={"scrape","firecrawl","spa","javascript","markdown","ai"},
      idempotent=true)
public String firecrawl(@ToolParam(description="Absolute URL to scrape (http or https)")
 String url)
                 throws IOException

Throws:

IOException