Website scraping

Website scraping ingests your public marketing site from a URL you provide. The system discovers pages and builds structured knowledge: paths and titles, high-level signals (structure, offer, pricing hints, branding cues), and raw page text that can be compressed into a single markdown-style corpus for the agent. You can re-run analysis when you change positioning or publish new key pages.

The agent’s answers stay tied to what you already published—value props, plans, legal pages—so you spend less time retyping the same facts into a separate FAQ.

When the crawl is refreshed, downstream compressed knowledge can be rebuilt so ongoing chats reflect your current site, within the limits of what is publicly reachable without signing in.

Point the analyzer at the canonical URL of your main site and run it after meaningful launches (pricing, positioning, new sections).

Keep critical conversion facts on crawlable HTML pages; content that only exists behind login or in client-only bundles may not be picked up. Combine scraping with PDFs uploaded under Dashboard → Agent → Knowledge for material that does not live cleanly on the web.