Ingestion Engine scrapes, crawls, and extracts the web on demand — powered by a direct REST API integration — and hands your app clean markdown or JSON-schema data, ready to use.
Point scrapeToMarkdown at any page — a product listing, a news article, a landing page — and get back clean, readable markdown with boilerplate, ads, and navigation chrome stripped out.
crawlSite follows links across a domain — with depth and page-count limits you control — and returns every page's content in one structured batch. Ideal for pulling in a supplier catalogue or a documentation site.
extractStructuredData pairs LLM-based extraction with a JSON schema you define. Point it at a page and get back exactly the fields you asked for — no manual parsing, no regex.
Ingestion Engine is a dedicated web scraping and structured extraction service. Two modules in TotalApp already call it directly, no setup required.
IngestionEngine.ts service — one connection, reused everywhereThe biggest weakness of a classic API-based model is that it has no idea what happened after training. Ingestion Engine can sit behind the scenes as a live RAG feeder, pulling in fresh context the moment it's needed.
Ingestion Engine wraps a direct web scraping REST API into three core operations, plus the module wiring that puts scraped data straight into your CRM and project workflows.
Single-page scraping that returns clean markdown with boilerplate stripped, ready for documents, prompts, or workflow nodes.
Whole-domain crawling with depth and page-limit controls so you can ingest a catalogue or doc site without runaway cost.
Define a JSON schema and let LLM-based extraction pull exactly the fields you need from any page layout.
Add Lead in Sales Intelligence auto-fills company details from a single pasted URL.
Import from URL in Project Assets crawls an RFP or spec site and returns structured markdown sections.
Every scrape, crawl, and extract call runs under the requesting tenant's own scope — no cross-tenant data ever mixes.