Ingestion Engine

The backend engine that turns any URL into clean markdown or structured JSON — TotalApp's web scraping, site crawling, and structured extraction service.

What Is the Ingestion Engine?

The Ingestion Engine is one of TotalApp's backend engines. Its job is to take a raw URL — a company website, a documentation site, a blog, an RFP page — and turn it into something the rest of the platform can actually use: clean markdown, or structured JSON that matches a schema you define.

It wraps a direct web scraping REST API so every module that needs live web data (CRM, Project Management, the AI Assistant) calls the same three operations instead of each screen re-implementing its own scraper. Every request runs under the requesting tenant's own scope, so scraped data never mixes between tenants.

In one sentence

Give the Ingestion Engine a URL — it returns clean markdown for a single page, a full crawl of a site within a page/depth limit, or a JSON object that matches a schema you supply, extracted by an LLM directly from the live page.

Core Operations

Every Ingestion Engine call is one of three primitives. All higher-level module integrations (CRM lead enrichment, project document import) are built on top of these same three functions.

scrapeToMarkdown(url)

Single-page scraping that returns clean markdown with boilerplate (nav bars, ads, cookie banners) stripped out. Ready to drop into a document, a prompt, or a workflow node. Returns the resolved source URL, the markdown body, and the page title.

crawlSite(url, options)

Whole-domain crawling starting from a URL. Takes a limit (max pages, default 25) and a maxDepth (max link-discovery depth, default 2) so a documentation site or catalogue can be ingested without runaway cost. Returns one markdown page per discovered URL.

extractStructuredData(url, jsonSchema)

Define a JSON schema describing exactly the fields you want, and LLM-based extraction pulls them straight from the live page — regardless of the page's HTML layout. This is the primitive behind CRM lead enrichment and technical document import.

Why three separate primitives?

Each one answers a different question. "Give me this one page as text" is scrapeToMarkdown. "Give me an entire site as text" is crawlSite. "Give me specific fields, structured, regardless of how the page is laid out" is extractStructuredData. Module integrations pick whichever primitive fits the job — lead enrichment needs structured fields, so it uses extraction; technical document import needs full page content across many pages, so it uses a crawl.

Module Integrations

Two modules already call the Ingestion Engine directly, with no setup required from the tenant.

Integration	Entry point	What it does
CRM Lead Enrichment `enrichCRMLead(companyUrl)`	Sales Intelligence → Add Lead form	Paste a company URL and the form auto-fills from the extracted schema — no manual data entry.
Technical Document Import `ingestTechnicalDoc(docUrl, moduleType)`	Project Assets → Import from URL	Crawls a spec, RFP, or documentation site (up to 40 pages, depth 3) and returns structured markdown sections tagged for the MES or PM module.

CRM lead enrichment schema

Extraction pulls exactly five fields from the pasted company URL:

companyName
industry
productList (array of strings)
contactEmail
valueProposition

MES / PM document import

Used for importing manufacturing and project specification documents (MES = manufacturing execution content, PM = project management content). The crawl walks the linked structure of a spec site and hands back each page as a titled markdown section with its own source URL, so the importing screen can render or file each section individually.

Live Knowledge — Acting as a RAG Feeder

A classic API-based language model only knows what was in its training data — it has no idea what happened on a company's website yesterday. The Ingestion Engine can sit behind TotalApp's chat and terminal surfaces as a live retrieval-augmented-generation (RAG) feeder, scraping fresh context the moment a query needs it instead of relying on a stale cache.

1. User asks a question referencing a live page → 2. Engine scrapes the page on the spot → 3. Fresh content is fed to the model as context

For example, asking "Summarize this company's latest blog posts" from the AI Assistant chat or a terminal triggers an on-the-spot scrape of the referenced page — the answer is grounded in what's actually on the page right now, not a stored snapshot.

Smart Wait

Modern sites are frequently JS-heavy — content loads asynchronously, animates in, or renders client-side after the initial page load. Smart Wait holds off extraction until the page has actually finished rendering, instead of grabbing a half-loaded DOM. This means even heavily animated or async-loaded modern web apps get scraped completely, not just their empty initial shell.

Developer Notes

The Ingestion Engine is a backend-only service — there is no dedicated screen for it. It's called from server-side handlers wired into server/server.ts, each requiring the standard app-level authentication (requireAppAuth) and a JSON POST body.

Handler	Body	Calls
`handleIngestionScrape`	`{ url }`	`scrapeToMarkdown`
`handleIngestionCrawl`	`{ url, limit?, maxDepth? }`	`crawlSite`
`handleIngestionExtract`	`{ url, jsonSchema }`	`extractStructuredData`
`handleIngestionCrmLeadEnrich`	`{ companyUrl }`	`enrichCRMLead`
`handleIngestionTechnicalDoc`	`{ docUrl, moduleType }`	`ingestTechnicalDoc`

Failures surface, they don't fail silently

Every core operation wraps its underlying request and re-throws a descriptive error (including the source URL) on failure — a scrape returning no markdown, a crawl returning no pages, or an extraction returning no data all raise explicit errors rather than returning empty results silently. Handlers propagate these as HTTP 502 responses with the message included.

Frequently Asked Questions

Does the Ingestion Engine store scraped content permanently?

No, not by itself — it returns the scraped/extracted result to the calling module, which decides what to do with it. CRM lead enrichment writes the extracted fields onto the new lead record; technical document import hands sections to the importing screen to file. The engine itself is stateless per request.

What happens if a page can't be scraped (blocked, times out, 404s)?

The call rejects with a descriptive error identifying the URL and the underlying failure reason, and the HTTP handler returns a 502 with that message. Nothing is silently returned as empty — the calling screen sees the real failure.

How is crawl cost controlled?

crawlSite always takes a limit (max pages) and maxDepth (max link-discovery depth), defaulting to 25 pages / depth 2. Callers that need a deeper crawl — like technical document import, which uses 40 pages / depth 3 — explicitly opt into the higher limit rather than crawling unbounded.

Is this the same thing as the Matrix Agent?

No. The Matrix Agent scores text you already have against weighted rules. The Ingestion Engine's job is upstream of that — getting the text (or structured data) off the web in the first place. A common pattern is scraping a page with the Ingestion Engine, then handing the resulting markdown to the Matrix Agent for scoring.

Is Ingestion Engine data tenant-isolated?

Yes. Every scrape, crawl, and extract call is authenticated and runs under the requesting tenant's own scope — one tenant's scraped or extracted data is never visible to another tenant.