Data Miner
Extract structured data (JSON) from any web page using a predefined or custom JSON schema, then chat with an AI Assistant over your extraction history.
What Is Data Miner?
Data Miner turns any web page into structured data. Give it a target URL and pick a JSON schema — CRM Lead, Product Catalog, Tech RFP, or a fully custom schema you write yourself — and Data Miner scrapes the page and returns data shaped exactly like that schema, ready to inspect, download, or send onward to another TotalApp module.
Every extraction is automatically saved to a persistent history table, so you can build up a library of structured pulls over time — CRM leads mined from a batch of vendor sites, product data mined from a competitor's catalog, RFP details mined from a client's tender page — and revisit or re-analyze them later.
In one sentence
Point Data Miner at a URL and a schema, and it hands back clean structured JSON instead of raw page text — with every run kept in a searchable, selectable history.
Schema Presets
Three built-in schemas — CRM Lead, Product Catalog, Tech RFP — cover the most common extraction shapes without writing any JSON yourself.
Custom Schema
Write your own JSON Schema for anything the presets don't cover — Data Miner validates the syntax before running the extraction.
History Table
Every extraction is saved automatically. The history table supports multi-select, so you can view, delete, or hand a batch of runs to the AI Assistant at once.
AI Assistant
Chat with your extraction history — summarize what was pulled, spot patterns or missing fields across runs, compare extractions, or ask what to do next.
How It Works
Data Miner scrapes the target URL and passes both the page content and your JSON Schema to an LLM, which returns data conforming to that schema. The result can be viewed as a flattened field-by-field Data View or raw JSON View, downloaded as a .json file, and — for CRM Lead-shaped extractions — sent straight into AI Sales Intelligence as a new pending lead.
Every successful extraction is written to a persistent history table (URL, schema name, and date), which the screen loads on open. Nothing is lost between sessions.
Extraction Schemas
| Schema | What it extracts |
|---|---|
| CRM Lead | Company name, industry, contact name, contact email, value proposition — ready to send to AI Sales Intelligence. |
| Product Catalog | A list of products, each with name, price, description, and SKU. |
| Tech RFP | Title, issuer, due date, requirements list, and scope summary from a tender/RFP page. |
| Custom Schema | Any JSON Schema you write in the built-in editor — validated for syntax before running. |
Sending CRM leads onward
After a successful CRM Lead extraction, a Send to AI Sales Intelligence button appears — it creates a new pending lead there, and you can jump straight to it with View in AI Sales Intelligence.
Extraction History
Below the extraction form, a Recent Extractions table lists every past run with its URL, schema name, and date. Each row supports:
- Select — a checkbox per row, plus a select-all checkbox in the header, for choosing a batch of extractions to hand to the AI Assistant.
- View — reloads that extraction's result into the Data View / JSON View panel above.
- Delete — permanently removes that extraction from history.
When one or more rows are selected, a selection counter and a Clear selection action appear above the table.
AI Assistant
A collapsible AI Assistant panel lives on the right edge of the screen (the tab). It reads either your selected extractions from the history table, or — if nothing is selected — the entire history, and can:
- Summarize what was extracted across the chosen runs
- Spot patterns or gaps across multiple extractions (missing fields, duplicate companies or products)
- Compare extractions against each other
- Suggest next steps — which leads or products look most promising, what to re-extract
Any assistant reply can be saved with Save as Report, which stores it in My Reports tagged with the source "Data Miner Assistant".
Follows your Writer Engine setting
The AI Assistant respects whichever Writer Engine is selected in Settings → Agentic — Local CLI, Cloud API, Ollama, or an in-browser model — so it behaves consistently with every other AI feature in TotalApp.