Integration & Webhook Engine
The front door between TotalApp and every outside platform that talks to it. It absorbs bursts of inbound webhooks from Shopify, WooCommerce, Odoo, Wix and custom systems, accepts each one in milliseconds, and processes the real work safely in the background — so a traffic spike from one integration never slows down the rest of the app.
What Is the Integration & Webhook Engine?
Every external platform TotalApp connects to — an ecommerce store, an ERP system, a website builder — needs a way to tell TotalApp "something just happened." A new order was placed. A product was updated. A customer record changed. The mechanism almost every platform uses for this is a webhook: the moment the event happens, the external platform fires an HTTP request straight at a URL TotalApp exposes, carrying the event's details in the request body.
The problem this engine solves is what happens on TotalApp's side of that request. If TotalApp tried to fully process every webhook — validate it, transform it, write it to storage, maybe call another AI engine — while the external platform is still waiting for a response, two things go wrong. First, if that processing takes more than a few seconds, the external platform times out and considers the delivery failed, so it retries, and now the same event might be processed twice. Second, if hundreds of these arrive at once — a flash sale, a bulk product import — the app's main server spends all of its time doing this work instead of serving the people actually using TotalApp at that moment.
The Integration & Webhook Engine exists specifically to prevent both problems. It splits every inbound webhook into two separate moments: an acceptance step that happens instantly (so the sender never times out and never needs to retry), and a processing step that happens afterward, in the background, at a pace the rest of the app can absorb.
In one sentence
The engine says "got it, I'll handle this" to the sender within milliseconds, then does the actual work — safely, with automatic retries — completely out of view of that original request.
How It Works — Accept Instantly, Process Later
Every inbound webhook goes through the same two-stage lifecycle:
| Stage | What happens |
|---|---|
| 1. Accept & Queue | The engine checks that the request has the two pieces of information it absolutely needs — which workspace it belongs to, and which platform sent it — validates the request's signature if one is configured, and immediately places the event on an internal queue. The sender gets an "accepted" response back within milliseconds, before any real processing has started. |
| 2. Process in the Background | A background worker picks events off the queue at a controlled pace (a handful at a time, never all at once) and runs the actual integration logic — for example, writing a validated record into a Schema & Data Engine collection, or feeding the content into the Matrix Agent for scoring. If this step fails for a transient reason, the event is automatically retried a few times before being marked as failed. |
Why not just process everything immediately?
Because the sender is a machine, not a person patiently waiting — most platforms give a webhook delivery only a few seconds before declaring it failed and resending it. If TotalApp did all of its work inline, a slow moment (a busy server, a large payload, a downstream engine taking longer than usual) would look like a failure to the sender, triggering duplicate deliveries of the same event. Separating "accept" from "process" removes that entire class of problem.
Input — What Every Inbound Webhook Must Carry
Regardless of which platform is sending it, every webhook accepted by this engine carries the same two required pieces of information, plus whatever event-specific data the platform includes:
Workspace Identifier
Which TotalApp workspace (tenant) this event belongs to. Every collection, log, and downstream record the engine touches is scoped to this workspace and to no other — one workspace's webhook traffic is never visible to another.
Source Platform
Which system sent the event — Shopify, WooCommerce, Odoo, Wix, or a custom integration. This determines which signature secret (if any) is used to validate the request and which downstream logic processes it.
Event Name (optional)
A short label for what happened — e.g. order.created, product.updated — used for filtering, logging, and routing the event to the right downstream handler.
Event Payload
Whatever data the source platform includes about the event itself — an order's line items, a product's updated price, a customer's contact details. This part is entirely platform-specific and passed through as-is.
Signature verification, when configured
If a workspace has registered a signing secret for a given source platform, every inbound request from that platform must include a matching signature header. A request with a missing or incorrect signature is rejected before it ever reaches the queue. Until a secret is registered, requests are accepted without this check — useful during initial setup, but a signing secret should always be configured before going live with a real integration.
Output — What You Get Back
The engine responds to the original webhook request almost immediately, and separately exposes a way to check on an event's progress afterward:
| Result | Meaning |
|---|---|
| Accepted (202) | The event passed its basic checks and was placed on the queue. The response includes a job identifier that can be used to check on its progress later. This does not mean the event has finished processing — only that it has been safely received. |
| Missing workspace or platform (400) | The request didn't include one of the two required identifiers, or the workspace identifier doesn't correspond to a real workspace. Nothing is queued. |
| Invalid signature (401) | A signing secret is configured for this platform, but the request's signature didn't match it. The request is rejected outright — this is the engine's primary defense against spoofed webhook traffic. |
| Payload too large (413) | The request body exceeded the engine's size limit. Extremely large payloads are rejected rather than accepted and left to fail later during processing. |
| Job status — queued / processing / done / error | Polling the job status endpoint with the identifier returned at accept time reports exactly where that event is in its lifecycle, including the specific error message if it ultimately failed after all retries. |
Accepted is not the same as processed
A 202 Accepted response is a receipt, not a confirmation of success. The event still has to be picked up by a background worker and run through its actual processing logic — which can fail, retry, and eventually succeed or give up. Any integration that needs to know the final outcome of an event should check the job status afterward rather than assuming acceptance means completion.
Reliability — Retries, Backoff, and Isolation
Because background processing can fail for reasons that have nothing to do with the event itself — a momentary disk hiccup, a downstream engine being busy — the engine treats failures as expected and recoverable rather than fatal:
Automatic Retries
An event that fails to process is retried automatically, up to three times, with an increasing delay between attempts (a few seconds, then longer). Only after all attempts are exhausted is the event marked as permanently failed.
Controlled Concurrency
The background worker only processes a small, fixed number of events at the same time, no matter how many are waiting in the queue. A burst of a thousand events is absorbed smoothly instead of overwhelming the app all at once.
Isolated Failures
One malformed or unusually large event failing to process never affects any other event in the queue, and never brings down the acceptance endpoint itself — a sender can always successfully deliver a new event even while a previous one is stuck retrying.
Durable History
Every event that finishes processing — successfully or not — is recorded in that workspace's own activity log, so a later "what happened to my webhooks this week" review doesn't depend on catching it in the moment.
What a restart can and can't lose
Events that have already finished processing — successfully or after exhausting their retries — are permanently recorded and cannot be lost. Events that are still waiting or mid-retry at the exact moment the server restarts are the only ones at risk, since they exist only in memory until they finish. For most webhook integrations this is an acceptable trade-off, since the sending platform's own retry behavior (most platforms retry failed/timed-out deliveries on their own schedule) provides a natural safety net.
Keeping Webhook Traffic Trustworthy
Anyone who knows a workspace's webhook URL could, in principle, send it fake events. To prevent that, each workspace can register a signing secret per source platform — the same secret is configured both in TotalApp and on the external platform's side, and every genuine webhook delivery includes a signature computed from that secret.
Once a signing secret is registered for a platform, every future request claiming to be from that platform must present a matching signature — anything else is rejected before it ever reaches the queue, let alone any downstream data.
Set this up before going live
During initial setup, before a signing secret has been registered, the engine still accepts webhooks from a new integration so it can be tested end-to-end. Once an integration is live and handling real data, registering a signing secret closes that window and is strongly recommended for every production integration.
How End Users Actually Use It
Connecting an External Platform
From an integrations settings screen, a workspace admin:
- Picks which platform to connect — Shopify, WooCommerce, Odoo, Wix, or a custom system.
- Copies the webhook URL TotalApp generates for that workspace and pastes it into the external platform's own webhook settings.
- Optionally generates a signing secret, entering the same value on both sides so future deliveries can be verified.
- Triggers a test event from the external platform (e.g. a test order) to confirm the connection is accepted end-to-end.
- From then on, every matching event from that platform arrives automatically — no further action required.
Watching Activity in Real Time
A "Recent Webhook Activity" view lists every event as it arrives — which platform it came from, when, and whether it's still queued, currently processing, finished successfully, or failed after all retries. This gives a workspace admin visibility into their integration traffic without needing to check server logs.
Staying Responsive Under Load
For a business running a busy online store, this engine is what keeps the rest of TotalApp — dashboards, reports, every other screen — feeling fast even during a sales spike that produces hundreds of order webhooks within a few minutes. The webhook traffic is absorbed and processed steadily in the background instead of competing with everyone else using the app at that same moment.
Where the Integration & Webhook Engine Fits
This engine is the entry point for external data — what it accepts gets handed off to other engines for the actual domain-specific work:
| Consumer | How it uses the Integration & Webhook Engine |
|---|---|
| Schema & Data Engine | An accepted webhook's payload — a new order, an updated product — can be written as a validated record into a user-designed collection, so unreliable external data always passes through the same field-level validation as a person filling in a form. |
| Matrix Agent | Freeform content arriving through a webhook — a new lead form submission, a vendor's updated profile — can be routed straight into a scoring evaluation, so external events feed the same 0–100 scoring pipeline as manually entered content. |
| Audit Agent | A webhook referencing a newly uploaded document can hand that document off to the same audit pipeline a manually uploaded file goes through. |
| Ecommerce integrations (Shopify, WooCommerce) | Order, product, and customer events from connected storefronts flow through this engine before appearing anywhere else in TotalApp — dashboards, reports, or automated workflows. |
Not every real-time integration goes through this engine
Some integrations are intentionally built for instant, always-live behavior instead — for example, a workflow node that must fire the moment an event arrives, with no queueing delay at all. Those integrations use a different, direct-delivery mechanism built specifically for that always-on requirement. The Integration & Webhook Engine is for the much more common case: an event that can be processed a moment later, safely and reliably, without needing to be seen live the instant it happens.