ML Prediction Engine - Engines

What Is the ML Prediction Engine?

The ML Prediction Engine is one of TotalApp's backend engines. It runs entirely server-side, in server/engines/MLPredictionEngine.ts, and its job is to turn raw CRM pipeline data — deal stage, deal value, and last-contact recency — into a deterministic RFM (Recency / Frequency / Monetary) score for every contact-deal pair, then group those scores into five behavioral clusters and compute a churn-risk percentage and a predicted lifetime value (LTV) for each one.

It was ported from an earlier client-side implementation so the resulting scores could be written back onto the CRM Deal records that produced them — a real data mutation, not just a read-only report. Every run is tenant-scoped, deterministic (the same input pipeline and weights always produce the same clusters), and recorded in a run history so past recalculations remain auditable.

In one sentence

Give the ML Prediction Engine your CRM pipeline data and a set of RFM weights — it returns five customer clusters (Champions, Loyal Core, Potential Growth, At Risk, Low-Value Hibernating), each with a churn-risk score and average LTV, and — if CRM Auto-Sync is on — writes a per-deal churn risk and predicted LTV back onto the source deals.

How It Works — the RFM Pipeline

Every recalculation run walks the same deterministic pipeline, regardless of which screen triggered it:

1. Build Vectors → 2. Score R / F / M → 3. Assign Clusters → 4. Compute Churn Risk & LTV → 5. Optional CRM Write-Back

Phase	What happens
1. Build Vectors	Every deal across the selected pipeline scope (or all pipelines) becomes one raw vector: recency in days since last contact (with a lookup table of stage-specific penalties — e.g. `closed-won` = 0 days, `churned` = 365 days — falling back to the deal's actual `lastContact` date when the stage isn't in the table), a frequency weight derived from the deal's pipeline stage (1.0 for active/won/renewed, down to 0.15 for anything else), and the deal's monetary value (clamped to zero or above).
2. Score R / F / M	Recency, frequency, and monetary values are each normalized to 0–100 relative to the current run's own min/max range (recency is inverted, so more recent contact scores higher). A composite score is then computed as the weighted sum of the three normalized axes using the run's RFM weights.
3. Assign Clusters	Each scored contact is matched against five fixed cluster definitions by its R/F/M score ranges (Champions, Loyal Core, Potential Growth, At Risk, Low-Value Hibernating). If no range matches exactly, a composite-score fallback assigns the closest cluster.
4. Compute Churn Risk & LTV	Each cluster gets a churn-risk percentage derived from its tone (At Risk and Hibernating clusters start from a high base risk; Champions from a low one) adjusted by the cluster's average recency score. Average LTV per cluster is the mean deal value of its members.
5. Optional CRM Write-Back	Only runs when CRM Auto-Sync is enabled in engine settings — see the dedicated section below.

Why five fixed clusters?

Champions, Loyal Core, Potential Growth, At Risk, and Low-Value Hibernating are the same segment names used across TotalApp's Marketing module. Keeping the cluster taxonomy fixed (rather than dynamically discovered) means every tenant's Customer Segmentation dashboard and every AI Predictive Models run history entry speaks the same vocabulary, and the RFM weights are the only thing a tenant can meaningfully tune.

Input — RFM Weights and Scope

Every recalculation is configured with the same two pieces of information, whether it's triggered from Customer Segmentation or from AI Predictive Models:

Recency Weight (r)

How heavily "days since last contact" counts toward the composite score. Default 0.34. Recency is inverted before weighting — more recent contact always scores higher.

Frequency Weight (f)

How heavily pipeline-stage-derived engagement weight counts. Default 0.33. Active, won, and renewed deals score highest; cold/lost stages score lowest.

Monetary Weight (m)

How heavily deal value counts. Default 0.33. Normalized against the highest deal value seen in that same run.

Scope

Either all pipelines or a single pipeline ID. Customer Segmentation lets an analyst pick a scope per ad-hoc run; AI Predictive Models always recalculates with scope all using the tenant's saved default weights.

Weights are always normalized

Whatever three numbers are submitted, the engine clamps each to the 0–1 range and rescales all three so they sum to exactly 1 before scoring. If all three weights are zero, it falls back to the default 0.34 / 0.33 / 0.33 split rather than dividing by zero.

CRM Auto-Sync — Dry-Run vs. Write-Back

The engine can operate in two modes, controlled by a single tenant-wide crmAutoSync boolean configured on the AI Predictive Models screen (default: on).

Mode	Behavior
CRM Auto-Sync OFF (dry-run)	The engine computes clusters, churn risk, and LTV, and persists the run as the latest segmentation snapshot — but never touches the underlying `crm-pipeline.json` file. Deal records are read-only inputs in this mode.
CRM Auto-Sync ON (write-back)	After clustering, the engine walks every deal in every pipeline, looks up its assigned cluster's churn-risk score and its own monetary value as the LTV estimate, and writes three fields directly onto the deal: `churnRiskScore`, `predictedLtv`, and `scoredAt`. The updated pipeline store is only written back to disk if at least one deal was actually scored.

Predicted LTV is not yet a separate model

In this phase, predictedLtv written onto a deal is simply that deal's own current monetary value — there is no separate LTV forecasting model yet. The churn-risk score, by contrast, is genuinely derived from the deal's cluster membership (which reflects its own recency/frequency/monetary standing relative to the rest of the tenant's pipeline). This is documented in the engine source as a known simplification for the current phase.

Read-only when there's nothing to sync

If a tenant has no persisted CRM pipeline data yet (a brand-new tenant, or one that never saved the CRM Sales Pipeline screen), the engine returns an empty run with zero members rather than erroring — there is simply nothing to cluster or write back until pipeline data exists.

Run History — Audit Trail

Every recalculation — whether triggered manually or scheduled — appends a summary entry to a tenant-scoped run history, capped at the most recent 20 runs. Older entries roll off automatically; only the summary is kept per run (member counts, revenue), not the full per-contact detail, to keep the store small.

Field	Meaning
`runId`	Unique identifier for the run, timestamp-based.
`computedAt`	ISO timestamp of when the run executed.
`scope`	`all` or a specific pipeline ID.
`totalMembers`	Total contact-deal pairs scored in this run.
`totalRevenue`	Sum of monetary value across all scored deals.
`crmMutated`	True only if CRM Auto-Sync was on and at least one deal was actually written to.
`dealsScored`	Count of deal records that received a write-back in this run (0 in dry-run mode).

This history is what powers the Run History table on the AI Predictive Models screen — each row shows exactly when a recalculation happened, how large it was, and whether it actually mutated CRM data, so a tenant admin can audit what changed and when.

Where the ML Prediction Engine Is Used

Both consuming screens call the exact same server-side engine and the exact same /api/ml-prediction/recalculate endpoint — they differ only in how they present the results and which settings they expose.

Customer Segmentation (Marketing)

Ad-hoc, analyst-driven segmentation. Lets a marketer pick a scope (all pipelines or one) and adjust RFM weights per run to explore different views of the same customer base, without changing the tenant's saved defaults. See Customer Segmentation.

AI Predictive Models (Orchestration)

The engine's config and monitoring surface. Sets the tenant-wide default RFM weights, toggles CRM Auto-Sync, exposes a "Recalculate Now" action using the saved defaults, and shows the KPI strip and Run History table. See AI Predictive Models.

One engine, one settings store

Both screens read and can trigger the same marketing/ml-prediction-settings.json tenant file and the same marketing/audience-clusters.json latest-run/run-history file. A recalculation triggered from either screen updates the shared run history that both screens display — there is no separate "segmentation" engine and "predictive models" engine under the hood.

Config Surface — Tenant Settings

Engine configuration is a single tenant-wide JSON blob, stored at marketing/ml-prediction-settings.json and managed through the plain /api/data/ml-prediction-settings GET/POST endpoint (no bespoke handler needed for this simple store, per the Module Folder Rule).

Setting	Type	Default	Effect
`crmAutoSync`	boolean	`true`	Gates whether a recalculation writes churn risk / predicted LTV back onto CRM deal records.
`defaultRfmWeights.r`	number (0–1)	`0.34`	Default recency weight used by AI Predictive Models' "Recalculate Now" and shown as the starting slider value on Customer Segmentation.
`defaultRfmWeights.f`	number (0–1)	`0.33`	Default frequency weight.
`defaultRfmWeights.m`	number (0–1)	`0.33`	Default monetary weight.

Endpoints

POST /api/ml-prediction/recalculate runs the full pipeline and returns the fresh AudienceCohortRun (accepts optional weights and scope in the body). GET/POST /api/data/ml-prediction-settings reads or saves the tenant's default weights and Auto-Sync toggle. Both require standard app-level authentication and resolve the tenant from the authenticated session — never from a client-supplied tenant ID.

Frequently Asked Questions

Does this overwrite my CRM data automatically?

Only if CRM Auto-Sync is turned on (it's on by default). When it's off, the engine still computes and saves segmentation results, but it never touches your CRM deal records — recalculating is a read-only, dry-run operation in that mode. Toggle it from AI Predictive Models.

What if I have no CRM pipeline data yet?

Recalculating returns an empty run with zero members and zero revenue rather than an error. Once pipeline deals exist (via the CRM Sales Pipeline screen), subsequent recalculations will start producing real clusters.

How is churn risk calculated?

Each of the five clusters has a base risk percentage tied to its tone — At Risk and Low-Value Hibernating start high (75 and 88), Potential Growth is mid (45), Loyal Core is low (20), and Champions is lowest (8). That base is then nudged upward the further the cluster's average recency score falls below 60, so two "At Risk" runs with different actual staleness still get differentiated risk scores.

Can I change the default RFM weights?

Yes, from the AI Predictive Models screen — the three sliders (Recency / Frequency / Monetary) save as the tenant's new defaults. Customer Segmentation lets you override weights for a single ad-hoc run without changing those saved defaults.

Is predicted LTV a real forecast, or just current deal value?

In this phase, predicted LTV written back to a deal is that deal's own current monetary value, not a projected future value — there is no dedicated LTV forecasting model yet. The churn-risk score, however, is a genuine derived signal based on the deal's cluster membership.

How far back does the Run History go?

The 20 most recent runs are kept per tenant; older entries are dropped automatically as new runs are appended. Each entry is a compact summary (counts and totals), not the full per-contact breakdown of that run.

Is this the same thing as the Matrix Agent?

No. The Matrix Agent scores unstructured text against weighted rules using an LLM. The ML Prediction Engine is a deterministic numeric pipeline — it does not call an LLM at all; it computes RFM scores and churn risk purely from structured CRM deal fields (stage, value, last contact date).

Is ML Prediction Engine data tenant-isolated?

Yes. Settings, the latest run, and run history are all stored per-tenant under marketing/, and the engine reads the source CRM pipeline from the same tenant's crm/crm-pipeline.json — one tenant's segmentation data and CRM deals are never visible to another tenant.