Data Gen
User Guide — Building and Running the Data Pipelines That Populate an Enterprise Object
| Last updated: August 2026
Contents
- Overview
- Key Concept: Mappings, Plans, and Pipelines
- Finding Your Way Around
- The Object Header: KPIs and Commit All Clean
- Build & Preview
- Checks Before You Commit: Preflight and Decision Gates
- Overriding the Generated Plan
- Committing, Draft Status, and Staleness
- Run: Executing the Pipelines
- History
- Quick Reference
1. Overview
Data Gen is where you turn an Enterprise Object's mappings into working data pipelines — and then run them to actually produce the object's data. In short: it reads how your source data maps to the object, builds the ETL (the extract-transform-load steps) automatically, lets you preview and fine-tune it, and then runs it to populate the object's target tables.
You work in Data Gen one mapping at a time (a mapping is one source feeding the object). For that mapping, Data Gen generates a plan, you review and optionally adjust it, you commit it — which turns it into real, runnable pipelines — and then you run it. A history of every commit is kept.
What you can do here
- Generate the ETL for a mapping automatically from its column mappings — no hand-building.
- Preview the plan table by table before anything is committed.
- Check for problems first — schema mismatches against the live source, and other validation gates.
- Override how the plan is built when the automatic result needs adjusting.
- Commit the plan to create the runnable pipelines and jobs.
- Run the pipelines — all at once or one at a time — and watch their progress.
- See the history of what was committed and when.
WHERE TO FIND IT — Data Gen is a tab on the Enterprise Object, next to Rule Pilot. Open the object, choose the layer, and select the Data Gen tab. Data Gen always works on the mapping you have selected for the object.
2. Key Concept: Mappings, Plans, and Pipelines
Three ideas connect everything in Data Gen.
A mapping describes how one source feeds this Enterprise Object — which source columns land in which of the object's target columns. An object can have several mappings (one per source). Data Gen operates on the mapping you're focused on.
A plan is what Data Gen builds when you click Generate ETL. It reads the mapping and works out, for each target table, the steps needed to move and shape the data — grouped into stages (extract the data, transform it, then confirm it into the target). The plan is a draft: nothing runs and nothing is written until you commit it.
A pipeline is what the plan becomes once you commit it — a real, runnable sequence of jobs (one pipeline per target table). Committing is the moment the draft plan turns into something you can run.
So the everyday rhythm is: Generate ETL (build the draft plan) → review and optionally Override → Commit (create the pipelines) → Run (produce the data).
NOTHING IS LIVE UNTIL YOU COMMIT — Generating and previewing a plan is completely safe: it reads the mapping and shows you what it would build. Only Commit creates the pipelines, and only Run actually executes them. You can generate and re-generate freely while you get the plan right.
3. Finding Your Way Around
Open the Enterprise Object, choose the layer, and select the Data Gen tab. Once you've selected a mapping for the object, the page has three parts:
- The object header — a strip of four KPI cards (Columns, Pipelines, Stale, Decision gates) and a Commit all clean button.
- The sub-tabs — Build & Preview, Run, and History.
- The working area — the content of whichever sub-tab is active.
The three sub-tabs map to the three things you do here: build and review the plan, run the resulting pipelines, and look back at what's been committed.
| Sub-tab | What it's for |
|---|---|
| Build & Preview | Generate the plan, review it table by table, override it if needed, and commit it. |
| Run | Execute the committed pipelines and watch their progress. Shows a count of committed pipelines. |
| History | The record of what's currently committed for this mapping. |
4. The Object Header: KPIs and Commit All Clean
Above the sub-tabs, four cards summarise where the current mapping stands. They're colour-accented — green when healthy, amber when something needs attention.
| KPI | What it shows |
|---|---|
| Columns | How many columns the plan will populate across all the object's target tables. |
| Pipelines | How many committed pipelines exist for this mapping. A badge tells you the state — "Pending commit," "All fresh," or how many are stale or pending. |
| Stale | How many pipelines are out of date because the mapping has changed since they were last built. |
| Decision gates | How many open advisories or validation issues apply. |
Clicking the Stale or Decision gates card opens a drill-down that shows exactly which mappings or issues are behind the number.
Next to the KPIs, Commit all clean is an object-wide shortcut: it commits every draft on the object that is clean — meaning it has no open decision gates. The button shows how many are eligible, and a confirmation lists them by name before committing, so you can commit a whole batch of ready mappings in one action.
IF A NUMBER LOOKS WRONG, CLICK IT — The Stale and Decision-gates cards are clickable. Rather than guessing why the count isn't zero, open the drill-down — it names each contributing mapping and, for gates, the specific issue.
5. Build & Preview
Build & Preview is where a plan is created and shaped. The main action button changes depending on where the plan stands, always offering the one thing you most likely want next.
5.1 Generating the plan
Click Generate ETL (it reads "Re-Generate ETL" once pipelines already exist). Data Gen reads the mapping's column mappings and builds the draft plan. Nothing is committed — you're looking at what it would build.
5.2 Reviewing the plan, table by table
The plan appears as a list of target tables, one expandable row each. Each row shows the table name, a shape badge, and a column count. Expand a row to see:
- The stage flow — the steps the data passes through for that table, shown as a chain (for example extract → transform → confirm).
- A stage summary — counts of extract columns, joins, transform columns, and confirm columns for that table.
This lets you confirm at a glance that every target table is covered and shaped the way you expect, without opening separate pages.
EMPTY TO START — Before you generate anything, the plan area shows "No plan yet. Click Generate ETL to build the first one." That's expected on a mapping you haven't built yet.
6. Checks Before You Commit: Preflight and Decision Gates
Data Gen runs two kinds of check so problems surface before you commit, not after a failed run.
6.1 Preflight schema check
The preflight schema check compares the live structure of your source against what the mapping expects. If the source has drifted — a column renamed, a type changed, something missing — each mismatch is listed, grouped by source table, with a note of how fresh the check is (for example "from connector metadata, 12 min ago"). A Re-check source schema button re-probes the source on demand.
For each mismatch you can:
- Acknowledge the variance — record that you've seen it and why it's acceptable (you supply a reason). Acknowledged variances stop blocking.
- Revoke a previous acknowledgement if circumstances change.
UNRESOLVED SCHEMA ERRORS BLOCK GENERATION — If the preflight check finds errors you haven't acknowledged, Data Gen won't let you commit or generate until you resolve or acknowledge them. This is deliberate — it stops you building a pipeline against a source that no longer matches the mapping. When everything matches, the panel shows a green "all clear" note so you know the check actually ran.
6.2 Decision gates
Decision gates are validations raised while building the plan — some are advisory warnings, some are blocking errors. They appear as banners above the plan (for example a low-confidence classification that's worth a second look). The Decision gates KPI counts the open ones.
Warnings are advisory: you can see them and still commit. Errors block the commit until they're resolved — the Commit button is disabled and its tooltip points you to the Decision gates card to see what's stopping you.
7. Overriding the Generated Plan
Most of the time the automatically generated plan is exactly what you want. When it isn't, Override lets you customise how the plan is built — before committing — without changing the underlying mapping. You work in plain forms; OnCoor turns your edits into the technical change behind the scenes.
7.1 Opening the Override drawer
While a draft is Open, click Override (the button reads Override…, or Edit Override once you've already made some changes). The Override drawer slides in from the side.
If the plan is already Committed, use Override (new draft) instead — it reopens the committed plan as a fresh draft so you can change it and re-commit.
7.2 How the drawer is laid out
The drawer is organised so you always know what you're editing and can see the effect immediately:
- A target chip-row across the top — a mapping usually has several target tables, and you pick which one you're editing here.
- Stage tabs for the selected target — the steps the data flows through (see the table below).
- A form on the left for the stage you're on, and a live preview on the right showing the SQL your edits produce plus a sample of the resulting data.
- A footer with Reset and Save override (and, where an administrator has enabled it, a Show raw JSON escape hatch).
7.3 What each stage tab lets you change
Each target's plan is broken into stages, one tab each. You only need the tabs relevant to your change.
| Stage tab | What you can change |
|---|---|
| Source & Joins | The source table or view the data is pulled from, and any joins to other sources. |
| Extract Filters | The conditions (a WHERE clause) that limit which source rows are pulled in. |
| Column Mappings | For each column: include or exclude it, and edit the source expression that fills it. |
| Transform Map | Reference-table joins, their match conditions, and the intermediate (transient) projection. Only appears for targets that need a transform step — it's hidden when data flows straight from source to the target. |
| Conform Map | The final step into the target table: a WHERE clause, per-column include/exclude, and the expressions feeding the target insert. |
| Operations | Read-only here. Reshaping operations (such as pivot, de-duplicate, merge) are managed in Mapping Manager, not in Data Gen. |
| Job | A read-only summary of the job that will run for this target. |
7.4 Editing a column's value
On the Column Mappings and Conform Map tabs, each target column can be filled from a plain source column, a fixed literal value, or a SQL expression for anything more involved. Small tags on a row show where its value comes from — system (sys), literal (lit), function (ƒ) — and flag a problem (!).
When you write an expression, the built-in editor checks it as you type — showing Valid or a specific error — and lets you insert functions and column names rather than typing them by hand. You can also add a column that isn't in the source: give it a name and a value (a column, a literal, or an expression).
7.5 Seeing the effect before you save
The right-hand side of the drawer keeps you honest:
- Live SQL Preview shows the actual SQL your edits generate, updating as you change the form — so you can confirm the shape of what will run.
- Sample runs a small sample and shows the resulting rows, so you can see real values your expression or filter produces before committing anything.
7.6 Saving, resetting, and the raw-JSON escape hatch
Save override applies your edits to the draft; you then Commit as usual to make them live. Reset discards your changes and returns to the auto-generated plan. If your environment has enabled it, Show raw JSON exposes the underlying change directly — an escape hatch for the rare edit the forms don't yet cover.
7.7 Reviewing overrides after a commit
Once a plan is committed, two affordances remain on Build & Preview:
- Override (new draft) — reopen the committed plan as a fresh draft to make further changes, then re-commit.
- View overrides — open the same drawer read-only, showing exactly what was customised on top of the auto-generated plan for the live commit. Use it to audit a committed pipeline without starting to edit it. (If the commit had no overrides, the button explains that the plan was committed exactly as auto-generated.)
OVERRIDES SIT ON TOP OF THE PLAN — An override customises the generated plan; it never alters the mapping itself. The base plan always comes from the mapping, and your overrides are layered on top and shown separately via View overrides — so it stays clear what was automatic and what you changed.
YOU EDIT FORMS, NOT CODE — Every change is made through friendly forms; OnCoor assembles the technical patch for you. The Live SQL Preview and Sample views are there so you never have to guess what a change will do before you commit it.
8. Committing, Draft Status, and Staleness
Commit is what turns your draft plan into real, runnable pipelines and jobs. Until you commit, nothing exists to run; after you commit, the mapping's pipelines are live and appear on the Run tab.
The action bar and a small status chip always tell you where the draft stands:
| Status | Meaning | The main action offered |
|---|---|---|
| No draft | Nothing generated yet for this mapping. | Generate ETL |
| Open | A draft plan exists and is being worked on. | Commit (with Override, and Recompute / Discard in the menu) |
| Submitting | The commit is in progress. | — (please wait) |
| Committed | The plan is live as pipelines. | Re-Generate ETL, plus Override (new draft), View overrides, History |
| Superseded | Replaced by a newer commit. | Re-Generate ETL |
| Discarded | The draft was thrown away. | Generate ETL |
Discard throws away an open draft without committing it. If you commit a new draft on top of already-committed pipelines, the commit supersedes them — a banner warns you before you do.
Recompute (in the Open draft's menu) rebuilds the draft's plan from the current mapping — use it when the mapping changed while the draft was open, so the draft picks up the latest definition.
Staleness. When the underlying mapping changes after you've committed, the pipelines become Stale — shown as a badge and counted in the KPI. Stale pipelines still run, but they no longer reflect the latest mapping; Re-Generate ETL rebuilds the plan so you can commit a fresh version.
COMMIT IS THE POINT OF NO SURPRISES — Committing materialises the pipelines and jobs, and (where configured) can promote the target tables. The confirmation and the supersede banner exist so a commit never quietly replaces work you meant to keep — read them before confirming.
9. Run: Executing the Pipelines
The Run sub-tab is where committed pipelines are actually executed — this is the step that generates and writes the object's data. It lists the mapping's pipelines in the order they'll run (parents before the children that depend on them).
- Run all jobs — kicks off every pipeline for the mapping. Because a run can take minutes and writes to the configured target server, it asks you to confirm first, telling you how many pipelines will run and against which mapping.
- Initiate (per pipeline) — runs just one pipeline, when you only need to re-run part of the set.
As jobs run, each pipeline shows live progress and a status (running, completed, or failed). While anything is still in flight, the Run-all button is held so you can't accidentally double-fire. Expanding a pipeline shows its canvas — a visual of the extract → transform → confirm flow with recent execution times.
Open-draft notices. If you have an uncommitted draft open, Run reminds you that the committed version is what will run, not your in-progress draft — commit it first to run the new version. Some environments enforce a stricter policy (Require Commit Before Run) that disables Run entirely until the draft is committed or discarded.
RUN WRITES TO THE TARGET SERVER — Running a pipeline executes real jobs against the configured target. That's the whole point — it's how the object's data gets produced — but it means Run is the one action here with a live effect, which is why it always goes through a confirmation.
10. History
The History sub-tab records what's committed for this mapping — one row per live pipeline (target table), each showing when it was committed, a fingerprint of the committed plan, and whether any overrides or warnings applied. It's read-only, and its row count matches the Pipelines KPI.
Two things are worth knowing about how it behaves:
- It reflects the current committed state, not a long history. When a new commit supersedes an earlier one, the previous version is torn down as part of that commit — so History shows what's live now and when each part was last built, rather than a running log of every past commit.
- There's no restore or roll-back. Use History to confirm what's live and when it was built; you can't revert to an earlier version from here. To change what's committed, generate a fresh plan and commit again from Build & Preview.
11. Quick Reference
A fast lookup for the most common actions.
| I want to… | Do this |
|---|---|
| Open Data Gen | Enterprise Object → choose the layer → Data Gen tab |
| Build the ETL for a mapping | Select the mapping → Build & Preview → Generate ETL |
| See what will be built | Expand the target-table rows in the plan |
| Check the source still matches | Read the Preflight schema check panel; Re-check to re-probe |
| Get past a schema mismatch | Acknowledge the variance (with a reason), or fix the source |
| See what's blocking a commit | Click the Decision gates KPI card |
| Customise how a column is filled | Override while the draft is open |
| See what was customised on a live commit | View overrides |
| Make the pipelines live | Commit |
| Throw away a draft | Discard |
| Commit every ready mapping at once | Commit all clean (object header) |
| Refresh after the mapping changed | Re-Generate ETL (clears the Stale flag once re-committed) |
| Produce the data | Run → Run all jobs (or Initiate one pipeline) |
| Re-run a single pipeline | Run → Initiate on that pipeline |
| See what's committed and when | History |
Source: OnCoor Data Gen product documentation, written for end users. For the latest screens and options, always refer to the in-app interface.