Capstone: duplicate supplier assistant · Indu Learning Platform

What you are building

A small tool that takes a supplier extract, finds records that are probably the same company, explains why, and hands the decision to a person. The last layer, turning approved merges into governed MDG changes, is what separates you from a data scientist. Keep scope small: one entity, one or two countries, a few thousand rows of synthetic data. Never export client data, and say so in your README.

Each layer consumes what the layer below produced. The top one is what interviewers remember.

Architecture

Plain Python scripts, each writing a file, so you can rerun one stage and show intermediate outputs in the demo. Nothing lets the model merge anything: the model suggests, the steward decides, MDG governs.

One file handed from each stage to the next.

Build it step by step

Click through the steps, then study the matching skeleton. Scoring combines weak signals: an equal tax number is strong evidence; an equal bank account is a flag (shared service centres reuse accounts); name and street similarity are useful but noisy. Start with simple weights, tune them against steward decisions, and record the weights version in every output.

Step through · 1 of 7

Measure fill rate, distinct values and pattern violations per column. Write the findings down before changing anything; this is your 'as-is' slide.

Seven steps from raw extract to governed merge requests.

python

# match.py (skeleton)
import pandas as pd
from rapidfuzz import fuzz, process
df = pd.read_parquet("out/suppliers_clean.parquet")
df["block_key"] = df["country"] + "|" + df["postal"].fillna("").str[:3]
W_NAME, W_STREET, TAX_BONUS, KEEP_ABOVE = 0.5, 0.3, 30, 70   # weights v1
rows = []
for key, g in df.groupby("block_key"):
    g = g.reset_index(drop=True)
    names = process.cdist(g["name_norm"], g["name_norm"], scorer=fuzz.token_sort_ratio)
    streets = process.cdist(g["street_norm"], g["street_norm"], scorer=fuzz.ratio)
    for i in range(len(g)):
        for j in range(i + 1, len(g)):
            a, b = g.iloc[i], g.iloc[j]
            tax_eq = int(bool(a.tax_id) and a.tax_id == b.tax_id)
            combined = W_NAME * names[i][j] + W_STREET * streets[i][j] + TAX_BONUS * tax_eq
            if combined >= KEEP_ABOVE:
                rows.append({"id_a": a.src_id, "id_b": b.src_id, "block_key": key, "combined": round(combined, 1)})
pd.DataFrame(rows).sort_values("combined", ascending=False).to_parquet("out/candidate_pairs.parquet")

process.cdist scores every pair in a block at once; fine for blocks of a few hundred rows.

Try it · Put in order

Put the capstone pipeline stages in order.

1Profile the raw supplier CSV
2Normalize names and addresses into *_norm columns
3Ask the LLM to explain each kept pair as JSON
4Ingest decisions and write MDG merge requests
5Score pairs inside each block with rapidfuzz
6Steward decides in the review sheet
7Build blocks (country + postal prefix)

The LLM call

Short prompt, both records as JSON, JSON back. If the answer does not parse or the verdict is not one of three allowed values, store UNSURE. Log model and prompt version. The SDK is an example; swap in SAP's generative AI hub or another provider, and fill the model name from current docs.

python

# explain.py (skeleton)
import json; from anthropic import Anthropic   # or your provider's SDK / SAP generative AI hub client
MODEL, PROMPT_VERSION = "YOUR_MODEL_ID", "explain-v1"   # check provider docs for model names
ALLOWED = {"LIKELY_SAME", "UNSURE", "LIKELY_DIFFERENT"}
client = Anthropic()               # reads the API key from an environment variable
SYSTEM = ("You help a master data steward review possible duplicate suppliers. You never decide. "
          'Reply with JSON only: {"verdict": "...", "reason": "<= 2 sentences", "conflicts": []}')
def explain(rec_a: dict, rec_b: dict, scores: dict) -> dict:
    user = json.dumps({"record_a": rec_a, "record_b": rec_b, "scores": scores})
    resp = client.messages.create(model=MODEL, max_tokens=300, system=SYSTEM,
                                  messages=[{"role": "user", "content": user}])
    try:
        out = json.loads(resp.content[0].text)
        if out.get("verdict") not in ALLOWED: raise ValueError("bad verdict")
    except (ValueError, KeyError, IndexError):
        out = {"verdict": "UNSURE", "reason": "Model output could not be parsed.", "conflicts": []}
    return {**out, "model": MODEL, "prompt_version": PROMPT_VERSION}

Send only the fields needed. Hash or drop bank details before they leave your machine.

Try it · Sort

How should each piece of evidence be treated?

Strong evidence

Flag for the steward

Useful but noisy

Click an item, then a bucket (or drag it).

The review sheet

Stewards work in Excel. One row per pair, both records, the evidence, the model's reason and a decision dropdown with three fixed values. When MERGE is chosen, the steward also picks the survivor; ingest.py rejects MERGE rows without one and writes an audit log with weights and prompt version.

Synthetic rows. A fixed decision vocabulary lets you measure acceptance rates later.

sql

-- candidate_pairs: one row per pair (SQLite or parquet; the columns matter)
CREATE TABLE candidate_pairs (
  pair_id      TEXT PRIMARY KEY,  -- "ECC_EU:100234|ECC_US:774501"
  id_a         TEXT NOT NULL,
  id_b         TEXT NOT NULL,
  block_key    TEXT NOT NULL,     -- "DE|803"
  name_score   REAL,              -- 0-100, token_sort_ratio
  street_score REAL,
  tax_equal    INTEGER,           -- 1 / 0 / NULL if either side empty
  combined     REAL,              -- weighted ranking score
  llm_verdict  TEXT,              -- LIKELY_SAME / UNSURE / LIKELY_DIFFERENT
  llm_reason   TEXT,
  decision     TEXT,              -- MERGE / NOT_DUPLICATE / NEEDS_INFO
  survivor_id  TEXT, decided_by TEXT, decided_at TEXT
);

From approved merge to MDG change request

You probably have no MDG system for the demo, so you design this part. For an ongoing process, each approved pair becomes a change request: update the survivor, block the duplicate, run the rule-based workflow (Lesson 1-3). A small loader can create CRs via the MDG governance API (Lesson 2-3). After activation, DRF replicates and key mapping links receiver IDs to the survivor (Lesson 1-4).

Central governance: one CR per approved pair.

Consolidation or central governance?

MDG consolidation and mass processing (Lesson 3-3) suits a one-time clean-up: load, match, best record, activate. Either way the merge is visible, approved and auditable; that is your closing line.

Many programmes use both: consolidation for the backlog, CRs for what appears later.

Try it · What would you do?

An interviewer asks: 'So your tool creates change requests in MDG?' You ran Python on synthetic data and designed the MDG part. What do you say?

Four-week plan and demo

About eight to ten hours a week. The demo is five minutes: problem and as-is profile, a live run of matching and explanation, the review sheet, one slide on the MDG design. Quote three honest, labelled-synthetic numbers: candidate pairs per thousand records, steward agreement with the model, and review time per pair with and without the explanation.

Small enough for evenings, complete enough to discuss for an hour.

Remember

A working demo with a governance design beats a list of tools on a CV.
Block before you score; combine weak signals into one explainable number.
The LLM explains, the steward decides, MDG governs and replicates.

Check yourself

What makes this capstone an MDG project rather than only a data science exercise?

Why do you build blocks before scoring pairs?

The model returns text that is not valid JSON. What should explain.py do?

For an ongoing process where new duplicates appear each month, which MDG path fits best?