โ† All projects

Data engineering ยท Python and SQLite

Build a reproducible contract ledger

Turn aggregate business inputs into a validated dataset that reconciles with the dashboard.

15synthetic learner IDs
45synthetic contract rows
2renewal scenarios

From inputs to reporting

  1. Read the versioned assumptions and validate integer counts, prices and renewal scenarios.
  2. Generate learner and contract tables in SQLite with foreign keys, integer cents and a unique scenario, learner and package key.
  3. Reconcile SQL totals against independent arithmetic, check database integrity, then atomically replace the database file.
  4. Publish a synthetic contract CSV and KPI JSON. Each row carries a synthetic provenance label.

The 45 rows combine two alternative versions of the same cohort. Use one scenario at a time when reporting EWF totals.

Quality and repeatability

The regression test verifies both scenario totals, a repeat build, duplicate rejection and preservation of the prior database when inputs are invalid.

python -m unittest -v test_pipeline
python pipeline.py ewf_inputs.json output
python -m pip install -r requirements.txt
python capacity.py ewf_inputs.json output/capacity_results.json
python train_router.py intent_dataset.json output

Production data model

At production scale, join learner and package IDs to separate booking and payment tables. Keep personal data in restricted systems, reconcile payments and refunds independently, and publish only aggregated reporting outputs.