# Where every number comes from — Advaitsi CoalMind

> This explainer backs the GM dashboard (`dashboard.html`). It is intentionally **not** on the dashboard itself — a GM does not need the plumbing, only the decision. It lives here for anyone auditing or extending the system.
>
> **All data referenced is synthetic** — a fictional demo mine, not SECL operations.

Every figure on the dashboard traces back to a real source, through one deterministic pipeline. Two buckets of sources, a strict time model so we never use information from the future, then a common event store.

---

## The two buckets of sources

### A · Systems of record — *high trust*
Machine-recorded, hard to fake, but they rarely say *why*.

| Source | What it gives | File |
|---|---|---|
| **SAP** | Daily & month-to-date coal + OB (no loss reasons; posting lag) | `synthetic_mine_data/04_sap_production.csv` |
| **Coalnet — road** | RFID weighbridge: trucks, net tonnes, grade, destination | `synthetic_mine_data/05_coalnet_road_weighbridge.csv` |
| **Coalnet — rail** | Rake placement/loading, wagons, delay, demurrage | `synthetic_mine_data/06_coalnet_rail_loading.csv` |
| **Stock yard** | Closing stock, dispatched, road/rail split | `synthetic_mine_data/07_stock_yard_daily.csv` |
| **Weather** | Rainfall, stoppage hours, rain-per-shift | `synthetic_mine_data/08_weather_daily.csv` |
| **Breakdown register** | Which unit failed, when, fault, fix | `synthetic_mine_data/09_breakdown_maintenance_log.csv` |
| **Plan** | Daily & shift targets, OB target | `synthetic_mine_data/02_production_plan.csv` |

### B · Human communication — *needs cross-check*
Rich on timing and cause, but messy and incomplete.

| Source | What it gives | File |
|---|---|---|
| **WhatsApp** | Live timestamps: breakdown/rain/rake start-stop (Hinglish, name variants EX3→EX-03) | `synthetic_mine_data/whatsapp/` |
| **Voice notes** | Speech-to-text with confidence scores | `synthetic_mine_data/whatsapp/voice_note_transcripts.csv` |
| **Shift reports** | Handwritten → OCR; 35–50% of reasons omitted | `synthetic_mine_data/03_shift_reports_ocr.csv` |
| **Meetings** | Evening (sets tomorrow) + morning (reviews yesterday); decisions & owners | `synthetic_mine_data/meetings/` |
| **Phone-call notes** | Verbal escalations — sparse | `synthetic_mine_data/12_phone_call_notes.csv` |
| **Incident history** | Prior similar events & observed recurrence | `synthetic_mine_data/10_incident_history.csv` |
| **Tacit knowledge** | Rules of thumb — marked *unverified* | `synthetic_mine_data/11_tacit_knowledge_notes.csv` |

> **Authority rule:** no single source is universally authoritative. SAP establishes totals, Coalnet establishes movement, WhatsApp establishes timing, shift reports establish explanations, meetings establish decisions. When sources disagree, the disagreement is **preserved**, not silently resolved.

---

## Three timestamps on every claim — so we never leak the future

| Timestamp | Meaning |
|---|---|
| **① source time** | When the thing actually happened (e.g. hose burst 15:10). |
| **② available-at** | When it *could first be known* — SAP posting lag, OCR time, message delivery. |
| **③ ingested-at** | When the pipeline read it. |

**As-of masking:** the dashboard is built "as of 19 Sep 10:30". Anything whose **available-at** is later is hidden — so a breakdown register row that only appears *after* the repair, or a figure spoken in a later meeting, cannot sneak into an earlier decision. That is why the morning-meeting figure is counted but a 09:57 breakdown is not.

---

## From messy source to one answer

1. **Ingest & normalise** — read chats, OCR slips, transcribe voice, fix equipment names, align every record to a *date + shift*. Join keys: `date + shift`, `equipment ID`, `location / face`, `rake / vehicle`.
2. **Canonical claims → events** — one row per thing that happened (breakdown, rain, delay, decision), each carrying its evidence link, confidence, and the three timestamps.
3. **Reconcile & attribute** — group sources by *lineage* → median headline; divide overlapping loss on the clock; learn tonnes/hour only from *earlier* shifts; keep an explicit residual.
4. **Brief & recommend** — plan vs actual, why, what we did last time — ranked, cited, advisory. Missing data is flagged with an owner, never invented.

---

## What each KPI stands on today

| KPI | Status | Source → owner |
|---|---|---|
| Coal produced / shift split | ✅ covered | SAP + shift reports + stock → MIS / Shift In-charge |
| Dispatch road / rail / stock | ✅ covered | Coalnet + stock yard → Weighbridge clerk / Siding in-charge |
| Loss causes (rain, breakdown) | ✅ covered (statistical) | Weather + register + WhatsApp → Shift In-charge |
| OB / stripping ratio | ⚠ partial — units unconfirmed | SAP OB → **Surveyor** |
| HEMM availability / idle | ⛔ missing | — → **Mechanical Foreman** |
| Coal quality (GCV / ash / moisture) | ⛔ missing (only declared grade) | — → **Lab / Quality In-charge** |
| Cost / OMS | ⛔ missing | — → **Finance / Cost cell** |
| Safety (DGMS / LTI) | ⛔ missing | — → **Safety Officer** |

Version 1 closes the daily plan-vs-actual loop end to end. Machine-hours, quality, cost and safety are the next layers — deliberately added after the first loop is trusted.

> **Ground-truth files are evaluation-only.** `synthetic_mine_data/_ground_truth_synthetic_only/` and any `TRUTH_*` files are never linked from the dashboard and never enter the inference pipeline — they exist only to score the system in a separate evaluation layer.

---

## At the real mine
Keep the column names and the pipeline; replace each synthetic file with the real, permissioned equivalent. The first job on site is to confirm which messy patterns are real (handwriting quality, WhatsApp language mix, SAP/Coalnet field names) and to obtain a limited, written-permissioned extract per source for one mine.
