Cleansing Operational Data into PostgreSQL (Cleansing Operational Data Into Postgresql)

Cleansing Operational Data into PostgreSQL

ETL validates plates, timestamps, and foreign keys before analytics tables accept a row.

MySQL rows → validate → typed PG tables

Temizlik yoksa PostGIS sorusu yalandır.

Concepts, defined where they first appear

📦 Operational Store
MySQL written by the ERP — right for daily ops, dirty for analytics.

📦 Analytics Store
Cleansed PostgreSQL/PostGIS for geo queries and reporting.

📦 Telemetry Store
Document store (MongoDB) for high-frequency location events.

📦 Search Index
Elasticsearch cluster for cross-entity ops search.

Blurring these layers produces the wrong truth in analytics and on the map UI.

Shape of the problem

ETL validates plates, timestamps, and foreign keys before analytics tables accept a row.

MySQL rows → validate → typed PG tables

The split that works

Temizlik yoksa PostGIS sorusu yalandır.

MySQL rows → validate → typed PG tables
        ↓
   explicit contract → frontend

Frontend bridge

Ops maps, wait-geofence UI, and MFEs consume these read models — they never bind to raw ERP/MySQL.

The mappings that get confused most often

❌ ERP MySQL = analytics database
✓ ERP is a write store; analytics lives in PostGIS

❌ Let the map trust public/vendor routers blindly
✓ Production needs your API/OSRM edge

❌ Log ML outputs with PII
✓ Evaluation and serving obey privacy boundaries

A checklist for auditing your own system

1. Which store is source of truth for this surface?
2. Which API contract does the frontend call?
3. What happens if one cluster node dies?
4. How is lag shown to operators?
5. Deny-list: any client domain/vendor leakage?

What to take away from this part

1. Polyglot stores exist for different workloads; one-DB fantasies are fragile.
2. Frontends consume clean read models, not the ERP.
3. ML/OSRM outputs need an API contract before they hit the map.

A beautiful map drawn from a dirty source is still a lie.