Seeing multiple operations standardize on S3 with Parquet/Delta for FMS, telemetry, and geology — this quarter we streamed Wenco and PI tags via Kafka and landed 12 TB in Databricks, taking cycle-time reporting from hours to minutes… The wins are clear; the pain is OT tag harmonization and time alignment across vendors, so I’m looking for what’s working for schema governance in brownfield sites (GMG guidance, OPC UA namespaces, or something else?).
We did similar: landed Wenco + PI into Delta on S3, and a tiny canonical tag dictionary in Glue with versioned IDs and a “one tag, one unit” rule kept harmonization sane. Confluent Schema Registry at the Kafka edge plus dbt tests in DLT caught most vendor drift, and a 1‑second watermark for time alignment stopped the zig‑zag joins. Caveat: the initial mapping workshop with ops took a week, but it beat whack‑a‑mole at 2 a.m.
On the harmonization pain: pushing contracts to the edge worked — Kafka topics validated by Schema Registry (Avro) with explicit unit and sampling_rate fields, then time alignment in Delta Live Tables using 1 s hops + watermarks… A tiny SCD table keyed by PI WebId/Wenco unit_id caught tag renames, and it kept our ‘hours to minutes’ flow at about 6 min end-to-end. If you don’t want Confluent, Glue Schema Registry + LF tags is fine, but Schema Registry caught more drift: Schema Registry for Confluent Platform | Confluent Documentation.