Has anyone migrated OSIsoft PI plus fleet telemetry into a cloud lakehouse without drowning in tag sprawl? We’ve got 48 haul trucks on Cat MineStar and about 120k PI points across two sites; curious if OPC UA to MQTT to Kafka with a schema registry has been reliable in the pit, especially with shift-change outages and flaky WiFi.
But we got OPC UA→MQTT→Kafka stable by running an edge MQTT broker with persistent disk and Sparkplug B, then bridging to Kafka with schema-reg’d Protobuf; that handled “shift-change outages” and flaky WiFi because QoS1/2 plus birth/death filled the gaps. The thing that cut tag sprawl was a Git-backed tag dictionary and enforcing Sparkplug metric namespaces before topics hit Kafka; if you can’t do Sparkplug, a tiny edge Kafka/Redpanda as store-and-forward also works. Are you already on Sparkplug B or publishing raw node-ids?
We kept the lake from turning into a swamp by anchoring everything to PI AF templates and mapping each template to a single Avro schema (asset_id, signal, ts, value, quality, unit), then rejecting any tags not in AF. At the pit edge we do a 5–10s tumbling window with dedupe/last-value and a small disk-backed buffer so “shift-change outages” only delay, not duplicate, records. Are you keeping AF as the source of truth for the schema registry, or letting Kafka discover and then backfilling AF?
Use OPC UA ‘Deadband=Percent’ (1–2%) to cut 120k noise; MQTT QoS 1 + Kafka idempotent stabilized 48 trucks…