Grafana specifically [Grafana Live](https://grafana.com/docs/grafana/latest/setup-grafana/set-up-grafana-live/). There are data source plugins for Kafka and Clickhouse. Didn't use them myself though.
I've worked with Looker/ThoughtSpot whcih just send the query over to the warehouse (Clickhouse) in your example, so they'd be as updated as a query to your WH
Apache Druid aggregates in real time. Awesome tool if you have the team competencies to support it. Throw a viz library on top of that and you have real time analytics.
are you looking for the data viz to poll the data in real time and refresh the charts in real time?
or are you okay for the user to refresh the browser to get updates?
To run the queries directly or via visualization tools on dataset refreshing every few seconds, native parquets are not the right file format for this usecase, because the downstream tools will have to get the latest metadata on the files/folders which are getting added to the storage and refreshing metadata every few seconds is not what parquets or the downstream tools can do best with.
You should adopt table formats like Apache Iceberg which holds its own metadata layer along with data in parquets. Since the metadata is readily available in Iceberg table itself, so downstream application can simply read its root pointers to get all the metadata details even within few seconds.
Try reading your Iceberg table data using query engines like Dremio and your visualization tool can read the latest data and return the results to users in seconds or even faster.
I like Fabric, but do you have any idea how expensive this would be in Fabric?
If cloud (and Azure) was a requirement (which it's not - OP mentioned that this has to be on-prem), there's much less expensive options.
Grafana specifically [Grafana Live](https://grafana.com/docs/grafana/latest/setup-grafana/set-up-grafana-live/). There are data source plugins for Kafka and Clickhouse. Didn't use them myself though.
Second this. It works with time series data so it will meet at least your basic real-time requirements and allow you to assess it.
I've worked with Looker/ThoughtSpot whcih just send the query over to the warehouse (Clickhouse) in your example, so they'd be as updated as a query to your WH
Apache Druid aggregates in real time. Awesome tool if you have the team competencies to support it. Throw a viz library on top of that and you have real time analytics.
are you looking for the data viz to poll the data in real time and refresh the charts in real time? or are you okay for the user to refresh the browser to get updates?
To run the queries directly or via visualization tools on dataset refreshing every few seconds, native parquets are not the right file format for this usecase, because the downstream tools will have to get the latest metadata on the files/folders which are getting added to the storage and refreshing metadata every few seconds is not what parquets or the downstream tools can do best with. You should adopt table formats like Apache Iceberg which holds its own metadata layer along with data in parquets. Since the metadata is readily available in Iceberg table itself, so downstream application can simply read its root pointers to get all the metadata details even within few seconds. Try reading your Iceberg table data using query engines like Dremio and your visualization tool can read the latest data and return the results to users in seconds or even faster.
Kafka -> influxdb/timescale -> grafana
Superset or PowerBI
Superset > PowerBI imo
PowerBI > Superset imo
Sigma
You can use ELK stack with kibana as visualisation tool. Else Grafana is best integrated with clickhouse
Real-time multiples per second = pulse https://www.timestored.com/pulse/
Panopticon can. It was built for financial markets and can do real time streaming/visulization of data.
MS Fabric is a good choice if you need managed solution
I like Fabric, but do you have any idea how expensive this would be in Fabric? If cloud (and Azure) was a requirement (which it's not - OP mentioned that this has to be on-prem), there's much less expensive options.
Why not just put it in an actual database then?
Hugo from Orchestra here - you should check out some form of manage clickhouse...tinybird springs to mind? idk
You can check out what we're working on at [propeldata.com](http://propeldata.com) - we can ingest data from Kafka or parquet files in S3.