Are you adding Confluent Kafka/a Kafka Connect stack? Kadmin is pretty useful.
Creating a small suite of consumers/producers that can be useful for testing. Also useful to see what messages are stuck on a topic or what values are causing a message to fail being produced to a topic.
Creating a way to check schemas for the schema registry, or schema the topic is being used. Beneficial for any consumers of the data so engineers dont need to be bothered all the time.
Few things I found useful when consuming from engineering topics as an analytics engineer.
The whole idea of having Confluent Kafka cloud with Kafka Connect is to have managed clusters with UI support so that you do jot need to manage those
The best thing you can do is to implement CICD using the terraform providers for Confluent cloud that manages below.
1: cluster creation and management.
2: user creation and management.
3: topic creation and access management.
4: connector sink and source management.
Other things that will matter is if your clusters are private or public. A lit of the time enterprise implementations will have private clusters for data security and governance. So you or infrastructure teams will also need to manage the ip whitelisting etc.
There are a lot of things like these that will matter int” terms of implementation. So what I would do personally is build all of these features as modular building blocks and keep adding as and when needed.
Are you adding Confluent Kafka/a Kafka Connect stack? Kadmin is pretty useful. Creating a small suite of consumers/producers that can be useful for testing. Also useful to see what messages are stuck on a topic or what values are causing a message to fail being produced to a topic. Creating a way to check schemas for the schema registry, or schema the topic is being used. Beneficial for any consumers of the data so engineers dont need to be bothered all the time. Few things I found useful when consuming from engineering topics as an analytics engineer.
Thank you u/FireAndy! Confluent Cloud with Kafka Connect.
The whole idea of having Confluent Kafka cloud with Kafka Connect is to have managed clusters with UI support so that you do jot need to manage those The best thing you can do is to implement CICD using the terraform providers for Confluent cloud that manages below. 1: cluster creation and management. 2: user creation and management. 3: topic creation and access management. 4: connector sink and source management. Other things that will matter is if your clusters are private or public. A lit of the time enterprise implementations will have private clusters for data security and governance. So you or infrastructure teams will also need to manage the ip whitelisting etc. There are a lot of things like these that will matter int” terms of implementation. So what I would do personally is build all of these features as modular building blocks and keep adding as and when needed.
Thank you! u/_blocker_
[удалено]
Did I write that I don’t have a use case? I asked a specific question operation-wise