Skip to content

Add Memq Metadata Client#134

Closed
ArtemTetenkin wants to merge 13 commits into4.0from
memq_metadata_client
Closed

Add Memq Metadata Client#134
ArtemTetenkin wants to merge 13 commits into4.0from
memq_metadata_client

Conversation

@ArtemTetenkin
Copy link
Copy Markdown
Contributor

@ArtemTetenkin ArtemTetenkin commented Mar 10, 2026

Add Memq Metadata Client
Here's a summary of all the changes:

Files created
psc/src/main/java/com/pinterest/psc/metadata/client/memq/PscMemqMetadataClient.java - The main Memq metadata client implementation extending PscBackendMetadataClient. Since Memq doesn't have a dedicated admin client (unlike Kafka's AdminClient), it uses short-lived MemqConsumer instances for all metadata queries. The implementation:

listTopicRns() - throws UnsupportedOperationException since Memq doesn't support listing all topics
describeTopicUris() - creates a consumer per topic, subscribes, and calls getPartition() to discover partitions
listOffsets() - groups partitions by topic, uses getEarliestOffsets() / getLatestOffsets() based on the requested option
listOffsetsForTimestamps() - groups by topic and uses offsetsOfTimestamps()
listOffsetsForConsumerGroup() - creates a consumer with the target consumer group ID and uses committed() per partition
psc/src/main/java/com/pinterest/psc/metadata/creation/PscMemqMetadataClientCreator.java - The plugin creator class annotated with @PscMetadataClientCreatorPlugin(backend = "memq", priority = 1), following the exact same pattern as PscKafkaMetadataClientCreator. It creates and initializes PscMemqMetadataClient instances and validates Memq topic URIs via MemqTopicUri.validate().

psc/src/main/java/com/pinterest/psc/config/PscMetadataClientToMemqConsumerConfigConverter.java - Config converter that extends PscMetadataClientToBackendMetatadataClientConfigCoverter, mapping PSC metadata client configuration to Memq consumer configs (e.g., PSC_METADATA_CLIENT_ID to ConsumerConfigs.CLIENT_ID).

File modified
psc/src/main/java/com/pinterest/psc/consumer/memq/MemqTopicUri.java - Changed the constructor from package-private to public so it can be used from the com.pinterest.psc.metadata.client.memq package (consistent with how KafkaTopicUri has a public constructor).

@ArtemTetenkin ArtemTetenkin requested a review from a team as a code owner March 10, 2026 18:52
@jeffxiang jeffxiang closed this Apr 15, 2026
@jeffxiang jeffxiang deleted the memq_metadata_client branch April 15, 2026 22:09
@jeffxiang jeffxiang restored the memq_metadata_client branch April 15, 2026 22:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants