Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DBZ-8517: Bump Kinesis version to 2.17.241 to fix Jackson incompatibility when using the sink Kinesis #141

Merged
merged 1 commit into from
Dec 18, 2024

Conversation

Sgitario
Copy link
Contributor

@Sgitario Sgitario commented Dec 12, 2024

Bumping the Kinesis version to 2.17.241 will resolve the jackson incompatibility issue when running the Debezium server:

2024-12-11 11:25:23,166 ERROR [io.deb.ser.ConnectorLifecycle] (pool-7-thread-1) Connector completed: success = 'false', message = 'java.lang.NoSuchFieldError: Class com.fasterxml.jackson.databind.PropertyNamingStrategy does not have member field 'com.fasterxml.jackson.databind.PropertyNamingStrategy PASCAL_CASE_TO_CAMEL_CASE'', error = 'java.lang.NoSuchFieldError: Class com.fasterxml.jackson.databind.PropertyNamingStrategy does not have member field 'com.fasterxml.jackson.databind.PropertyNamingStrategy PASCAL_CASE_TO_CAMEL_CASE'': java.lang.NoSuchFieldError: Class com.fasterxml.jackson.databind.PropertyNamingStrategy does not have member field 'com.fasterxml.jackson.databind.PropertyNamingStrategy PASCAL_CASE_TO_CAMEL_CASE'
	at software.amazon.awssdk.regions.internal.util.EC2MetadataUtils.<clinit>(EC2MetadataUtils.java:95)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getToken(InstanceProfileCredentialsProvider.java:83)
	at software.amazon.awssdk.auth.credentials.InstanceProfileCredentialsProvider.getCredentialsEndpointProvider(InstanceProfileCredentialsProvider.java:69)
	at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.refreshCredentials(HttpCredentialsProvider.java:74)
	at software.amazon.awssdk.utils.cache.CachedSupplier.refreshCache(CachedSupplier.java:132)
	at software.amazon.awssdk.utils.cache.CachedSupplier.get(CachedSupplier.java:89)
	at java.base/java.util.Optional.map(Optional.java:260)
	at software.amazon.awssdk.auth.credentials.HttpCredentialsProvider.resolveCredentials(HttpCredentialsProvider.java:146)
	at software.amazon.awssdk.auth.credentials.AwsCredentialsProviderChain.resolveCredentials(AwsCredentialsProviderChain.java:91)
	at software.amazon.awssdk.auth.credentials.internal.LazyAwsCredentialsProvider.resolveCredentials(LazyAwsCredentialsProvider.java:45)
	at software.amazon.awssdk.auth.credentials.DefaultCredentialsProvider.resolveCredentials(DefaultCredentialsProvider.java:104)

How to reproduce

To reproduce this issue, follow the next steps:
1.- Build the Debezium server distribution: ./mvnw clean package -DskipITs -DskipTests -Passembly
2.- Start Up postgres listening at localhost:5432, use the following docker-compose.yml as a reference: https://github.com/debezium/debezium-server/pull/140/files#diff-56a998a8165ad21afa94ccb494c4b08f843d129dfd9e897780e5d766f59c1aaeR103
3.- Copy the application properties of the distro folder to the Debezium target folder:

mkdir debezium-server-dist/target/config
cp debezium-server-dist/src/main/resources/distro/config/application.properties.example debezium-server-dist/target/config/application.properties

The relevant part of this application properties file is:

debezium.sink.type=kinesis
debezium.sink.kinesis.region=eu-central-1

4.- Then, when running the Debezium server:

cd debezium-server-dist/target
chmod +x /classes/distro/run.sh
/classes/distro/run.sh

In main, the run.sh fails with the above exception.

Fixes https://issues.redhat.com/browse/DBZ-8517

@@ -12,7 +12,7 @@
<packaging>jar</packaging>

<properties>
<version.kinesis>2.13.13</version.kinesis>
<version.kinesis>2.17.241</version.kinesis>
<version.sqs>2.13.13</version.sqs>
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that the version 2.17.241 is a very old version as well, but it's the one used in Debezium core.
Moreover, the version.sqs should use the same version as the version.kinesis to avoid incompatibilities.
But we would like to get some feedback from you about how to verify these changes and cover both kinesis and sqs (ideally, using a local machine).

Copy link
Contributor

@jpechane jpechane Dec 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the sink was developed we tested it against rela Kinesis. Today it might be possible to use localstack to give it a try. I can't confirm it will work but it would be good to check.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using localstack works fine, I can rebase this PR with #140, so we can add an additional sink job to validate this change and also cover the Kinesis sink, wdyt?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sgitario Sounds good! Could you please create a Jira and attribute commits in this PR to it using DBZ-xxx prefix in commit message. We'll merge the PR then and you can add the localstackt to the server validation PR.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Sgitario Sgitario changed the title Bump Kinesis version to 2.17.241 Bump Kinesis version to 2.17.241 to fix Jackson incompatibility when using the sink Kinesis Dec 12, 2024
@Sgitario Sgitario force-pushed the fix_jackson_issue_with_kinesis branch from 96284dc to b200b85 Compare December 16, 2024 13:22
@Sgitario Sgitario changed the title Bump Kinesis version to 2.17.241 to fix Jackson incompatibility when using the sink Kinesis DBZ-8517: Bump Kinesis version to 2.17.241 to fix Jackson incompatibility when using the sink Kinesis Dec 16, 2024
@Sgitario Sgitario requested a review from jpechane December 16, 2024 13:23
@jpechane jpechane merged commit ba91cea into debezium:main Dec 18, 2024
2 checks passed
@jpechane
Copy link
Contributor

@Sgitario Applied, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants