Add documentation for MilliSecondsBehindSource growing exponentially. #700
Conversation
Add documentation for MilliSecondsBehindSource growing exponentially.
| @@ -351,3 +351,21 @@ To solve the issue the configuration option `producer.max.request.size` must be | |||
| If the global change is not desirable then the connector can override the default setting using configuration option `producer.override.max.request.size` set to a larger value. | |||
|
|
|||
| In the latter case it is also necessary to configure `connector.client.config.override.policy=ALL` option in Kafka Connect worker config file `connect-distributed.properties`. For Debezium `connect` Docker image the environment variable `CONNECT_CONNECTOR_CLIENT_CONFIG_OVERRIDE_POLICY` can be used to configure the option. | |||
|
|
|||
| == Why MilliSecondsBehindSource is growing exponentially ? | |||
There was a problem hiding this comment.
Thanks for this change, this sounds a bit concerning though. Can you provide some more context about this situation: what's your change event rate and your set-up in general? I don't quite agree to the statement on roundtrip times below; the connection throughput may be a limiting factor, but latency itself should be constant, also with larger roundtrip times between the DB and the Kafka Connect host.
There was a problem hiding this comment.
Hi @gunnarmorling ,
Thanks for your comments.
MilliSecondsBehindSource growing exponentially issue occurred in our environment and I had reached out to Jiri Pechanec about this issue. Link to Gitter Thread with details - https://gitter.im/debezium/user?at=6050ec1ed1aee44e2def7b89
Created this pull request as Jiri had recommended contributing to the documentation.
Please let me know if you need more details.
There was a problem hiding this comment.
Hi @rnatarajan so I definitely think adding an FAQ entry, particularly when MilliSecondsBehindSource is high is worthwhile given the scenario you describe, but I think perhaps the way this is conveyed, particularly with the use of "exponentially" gives a bad impression.
Perhaps, this could be reworded to something like "High MilliSecondsBehindSource when connector deployed with WAN connection to database" or similar. I think speaking in terms of higher than usual latency when using a WAN connection rather than a LAN connection is what we should strive for on this PR.
There was a problem hiding this comment.
@Naros Thanks for the comment. I can reword the README.
There was a problem hiding this comment.
Awesome, once you have, if you could just request a re-review of the PR that'd be wonderful.
No description provided.