Log reader stops consumption of log records without an error

[vveloso]

What did you do?

Wrote one synchronous writer process and one synchronous reader process according to the tutorials for using the core library. The writer produces small messages (<100 bytes) continuously with at least a 1ms delay between them, using org.apache.distributedlog.api.LogWriter#write. The reader polls for messages continuously with at least a 1ms delay between polls, using org.apache.distributedlog.api.LogReader#readNext.

One bookkeeper instance is running as well as one zookeeper instance, both on the same machine. Ensemble, ack quorum and write quorum sizes are therefore 1. Immediate flush is enabled. Log rolling by time is disabled (set to -1) and rolling by size is enabled by setting it to 100MB. All configurations were done programmatically using org.apache.distributedlog.DistributedLogConfiguration.

Using distributedlog-core Maven dependency version 0.5.0.

What did you expect to see?

A continuous stream of messages being read and counted by the reader process, indefinitely.

What did you see instead?

The reader process will eventually stop reading messages without any error reported to client code.

The writer process continues to write messages to the log, without any exceptions.

In the run that produced the information attached, LogReader#readNext always returns null after timing out on org.apache.distributedlog.ReadAheadEntryReader#getNextReadAheadEntry.

In one other run, org.apache.distributedlog.BKSyncLogReader#readNextEntry was stuck on an infinite loop calling ReadAheadEntryReader#getNextReadAheadEntry. Unfortunately there are no logs available for this run.

distributedlog-consumer-stoppage-heap-dump.zip
distributedlog-consumer-stoppage-thread-dumps.zip
distributedlog-consumer-stoppage-keeper-logs.zip
distributedlog-consumer-stoppage-consumer-logs.zip

distributedlog-consumer-stoppage-code-extract.zip

[sijie]

@vveloso thank you for reporting this. I will take a look today.

[hugomiguelabreu]

@sijie How is this issue? Does it have a source?