Turn repeater DDS Router Segmentaion Fault [17325] #353

kim3097 · 2023-02-13T07:21:59Z

kim3097
Feb 13, 2023

Hello everybody, I'm trying to connect three machines running ROS2 nodes with this router.
I am using it for sending and receiving camera compressed images. However, when using the repeater mode of version 1.1.0, the following Segmentation Fault occurred.

root@root# ddsrouter -c repeater.yaml
Starting DDS Router Tool execution.
DDS Router running.
2023-02-13 16:04:20.206 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.206 [RTPS_WRITER Error] Error sending fragment (19, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.240 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.240 [RTPS_WRITER Error] Error sending fragment (20, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.275 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.275 [RTPS_WRITER Error] Error sending fragment (21, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.322 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.322 [RTPS_WRITER Error] Error sending fragment (22, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.365 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.365 [RTPS_WRITER Error] Error sending fragment (23, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.411 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.411 [RTPS_WRITER Error] Error sending fragment (24, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.462 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.462 [RTPS_WRITER Error] Error sending fragment (25, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.482 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.482 [RTPS_WRITER Error] Error sending fragment (26, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.525 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.525 [RTPS_WRITER Error] Error sending fragment (27, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.612 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.612 [RTPS_WRITER Error] Error sending fragment (28, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.662 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.662 [RTPS_WRITER Error] Error sending fragment (29, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.695 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.695 [RTPS_WRITER Error] Error sending fragment (30, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.722 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.722 [RTPS_WRITER Error] Error sending fragment (31, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.784 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.784 [RTPS_WRITER Error] Error sending fragment (32, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.821 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.821 [RTPS_WRITER Error] Error sending fragment (33, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.852 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.852 [RTPS_WRITER Error] Error sending fragment (34, 2) -> Function deliver_sample_nts
2023-02-13 16:04:20.902 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-13 16:04:20.902 [RTPS_WRITER Error] Error sending fragment (35, 2) -> Function deliver_sample_nts
Segmentation fault (core dumped)

I really want to use repeater mode. Is there any solution?

juanlofer-eprosima · 2023-02-13T13:38:49Z

juanlofer-eprosima
Feb 13, 2023
Maintainer

Hi @kim3097

With only this information there is not much we can do, would you please share a reproducer and/or the core dump you are getting?

Thanks

0 replies

kim3097 · 2023-02-15T06:21:10Z

kim3097
Feb 15, 2023
Author

Hi @juanlofer-eprosima

Core dump file is attached.

When I analyzed the code, it seems that the underflow occurs in the add_data_frag() function.

File : DDS-Router/src/fastdds/src/cpp/rtps/messages/RTPSMessageGroup.cpp

bool RTPSMessageGroup::add_data_frag(
        const CacheChange_t& change,
        const uint32_t fragment_number,
        bool expectsInlineQos)
{
    assert(nullptr != sender_);

    EPROSIMA_LOG_INFO(RTPS_WRITER, "Sending relevant changes as DATA/DATA_FRAG messages");

    // Calculate fragment start
    uint32_t fragment_start = change.getFragmentSize() * (fragment_number - 1);
    // Calculate fragment size. If last fragment, size may be smaller
    uint32_t fragment_size = fragment_number < change.getFragmentCount() ? change.getFragmentSize() :
            change.serializedPayload.length - fragment_start;

change.serializedPayload.length is smaller than fragment_start. So fragment_size variable underflow occurs.

For reference, the mode I use is the repeater mode(turn).

Thanks

core.ddsrouter.zip

0 replies

juanlofer-eprosima · 2023-02-16T10:59:26Z

juanlofer-eprosima
Feb 16, 2023
Maintainer

Hi @kim3097

Thank you for sharing this information. A priori, it looks like a middleware issue rather than a bug in the DDS-Router itself. However, to further investigate this we would still need more information regarding your setup:

Have you compiled DDS-Router yourself or are you using a Docker image? If the former, which version of Fast-DDS is being used? Are you passing any CMake options?
Where is your application hosted? Is it using ROS 2? Which version? Docker or host installation?

If possible, a reproducer would also be very useful, so we are able to reproduce the bug in situ and test possible solutions.

Best

0 replies

kim3097 · 2023-02-17T01:27:36Z

kim3097
Feb 17, 2023
Author

Hi @juanlofer-eprosima

Thank you for the reply. I answer your questions.

Have you compiled DDS-Router yourself or are you using a Docker image? If the former, which version of Fast-DDS is being used? Are you passing any CMake options?
-> I compiled DDS-Router myself.
-> I think fastdds version is v2.9.0. Because i just pull master version using ddsrouter.repos file.
-> I not passed CMake options
Where is your application hosted? Is it using ROS 2? Which version? Docker or host installation?
-> I used ROS2 Galactic.

And it happens unconditionally in the following scenario.
Publisher node publishes H.265 encoded image using Oak-D-Pro-W camera. And Subscriber node subscribes to compressed image.

Publisher node(ros2 galactic node) --> ddsrouter(wan mode, tcp) --> ddsrouter(repeater mode) --> ddsrouter(wan mode, tcp) --> Subscriber node(ros2 galactic node)

If you have any additional questions, please let me know.

4 replies

juanlofer-eprosima Feb 17, 2023
Maintainer

Hi @kim3097

I think fastdds version is v2.9.0. Because i just pull master version using ddsrouter.repos file.

Just to be clear, are you using main or v1.1.0 DDS-Router branch? In the former case, Fast-DDS version in the .repos file would be master, while v2.9.0 in the latter.

We have conducted a quick experiment but have not been able to reproduce the issue so far. We believe the problem might have to do with a DDS middleware incompatibility issue, as DDS-Router leverages Fast-DDS but this is not the default implementation in ROS 2 Galactic. DDS-Router is designed to be compatible with any DDS middleware implementation in most cases, however some of its features rely on vendor specific characteristics.

Would you please try switching to Fast-DDS in your ROS 2 setup? All you would need to do is to export the following environment variable: export RMW_IMPLEMENTATION=rmw_fastrtps_cpp
For more information, please have a look at eProsima Fast DDS.

Cheers

kim3097 Feb 20, 2023
Author

Hi @juanlofer-eprosima

First, fastdds version is main.

I tried as per your suggestion using export RMW_IMPLEMENTATION=rmw_fastrtps_cpp. However the following error occurred when running ddsrouter.

~/DDS-Router$ ddsrouter -c wan_client.yaml
Starting DDS Router Tool execution.
2023-02-20 09:04:27.376 [RTPS_TRANSPORT_SHM Error] Failed init_port fastrtps_port7412: open_and_lock_file failed -> Function open_port_internal
2023-02-20 09:04:30.378 [RTPS_TRANSPORT_SHM Error] Failed init_port fastrtps_port7413: open_and_lock_file failed -> Function open_port_internal
DDS Router running.
ddsrouter: /home/root/DDS-Router/src/fastdds/thirdparty/boost/include/boost/interprocess/segment_manager.hpp:861: void* boost::interprocess::segment_manager<CharType, MemoryAlgorithm, IndexType>::priv_generic_find(const CharT*, IndexType<boost::interprocess::ipcdetail::index_config<CharT, MemoryAlgorithm> >&, boost::interprocess::ipcdetail::in_place_interface&, boost::interprocess::segment_manager<CharType, MemoryAlgorithm, IndexType>::size_type&, boost::interprocess::ipcdetail::true_, bool) [with CharT = char; CharType = char; MemoryAlgorithm = boost::interprocess::rbtree_best_fit<boost::interprocess::mutex_family, boost::interprocess::offset_ptr<void, unsigned int, long unsigned int> >; IndexType = boost::interprocess::iset_index; boost::interprocess::segment_manager<CharType, MemoryAlgorithm, IndexType>::size_type = unsigned int; boost::interprocess::ipcdetail::true_ = boost::interprocess::ipcdetail::bool_]: Assertion `(ctrl_data->m_value_bytes % table.size) == 0' failed.
Aborted (core dumped)

How can i solve this error?

Thanks for your help.

juanlofer-eprosima Feb 20, 2023
Maintainer

Hi @kim3097

I believe this error is the same as #349, here is the explanation for it.

The fix tackling this issue has already been added to Fast-DDS master branch, so you may either pull to get these changes, or checkout v2.9.0, which is the version tested in DDS-Router release v1.1.0 and does not present this incompatibility issue.

Best

kim3097 Feb 21, 2023
Author

Hi @juanlofer-eprosima

With your help, this error has been resolved. However, the problem is the same even if I change RMW_IMPLEMENTATION to the fastdds library. So to help you, I added the following log and it worked.

File : DDS-Router/src/fastdds/src/cpp/rtps/messages/RTPSMessageGroup.cpp

bool RTPSMessageGroup::add_data_frag(
        const CacheChange_t& change,
        const uint32_t fragment_number,
        bool expectsInlineQos)
{
    assert(nullptr != sender_);

    EPROSIMA_LOG_INFO(RTPS_WRITER, "Sending relevant changes as DATA/DATA_FRAG messages");

    // Calculate fragment start
    uint32_t fragment_start = change.getFragmentSize() * (fragment_number - 1);
    // Calculate fragment size. If last fragment, size may be smaller
    uint32_t fragment_size = fragment_number < change.getFragmentCount() ? change.getFragmentSize() :
            change.serializedPayload.length - fragment_start;

    // Check limitation
    if (data_exceeds_limitation(fragment_size, sent_bytes_limitation_, current_sent_bytes_, full_msg_->length))
    {
        flush_and_reset();
        throw limit_exceeded();
    }

    // Check preconditions. If fail flush and reset.
    check_and_maybe_flush();
    add_info_ts_in_buffer(change.sourceTimestamp);

    CacheChangeInlineQoSWriter qos_writer(change);
    InlineQosWriter* inline_qos;
    inline_qos = (change.inline_qos.length > 0 && nullptr != change.inline_qos.data) ? &qos_writer : nullptr;

#if HAVE_SECURITY
    uint32_t from_buffer_position = submessage_msg_->pos;
#endif // if HAVE_SECURITY
    const EntityId_t& readerId = get_entity_id(sender_->remote_guids());

    EPROSIMA_LOG_ERROR(RTPS_WRITER, "LOG :: fragment_size = " << fragment_size << " fragment_number = " << fragment_number << " change.getFragmentCount() = " << change.getFragmentCount() << " change.getFragmentSize() = " << change    .getFragmentSize() << " change.serializedPayload.length = " << change.serializedPayload.length << " fragment_start = " << fragment_start);

    // TODO (Ricardo). Check to create special wrapper.
    CacheChange_t change_to_add;
    change_to_add.copy_not_memcpy(&change);
    change_to_add.serializedPayload.data = change.serializedPayload.data + fragment_start;
    change_to_add.serializedPayload.length = fragment_size;

Bash Display

2023-02-21 09:42:36.258 [RTPS_WRITER Error] LOG :: fragment_size = 65324 fragment_number = 1 change.getFragmentCount() = 2 change.getFragmentSize() = 65324 change.serializedPayload.length = 21124 fragment_start = 0 -> Function add_data_frag
2023-02-21 09:42:36.258 [RTPS_WRITER Error] LOG :: msg->pos = 64 msg->max_size = 65504 payload.length = 65324 added_no_error = 1 -> Function addSubmessageDataFrag
2023-02-21 09:42:36.258 [RTPS_WRITER Error] LOG :: fragment_size = 4294923096 fragment_number = 2 change.getFragmentCount() = 2 change.getFragmentSize() = 65324 change.serializedPayload.length = 21124 fragment_start = 65324 -> Function add_data_frag
2023-02-21 09:42:36.258 [RTPS_WRITER Error] LOG :: msg->pos = 48 msg->max_size = 65504 payload.length = 4294923096 added_no_error = 1 -> Function addSubmessageDataFrag
2023-02-21 09:42:36.258 [RTPS_WRITER Error] Cannot add DATA_FRAG submsg to the CDRMessage. Buffer too small -> Function add_data_frag
2023-02-21 09:42:36.258 [RTPS_WRITER Error] Error sending fragment (31, 2) -> Function deliver_sample_nts

So change.serializedPayload.length is less than fragment_start. That's why the fargment_size variable is underflowed. I do not understand.
Packet fragment situation occurred, why is change.serializedPayload.length less than 65324?

I'm asking for help because it's taking too long for me to debug. thank you

kim3097 · 2023-02-28T01:57:06Z

kim3097
Feb 28, 2023
Author

Hi @juanlofer-eprosima

Please inquire if you are following the issue. thank you

2 replies

juanlofer-eprosima Mar 1, 2023
Maintainer

Hi @kim3097

We are working on it, will let you know as soon as there is an update. On the meantime, and as I have already mentioned in previous comments, it would be extremely useful for us if you could provide a simple reproducer, as we are still incapable of reproducing the issue in our machines.

Best

kim3097 Mar 6, 2023
Author

Hi @juanlofer-eprosima

Unfortunately, the publisher ros2 node code is not mine, so I don't provide publisher ros2 node code.
But in my machine it is 100% reproducible.

As explained before, it consists of the following environment.

Publisher node(ros2 galactic node) --> ddsrouter(wan mode, tcp) --> ddsrouter(repeater mode) --> ddsrouter(wan mode, tcp) --> Subscriber node(ros2 galactic node)

I know that the publisher node is taken from the depthai open source.
(github url : https://github.com/luxonis/depthai-ros-examples/blob/main/depthai_examples/ros2_src/stereo_publisher.cpp)

We worked on h.265 encoding in stereo_publisher.cpp code.
Do you have any h.265 streaming example code?
I think it will happen in a streaming environment. I hope this information helps.
Thanks.

Tempate · 2024-06-11T06:20:55Z

Tempate
Jun 11, 2024

Hi @kim3097,

Recently we were able to come up with a test (#456) to replicate the bug you discovered more than a year ago. We have implemented a fix in the DDS Pipe (eProsima/DDS-Pipe#9).

I hope this solves your issue.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Turn repeater DDS Router Segmentaion Fault [17325] #353

{{title}}

Replies: 6 comments 6 replies

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Turn repeater DDS Router Segmentaion Fault [17325] #353

kim3097 Feb 13, 2023

Replies: 6 comments · 6 replies

juanlofer-eprosima Feb 13, 2023 Maintainer

kim3097 Feb 15, 2023 Author

juanlofer-eprosima Feb 16, 2023 Maintainer

kim3097 Feb 17, 2023 Author

juanlofer-eprosima Feb 17, 2023 Maintainer

kim3097 Feb 20, 2023 Author

juanlofer-eprosima Feb 20, 2023 Maintainer

kim3097 Feb 21, 2023 Author

kim3097 Feb 28, 2023 Author

juanlofer-eprosima Mar 1, 2023 Maintainer

kim3097 Mar 6, 2023 Author

Tempate Jun 11, 2024

kim3097
Feb 13, 2023

Replies: 6 comments 6 replies

juanlofer-eprosima
Feb 13, 2023
Maintainer

kim3097
Feb 15, 2023
Author

juanlofer-eprosima
Feb 16, 2023
Maintainer

kim3097
Feb 17, 2023
Author

juanlofer-eprosima Feb 17, 2023
Maintainer

kim3097 Feb 20, 2023
Author

juanlofer-eprosima Feb 20, 2023
Maintainer

kim3097 Feb 21, 2023
Author

kim3097
Feb 28, 2023
Author

juanlofer-eprosima Mar 1, 2023
Maintainer

kim3097 Mar 6, 2023
Author

Tempate
Jun 11, 2024