Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Endorser client failed/refused to connect when joining a channel (remote development) #378

Open
lcvalves opened this issue Feb 14, 2023 · 2 comments
Open

Endorser client failed/refused to connect when joining a channel (remote development) #378

lcvalves opened this issue Feb 14, 2023 · 2 comments

Comments

@lcvalves
Copy link

Greetings!

I have a multi-org single peer network running on a remote single Azure CentOS VM.

SOMETIMES, whenever I try to recreate/reset the network, some peers fail to join the defined channel on fablo-config.json (in logs: stvgd-channel) . This happens occasionally and randomly as the peer that fails to join can be from any of the orgs besides the one creating it (in logs: Inovafil).

E.g., in the logs below, the peer in ASampaio Org fails to join, but sometimes it joins successfully. If that happens however, one of the peers in the remaining organizations can also fail. Listed below are the aforementioned logs:

============ 😻 Creating 'stvgd-channel' on Inovafil/peer0 😻 ==============
Creating channel with name (TLS): stvgd-channel
   Orderer: orderer0.ordering-service.inovafil.stvgd.com:7030
   CORE_PEER_LOCALMSPID: InovafilMSP
   CORE_PEER_ADDRESS: peer0.inovafil.stvgd.com:7021
   CORE_PEER_MSPCONFIGPATH: /var/hyperledger/cli/crypto/users/[email protected]/msp
   TLS_CA_CERT_PATH is: /var/hyperledger/cli/crypto-orderer/tlsca.inovafil.stvgd.com-cert.pem
   CORE_PEER_TLS_CERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.crt
   CORE_PEER_TLS_KEY_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.key
   CORE_PEER_TLS_ROOTCERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/ca.crt
2023-02-13 16:12:41.807 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2023-02-13 16:12:41.862 UTC [cli.common] readBlock -> INFO 002 Expect block, but got status: &{NOT_FOUND}
2023-02-13 16:12:41.866 UTC [channelCmd] InitCmdFactory -> INFO 003 Endorser and orderer connections initialized
2023-02-13 16:12:42.068 UTC [cli.common] readBlock -> INFO 004 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.072 UTC [channelCmd] InitCmdFactory -> INFO 005 Endorser and orderer connections initialized
2023-02-13 16:12:42.273 UTC [cli.common] readBlock -> INFO 006 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.277 UTC [channelCmd] InitCmdFactory -> INFO 007 Endorser and orderer connections initialized
2023-02-13 16:12:42.478 UTC [cli.common] readBlock -> INFO 008 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.483 UTC [channelCmd] InitCmdFactory -> INFO 009 Endorser and orderer connections initialized
2023-02-13 16:12:42.686 UTC [cli.common] readBlock -> INFO 00a Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.693 UTC [channelCmd] InitCmdFactory -> INFO 00b Endorser and orderer connections initialized
2023-02-13 16:12:42.895 UTC [cli.common] readBlock -> INFO 00c Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.900 UTC [channelCmd] InitCmdFactory -> INFO 00d Endorser and orderer connections initialized
2023-02-13 16:12:43.104 UTC [cli.common] readBlock -> INFO 00e Received block: 0
2023-02-13 16:12:43.177 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2023-02-13 16:12:43.438 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel
==== 😸 Joining 'stvgd-channel' on  ASampaio/peer0 😸 ====
Fetching channel with name (TLS): stvgd-channel
   Orderer: orderer0.ordering-service.inovafil.stvgd.com:7030
   CORE_PEER_LOCALMSPID: ASampaioMSP
   CORE_PEER_ADDRESS: peer0.asampaio.stvgd.com:7041
   CORE_PEER_MSPCONFIGPATH: /var/hyperledger/cli/crypto/users/[email protected]/msp
   TLS_CA_CERT_PATH is: /var/hyperledger/cli/crypto-orderer/tlsca.inovafil.stvgd.com-cert.pem
   CORE_PEER_TLS_CERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.crt
   CORE_PEER_TLS_KEY_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.key
   CORE_PEER_TLS_ROOTCERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/ca.crt
2023-02-13 16:12:43.617 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2023-02-13 16:12:43.621 UTC [cli.common] readBlock -> INFO 002 Received block: 0
Error: error getting endorser client for channel: endorser client failed to connect to peer0.asampaio.stvgd.com:7041: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 172.21.0.6:7041: connect: connection refused"

My current workaround this issue is to simple iterate through a fablo reset/prune/recreate cycle until it starts working because this issue happens randomly. One thing that I've found is that if I run this locally it always starts successfully with no critical errors.

Any clues on what this might be? I'm thinking it has to do with initializing the docker containers and its timeouts inside the VM but I've decided to consult the creators and the Fablo community, hence this issue.

Cheers and thanks in advance.
@dzikowski
Copy link
Contributor

dzikowski commented Feb 14, 2023 •
edited
Loading

We observed similar issues when there was not enough memory available for Docker. What are the limits on your remote VM? (both VM RAM and Docker RAM limits). I feel for a multiorg setup (2-4 orgs, single peer per org) it is save to have like 6-8 GB RAM. (But we don't have any benchmarks for that).

@lcvalves
Copy link
Author

In the current scenario, there are 5 single peer Orgs.

I'm using an 8GB RAM remote VM and each Docker container is limited to 8GB of RAM as well.

For reference, here is the output of docker stats command:

CONTAINER ID   NAME                                                                                                                      CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
fd97b05127ee   dev-peer0.tintex.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2            0.00%     9.93MiB / 2GiB        0.48%     1.7MB / 1.87MB    0B / 0B           6
ace2ad35f5ba   dev-peer0.inovafil.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2          0.00%     10.13MiB / 2GiB       0.49%     1.81MB / 1.97MB   0B / 0B           6
ee2664303a58   dev-peer0.asampaio.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2          0.00%     10.17MiB / 2GiB       0.50%     1.74MB / 1.91MB   0B / 0B           6
b528abfe2489   dev-peer0.domingosdesousa.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2   0.00%     9.977MiB / 2GiB       0.49%     1.79MB / 1.95MB   0B / 0B           6
026ac65fd7e8   dev-peer0.tmg.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2               0.00%     10.07MiB / 2GiB       0.49%     1.79MB / 1.96MB   0B / 0B           6
f51aeb5af6f7   explorer.example.com                                                                                                      12.41%    436.5MiB / 7.638GiB   5.58%     3.76GB / 1.21GB   2.2MB / 12.3kB    79
9b67acfc9f58   db.explorer.example.com                                                                                                   2.19%     44MiB / 7.638GiB      0.56%     755MB / 368MB     4.1kB / 57.2MB    13
a3a1cd1d9402   cli.tmg.stvgd.com                                                                                                         0.00%     564KiB / 7.638GiB     0.01%     109kB / 3.61MB    0B / 0B           1
7b3472db2878   couchdb.peer0.asampaio.stvgd.com                                                                                          0.57%     48.36MiB / 7.638GiB   0.62%     12.2MB / 18.6MB   0B / 3.65MB       38
e7f8a6f4d2af   orderer0.ordering-service.inovafil.stvgd.com                                                                              0.47%     29.43MiB / 7.638GiB   0.38%     855MB / 858MB     0B / 1.27MB       9
763d30ad33c7   orderer0.ordering-service.tintex.stvgd.com                                                                                0.57%     25.91MiB / 7.638GiB   0.33%     860MB / 862MB     0B / 1.24MB       9
4093d41cfe1b   ca.tintex.stvgd.com                                                                                                       0.00%     5.867MiB / 7.638GiB   0.08%     13.4kB / 142B     0B / 311kB        6
0d8c0f922391   orderer0.ordering-service.tmg.stvgd.com                                                                                   0.28%     23.19MiB / 7.638GiB   0.30%     356MB / 354MB     0B / 1.28MB       8
0e4d455e5aed   couchdb.peer0.inovafil.stvgd.com                                                                                          2.17%     48.7MiB / 7.638GiB    0.62%     11.8MB / 18.6MB   3.89MB / 3.58MB   38
aff00e35cae9   cli.tintex.stvgd.com                                                                                                      0.00%     564KiB / 7.638GiB     0.01%     106kB / 3.61MB    0B / 0B           1
48f0293fdf2b   orderer0.ordering-service.asampaio.stvgd.com                                                                              0.32%     23.92MiB / 7.638GiB   0.31%     358MB / 356MB     0B / 1.3MB        9
a302eb605979   couchdb.peer0.domingosdesousa.stvgd.com                                                                                   0.55%     49.39MiB / 7.638GiB   0.63%     11.9MB / 18.6MB   0B / 3.24MB       38
705cdfbf643b   couchdb.peer0.tintex.stvgd.com                                                                                            0.77%     47.34MiB / 7.638GiB   0.61%     11.7MB / 18.6MB   0B / 3.28MB       38
c12860ea623c   ca.inovafil.stvgd.com                                                                                                     0.00%     5.922MiB / 7.638GiB   0.08%     22.8kB / 27kB     0B / 569kB        6
5f8f1e69c5bd   ca.tmg.stvgd.com                                                                                                          0.00%     5.93MiB / 7.638GiB    0.08%     38.9kB / 34.2kB   0B / 307kB        6
1d3761224ee1   fablo-rest.inovafil.stvgd.com                                                                                             0.00%     41.61MiB / 7.638GiB   0.53%     7.92MB / 8.09MB   0B / 0B           11
b85a80ae8595   cli.asampaio.stvgd.com                                                                                                    0.00%     556KiB / 7.638GiB     0.01%     104kB / 3.61MB    0B / 0B           1
edb3f6c5fe73   peer0.inovafil.stvgd.com                                                                                                  4.57%     78.73MiB / 7.638GiB   1.01%     625MB / 1.2GB     0B / 1.58MB       9
6404e30c1ac3   peer0.asampaio.stvgd.com                                                                                                  1.89%     81.43MiB / 7.638GiB   1.04%     625MB / 1.2GB     0B / 1.72MB       9
0b7dfa9f11fe   fablo-rest.tintex.stvgd.com                                                                                               0.00%     33.52MiB / 7.638GiB   0.43%     12.7kB / 142B     0B / 0B           7
344a4390f1a4   peer0.tmg.stvgd.com                                                                                                       2.20%     89.45MiB / 7.638GiB   1.14%     624MB / 1.2GB     0B / 1.62MB       9
863d54334d0c   couchdb.peer0.tmg.stvgd.com                                                                                               0.61%     48.23MiB / 7.638GiB   0.62%     11.8MB / 18.6MB   0B / 3.08MB       38
5f992dffc3fe   fablo-rest.asampaio.stvgd.com                                                                                             0.00%     41.9MiB / 7.638GiB    0.54%     8.1MB / 8.16MB    0B / 0B           11
f81d9913d221   orderer0.ordering-service.domingosdesousa.stvgd.com                                                                       0.32%     20.86MiB / 7.638GiB   0.27%     360MB / 358MB     0B / 1.26MB       9
a8cccdcd3da7   fablo-rest.tmg.stvgd.com                                                                                                  0.00%     34.09MiB / 7.638GiB   0.44%     11.5kB / 142B     0B / 0B           7
0ef9116e9a29   ca.domingosdesousa.stvgd.com                                                                                              0.00%     5.922MiB / 7.638GiB   0.08%     14.7kB / 1.17kB   0B / 324kB        6
2976324ffc3e   cli.domingosdesousa.stvgd.com                                                                                             0.00%     556KiB / 7.638GiB     0.01%     111kB / 3.61MB    0B / 0B           1
8ac7f9881666   ca.asampaio.stvgd.com                                                                                                     0.00%     5.918MiB / 7.638GiB   0.08%     31.2kB / 53kB     0B / 1.2MB        6
fc0f239cfe44   peer0.domingosdesousa.stvgd.com                                                                                           1.82%     89.14MiB / 7.638GiB   1.14%     628MB / 1.21GB    0B / 1.86MB       8
2dbe3b898482   fablo-rest.domingosdesousa.stvgd.com                                                                                      0.00%     34.12MiB / 7.638GiB   0.44%     11.3kB / 0B       0B / 0B           7
62a62e6baf37   peer0.tintex.stvgd.com                                                                                                    1.92%     88.8MiB / 7.638GiB    1.14%     625MB / 1.2GB     0B / 1.76MB       8
48ad2c8e3b46   cli.inovafil.stvgd.com                                                                                                    0.00%     37.04MiB / 7.638GiB   0.47%     170kB / 3.67MB    72.7MB / 0B       1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@dzikowski @lcvalves