Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

gossipd spamming log, lightning_connectd eating all upload #7899

Open
pabpas opened this issue Dec 3, 2024 · 5 comments
Open

gossipd spamming log, lightning_connectd eating all upload #7899

pabpas opened this issue Dec 3, 2024 · 5 comments

Comments

@pabpas
Copy link

pabpas commented Dec 3, 2024

Issue and Steps to Reproduce

Noticed this today after upgrading to v24.11rc2 a couple of days ago.
Symptoms:

  • lightning_connectd eating all upload bandwidth
  • Number of established connections by lightning_connectd seem stable at ~23
  • gossipd hammering the logs with:
2024-12-03T10:34:21.124Z DEBUG   0359149077d5a361134dd8838633b35c5c7558198e7f01c3bfa517b8a0c482a5f6-gossipd: Bad gossip order: Unknown channel 790122x3526x1
2024-12-03T10:34:21.124Z DEBUG   0359149077d5a361134dd8838633b35c5c7558198e7f01c3bfa517b8a0c482a5f6-gossipd: Bad gossip order: node_announcement: unknown node 02b70b805ee7419b63f91f735eadf9d943be41a93b4dcb9d3c778deaedc3caf4c0
2024-12-03T10:34:21.124Z DEBUG   0359149077d5a361134dd8838633b35c5c7558198e7f01c3bfa517b8a0c482a5f6-gossipd: Bad gossip order: Unknown channel 790122x3526x1
2024-12-03T10:34:21.124Z DEBUG   gossipd: channel_announcement: Adding 790148x4442x1 to pending...
2024-12-03T10:34:21.124Z DEBUG   gossipd: channel_announcement: Adding 790150x218x0 to pending...
2024-12-03T10:34:21.124Z DEBUG   gossipd: channel_announcement: Adding 790162x1025x0 to pending...
2024-12-03T10:34:21.183Z DEBUG   gossipd: ...[1004 debug/trace messages suppressed]...
2024-12-03T10:34:21.183Z DEBUG   gossipd: channel_announcement: Adding 791401x1029x0 to pending...
2024-12-03T10:34:21.184Z DEBUG   gossipd: channel_announcement: Adding 791409x545x0 to pending...
2024-12-03T10:34:21.184Z DEBUG   gossipd: channel_announcement: Adding 791411x421x1 to pending...
2024-12-03T10:34:21.184Z DEBUG   gossipd: channel_announcement: Adding 791485x1220x0 to pending...
2024-12-03T10:34:21.200Z DEBUG   gossipd: ...[530 debug/trace messages suppressed]...
2024-12-03T10:34:21.200Z DEBUG   gossipd: channel_announcement: Adding 792164x138x0 to pending...
2024-12-03T10:34:21.200Z DEBUG   gossipd: channel_announcement: Adding 792166x1177x1 to pending...
2024-12-03T10:34:21.200Z DEBUG   gossipd: channel_announcement: Adding 792166x1728x1 to pending...
2024-12-03T10:34:21.200Z DEBUG   gossipd: channel_announcement: Adding 792167x1601x0 to pending...
2024-12-03T10:34:21.203Z DEBUG   gossipd: ...[121 debug/trace messages suppressed]...
[...]
2024-12-03T10:38:11.696Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 02b9f5eaad9609d209e4acf1411bd1346d012ee06545fa2cbef2de10a3e5fbf7e0
2024-12-03T10:38:11.696Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 038ba8f67ba8ff5c48764cdd3251c33598d55b203546d08a8f0ec9dcd9f27e3637
2024-12-03T10:38:11.733Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 03c5bfa5cab7e52e7cebc45fd1b86ea30ad518648651314563452b78b822333660
2024-12-03T10:38:11.733Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 031ae9d12b098017df39c28647fa78b9d97c40f098e2ce134dcd11162541a60faa
2024-12-03T10:38:11.734Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 033328260071d0dca64af19c71fff947b894026e33cafeedbb5b41e87e752075a4
2024-12-03T10:38:11.734Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 02903e2ea67a3cc2e0ff75053ac64be9d5b17840274412be4c9cccf62b10be49f3
2024-12-03T10:38:11.734Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 026d1de80e59f3fe2b27aa3dc8012f4437165bf99971bd9b7d8afedb066f92d695
2024-12-03T10:38:11.734Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 02f2ef56264092d7ad5e2286af6979be59fdd0f3384b37f407fd41fd0c61f6722f
2024-12-03T10:38:11.734Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 0254bb156ecd0eac318844415a91a377bc6947ea4c9fbe5d248e563c29a1662835
2024-12-03T10:38:11.734Z DEBUG   02c80136d51ee2605a32fdf015bf15f5651aafed07934091011cec88822a1a26dd-gossipd: Bad gossip order: node_announcement: unknown node 03b64315b0102bd8e49121ff36ed1418bfaeb2a83d9d239bfe948fa7e442f63aab
2024-12-03T10:38:12.376Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 776657x1611x1
2024-12-03T10:38:12.376Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 832006x2237x0
2024-12-03T10:38:12.376Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 861028x1780x1
2024-12-03T10:38:12.376Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 804737x755x1
2024-12-03T10:38:12.376Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 809912x1990x3
2024-12-03T10:38:12.401Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 872601x1115x1
2024-12-03T10:38:12.401Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 762504x1739x1
2024-12-03T10:38:12.402Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 861057x1872x2
2024-12-03T10:38:12.402Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 829167x653x1
2024-12-03T10:38:12.402Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 833022x2543x0
2024-12-03T10:38:12.402Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 813188x2939x0
2024-12-03T10:38:12.402Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 813310x1644x1
2024-12-03T10:38:12.402Z DEBUG   0211951b1bec28fe3173db1147bdbb2f377abb552123cbc2ec6b38637a57e18cd5-gossipd: Bad gossip order: Unknown channel 864971x1255x3
2024-12-03T10:38:12.838Z DEBUG   02c5333bbaee6c7ce9bb80ad601970140434ee180ab7063ffde8f9ec6bfc35825f-gossipd: Bad gossip order: Unknown channel 799338x773x0
2024-12-03T10:38:12.838Z DEBUG   02c5333bbaee6c7ce9bb80ad601970140434ee180ab7063ffde8f9ec6bfc35825f-gossipd: Bad gossip order: Unknown channel 627732x1261x0
2024-12-03T10:38:12.838Z DEBUG   02c5333bbaee6c7ce9bb80ad601970140434ee180ab7063ffde8f9ec6bfc35825f-gossipd: Bad gossip order: Unknown channel 798016x2358x0

other outputs

"version": "v24.11rc2",
"blockheight": 873056,
 "network": "bitcoin",

ls -lah .lightning/bitcoin/
2.9G Dec  3 10:27 gossip_store
44M Dec  1 06:57 gossip_store.corrupt
@pabpas
Copy link
Author

pabpas commented Dec 3, 2024

Probably related, alias are gone.

$ lightning-cli listnodes
{
   "nodes": [
      {
         "nodeid": "0225e29bcf897f22d9f546ad71185f9519db538d484b2fd2ea2be02e20b5c7224b"
      },
      {
         "nodeid": "02c095d069538f96bf14c5f90f6c0851bdf354a0ec86039a24bf38a73f705adc2c"
      },
      {
         "nodeid": "033533e2db6311c1d417d41279c067d8713ad8a7d577a0d91e65df2f6a82c5b862"
      },
      {
         "nodeid": "02a20247f515d978cbf9e9ce6a4287b5931d724068b7b88bbaead6380db3dd8e9a"
      },
      {
         "nodeid": "022bd0aa893db4ac890e457cca8c83f112518d6941bf9153dab4bf904620503a78"
      },
      {
         "nodeid": "032824b8e1eda7081692ab7884426c276ee937fd3d4aa6e5170f178ff866f909ab"
      },
      {
         "nodeid": "0391e2edea5191627a25ecbd327c0dc2a95c880a5b0e73af38dc4a5a8964263b3f"
      },
      {
         "nodeid": "0329d096c7574676082e2fccccd3d401bacfa742c644e4452e797f43d9812c9362"
      },
      {
         "nodeid": "02de1e6a5fc53775e7c96e70c9f81827795d5b36c63345c54ab64651abec8b200d"
      },
      {
         "nodeid": "02966a90f51a0874f706d243878b58eafdf15a2369eaf6fbd1f3273159f0d0e130"
      },
      {
         "nodeid": "0368a6ba0859074d665af172333818c4b94d202909c8745008eccfd4d6aef464f8"
      },
...

@pabpas
Copy link
Author

pabpas commented Dec 3, 2024

The issue with bandwidth is temporarily solved by disabling Tor. No idea what I was sending but it was constant and maxing out upload.

After disabling Tor:
Screenshot_20241203_185853

@rustyrussell
Copy link
Contributor

Thanks for the report!

  1. gossip_store.corrupt means your gossip store was emptied and started from scratch. You should have a message containing "Moving to gossip_store.corrupt" in your logs? Can you share?
  2. Your node should have caught up (you can see you're missing node announcements), it's extremely odd that it hasn't.
  3. Someone on Tor was asking you for a lot of gossip. I'm not sure why?
  4. Gossipd debug-level logs are spammy, esp. on large nodes. This I can fix easily.

We ask a random peer for all its gossip once an hour now. So your node should get gossip pretty fast.

rustyrussell added a commit to rustyrussell/lightning that referenced this issue Dec 3, 2024
@rustyrussell
gossipd: log at trace, not debug for regular messages.
a20eb3e 
See: ElementsProject#7899

A node with 23 connections gets far too many debug messages.

Changelog-Fixed: `gossipd` now does logging at trace, not debug level.
Signed-off-by: Rusty Russell <[email protected]>
@rustyrussell rustyrussell mentioned this issue Dec 3, 2024
Open
@hMsats
Copy link
Contributor

hMsats commented Dec 4, 2024 •
edited
Loading

I get these:

2024-12-04T01:54:39.381Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 760618x1099x0
2024-12-04T01:54:39.451Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 761478x985x0
2024-12-04T01:54:39.451Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 763089x1549x1
2024-12-04T01:54:39.451Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 764054x1059x0
2024-12-04T01:54:39.451Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 764212x828x0
2024-12-04T01:54:39.451Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 764618x545x1
2024-12-04T01:54:39.452Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 764622x1573x0
2024-12-04T01:54:41.031Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 764992x1167x0
2024-12-04T01:54:41.031Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 765971x1408x3
2024-12-04T01:54:41.031Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 770301x238x1
2024-12-04T01:54:41.031Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 774164x2700x4
2024-12-04T01:54:41.031Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 774355x2191x1
2024-12-04T01:54:41.031Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 779330x1492x1
2024-12-04T01:54:41.032Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 785430x1239x1
2024-12-04T01:54:41.067Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 785588x1777x1
2024-12-04T01:54:41.067Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 794914x2075x0
2024-12-04T01:54:41.088Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 796759x2186x1
2024-12-04T01:54:41.088Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 797021x3065x0
2024-12-04T01:54:41.088Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 797696x149x0
2024-12-04T01:54:41.088Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 797998x601x0
2024-12-04T01:54:41.088Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 804043x2261x0
2024-12-04T01:54:41.088Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 804200x2134x1
2024-12-04T01:54:41.110Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 806542x1406x24
2024-12-04T01:54:41.110Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 807565x1302x1
2024-12-04T01:54:41.110Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 808623x982x1
2024-12-04T01:54:41.110Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 809492x3020x4
2024-12-04T01:54:41.110Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 809847x832x1
2024-12-04T01:54:41.110Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 809960x1364x1
2024-12-04T01:54:41.110Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 810171x70x1
2024-12-04T01:54:41.111Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 810628x2045x1
2024-12-04T01:54:41.111Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 822516x3829x0
2024-12-04T01:54:41.111Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 825931x1098x1
2024-12-04T01:54:41.111Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 827795x3068x1
2024-12-04T01:54:41.111Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 829095x2021x1
2024-12-04T01:54:41.111Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 829574x928x1
2024-12-04T01:54:41.150Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 831528x1085x4
2024-12-04T01:54:41.150Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 832338x664x1
2024-12-04T01:54:41.150Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 832682x2607x1
2024-12-04T01:54:41.150Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 832890x2527x2
2024-12-04T01:54:41.151Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 833019x927x1
2024-12-04T01:54:41.151Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 833355x150x1
2024-12-04T01:54:41.151Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 833515x3419x0
2024-12-04T01:54:41.151Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 833699x3635x0
2024-12-04T01:54:41.151Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 834991x2011x1
2024-12-04T01:54:41.221Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 835082x1360x0
2024-12-04T01:54:41.222Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 835123x1107x1
2024-12-04T01:54:41.455Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 835491x1640x1
2024-12-04T01:54:41.907Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 771471x1184x1
2024-12-04T01:54:42.471Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 824453x1985x1
2024-12-04T01:56:38.815Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 689464x1242x0
2024-12-04T02:00:38.870Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 590115x1632x0
2024-12-04T02:00:39.038Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 590599x433x0
2024-12-04T02:00:39.200Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 600975x1220x0
2024-12-04T02:00:39.200Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 601887x1641x0
2024-12-04T02:00:39.298Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 643789x2799x0
2024-12-04T02:00:39.298Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 674091x2347x1
2024-12-04T02:00:39.299Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 684909x1739x1
2024-12-04T02:00:39.299Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 686448x1004x0
2024-12-04T02:00:39.299Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 687689x1725x1
2024-12-04T02:00:39.299Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 688186x1190x1
2024-12-04T02:00:39.299Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 690107x1433x0
2024-12-04T02:00:39.417Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 694215x1728x0
2024-12-04T02:00:39.417Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 694533x1601x0
2024-12-04T02:00:39.417Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 694808x535x0
2024-12-04T02:05:40.129Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 835796x1011x0
2024-12-04T02:05:40.139Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 835964x1421x1
2024-12-04T02:05:40.172Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 836249x1032x0
2024-12-04T02:05:40.172Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 836252x3018x0
2024-12-04T02:05:40.172Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 836453x1477x1
2024-12-04T02:05:40.172Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 836743x2765x1
2024-12-04T02:05:40.172Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 836868x2684x1
2024-12-04T02:05:40.172Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 837607x391x0
2024-12-04T02:05:40.172Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 837831x1952x1
2024-12-04T02:05:40.247Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 841754x1405x2
2024-12-04T02:05:40.247Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 842620x3824x0
2024-12-04T02:05:40.247Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 842622x3525x0
2024-12-04T02:05:40.247Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 842628x2838x1
2024-12-04T02:05:40.247Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 842773x2271x0
2024-12-04T02:05:40.312Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843006x1451x1
2024-12-04T02:05:40.479Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843142x400x1
2024-12-04T02:05:40.479Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843143x518x1
2024-12-04T02:05:40.480Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843335x507x0
2024-12-04T02:05:40.480Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843395x1114x0
2024-12-04T02:05:40.481Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843880x2972x0
2024-12-04T02:05:40.585Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843948x481x0
2024-12-04T02:05:40.585Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843991x2997x1
2024-12-04T02:05:40.585Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 844017x1969x0
2024-12-04T02:05:40.585Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 844095x3769x1
2024-12-04T02:05:40.710Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 844339x2509x0
2024-12-04T02:05:40.710Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 845106x1637x1
2024-12-04T02:05:40.765Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 845344x2636x1
2024-12-04T02:05:40.822Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 845403x3836x1
2024-12-04T02:05:40.873Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 848227x814x1
2024-12-04T02:05:40.873Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 848410x1020x0
2024-12-04T02:05:41.093Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 849245x4074x1
2024-12-04T02:05:41.094Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 849584x2081x1
2024-12-04T02:05:41.094Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 849817x534x3
2024-12-04T02:05:41.169Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 850428x4311x1
2024-12-04T02:05:41.328Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 850521x304x1
2024-12-04T02:05:41.328Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 850945x557x1
2024-12-04T02:05:41.328Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851146x1909x1
2024-12-04T02:05:41.328Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851280x3087x5
2024-12-04T02:05:41.463Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851734x2347x0
2024-12-04T02:05:41.463Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851849x2242x0
2024-12-04T02:05:41.463Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851997x1269x1
2024-12-04T02:05:41.464Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852065x687x1
2024-12-04T02:05:41.596Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852200x1895x1
2024-12-04T02:05:41.596Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852268x2153x0
2024-12-04T02:05:41.596Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852307x2810x1
2024-12-04T02:05:41.596Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852623x3614x1
2024-12-04T02:05:41.596Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852658x755x1
2024-12-04T02:05:41.861Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 845810x1806x1
2024-12-04T02:07:41.566Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 853433x2101x1
2024-12-04T02:07:41.566Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 853895x502x1
2024-12-04T02:07:41.566Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 853946x1900x1
2024-12-04T02:07:41.566Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 855083x772x1
2024-12-04T02:07:41.566Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 855300x1859x1
2024-12-04T02:07:41.607Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 855778x980x0
2024-12-04T02:07:41.678Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 856589x1357x1
2024-12-04T02:07:41.678Z INFO    0273ee10f8dd8a4796b907a85bdecdef918734ceef59f057cdaa9a45dd40bbb2c9-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 857098x1573x3
2024-12-04T03:13:18.471Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 702575x1973x1
2024-12-04T03:13:19.399Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 739063x1068x1
2024-12-04T03:13:19.400Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 755311x1310x1
2024-12-04T03:13:19.475Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 760618x1099x0
2024-12-04T03:13:24.032Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 764992x1167x0
2024-12-04T03:13:24.033Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 765971x1408x3
2024-12-04T03:13:24.033Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 771471x1184x1
2024-12-04T03:13:24.033Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 774164x2700x4
2024-12-04T03:46:20.585Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 785430x1239x1
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 794914x2075x0
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 796759x2186x1
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 797021x3065x0
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 806542x1406x24
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 822516x3829x0
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 833699x3635x0
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 836743x2765x1
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 841754x1405x2
2024-12-04T03:46:21.364Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 842622x3525x0
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843143x518x1
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 843880x2972x0
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 844017x1969x0
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851146x1909x1
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851280x3087x5
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 851997x1269x1
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852268x2153x0
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 852658x755x1
2024-12-04T03:46:21.365Z INFO    021490e3e941dc37aaf770096fd43393cdddc28750b03b4a32c0a0a04d7310a621-connectd: Received WIRE_WARNING: WARNING: channel_announcement: no unspent txout 855778x980x0
                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         ```

@pabpas
Copy link
Author

pabpas commented Dec 4, 2024

Thanks for the report!

1. gossip_store.corrupt means your gossip store was emptied and started from scratch.  You should have a message containing "Moving to gossip_store.corrupt" in your logs?  Can you share?

Thank you for your software!

These are the relevant parts I believe. Note that this rc2 had been running for 2 days, it did not happen right after updating.

2024-12-01T06:57:03.914Z **BROKEN** gossipd: gossip_store: get delete entry offset 17733380/7989 (version v24.11rc2)
2024-12-01T06:57:03.914Z **BROKEN** gossipd: backtrace: common/daemon.c:38 (send_backtrace) 0x5613a12e18fa
2024-12-01T06:57:03.914Z **BROKEN** gossipd: backtrace: common/status.c:221 (status_failed) 0x5613a12f049f
2024-12-01T06:57:03.914Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:466 (gossip_store_get_with_hdr) 0x5613a12d4eef
2024-12-01T06:57:03.914Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:491 (check_msg_type) 0x5613a12d5032
2024-12-01T06:57:03.914Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:509 (gossip_store_set_flag) 0x5613a12d510a
2024-12-01T06:57:03.915Z **BROKEN** gossipd: backtrace: gossipd/gossip_store.c:561 (gossip_store_del) 0x5613a12d53e4
2024-12-01T06:57:03.915Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:793 (process_channel_update) 0x5613a12d77b5
2024-12-01T06:57:03.915Z **BROKEN** gossipd: backtrace: gossipd/gossmap_manage.c:905 (gossmap_manage_channel_update) 0x5613a12d7d1c
2024-12-01T06:57:03.915Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:210 (handle_recv_gossip) 0x5613a12d2cb0
2024-12-01T06:57:03.915Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:302 (connectd_req) 0x5613a12d2e64
2024-12-01T06:57:03.916Z **BROKEN** gossipd: backtrace: common/daemon_conn.c:35 (handle_read) 0x5613a12e1f79
2024-12-01T06:57:03.916Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:60 (next_plan) 0x5613a1387ad7
2024-12-01T06:57:03.916Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:422 (do_plan) 0x5613a13886a2
2024-12-01T06:57:03.916Z **BROKEN** gossipd: backtrace: ccan/ccan/io/io.c:439 (io_ready) 0x5613a1388705
2024-12-01T06:57:03.916Z **BROKEN** gossipd: backtrace: ccan/ccan/io/poll.c:455 (io_loop) 0x5613a138a9a5
2024-12-01T06:57:03.916Z **BROKEN** gossipd: backtrace: gossipd/gossipd.c:665 (main) 0x5613a12d3c0d
2024-12-01T06:57:03.917Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0x7fa8a4955d79
2024-12-01T06:57:03.917Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0x5613a12cfd59
2024-12-01T06:57:03.917Z **BROKEN** gossipd: backtrace: (null):0 ((null)) 0xffffffffffffffff
2024-12-01T06:57:03.917Z **BROKEN** gossipd: STATUS_FAIL_INTERNAL_ERROR: gossip_store: get delete entry offset 17733380/7989




2024-12-01T06:57:04.269Z INFO    lightningd: v24.11rc2
2024-12-01T06:57:04.304Z DEBUG   lightningd: Opened log file ~/.lightning/debug.log


2024-12-01T06:57:42.274Z DEBUG   gossipd: pid 1340, msgfd 68
2024-12-01T06:57:42.275Z DEBUG   hsmd: new_client: 0
2024-12-01T06:57:42.287Z **BROKEN** gossipd: gossip_store: checksum verification failed? e456e3f8 should be 4ba09bff (offset 7989). Moving to gossip_store.corrupt and truncating
2024-12-01T06:57:42.287Z DEBUG   gossipd: Store compact time: 9 msec
2024-12-01T06:57:42.288Z DEBUG   gossipd: gossip_store: Read 0/27/9/0 cannounce/cupdate/nannounce/delete from store in 52892 bytes, now 1 bytes (populated=false)
2024-12-01T06:57:42.288Z DEBUG   gossipd: seeker: state = STARTING_UP New seeker2024-12-01T06:57:42.288Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_INIT_REPLY with 0 fds
2024-12-01T06:57:42.288Z UNUSUAL 02c095d069538f96bf14c5f90f6c0851bdf354a0ec86039a24bf38a73f705adc2c-chan#706: gossipd lost track of announced channel: re-announcing! 
2024-12-01T06:57:42.288Z UNUSUAL 033533e2db6311c1d417d41279c067d8713ad8a7d577a0d91e65df2f6a82c5b862-chan#22: gossipd lost track of announced channel: re-announcing!
...
2024-12-01T06:57:42.399Z DEBUG   lightningd: io_break: gossipd_init_done
2024-12-01T06:57:42.399Z DEBUG   lightningd: io_loop: gossip_init
...
2024-12-01T06:57:42.413Z DEBUG   gossipd: inject_gossip: WIRE_CHANNEL_ANNOUNCEMENT
2024-12-01T06:57:42.413Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_ADDGOSSIP_REPLY with 0 fds
2024-12-01T06:57:42.413Z DEBUG   gossipd: inject_gossip: WIRE_CHANNEL_UPDATE
2024-12-01T06:57:42.413Z DEBUG   gossipd: Bad gossip order: Unknown channel 855094x2343x2
2024-12-01T06:57:42.413Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_ADDGOSSIP_REPLY with 0 fds
2024-12-01T06:57:42.413Z DEBUG   gossipd: inject_gossip: WIRE_NODE_ANNOUNCEMENT
2024-12-01T06:57:42.413Z DEBUG   gossipd: Bad gossip order: node_announcement: unknown node 0225e29bcf897f22d9f546ad71185f9519db538d484b2fd2ea2be02e20b5c7224b
2024-12-01T06:57:42.413Z DEBUG   gossipd: REPLY WIRE_GOSSIPD_ADDGOSSIP_REPLY with 0 fds

2. Your node _should_ have caught up (you can see you're missing node announcements), it's extremely odd that it hasn't.

gossip_store was getting bigger and bigger so yesterday I started a new one. Now it is 70M and looks like output of listnodes is now normal.

3. Someone on Tor was asking you for a lot of gossip.  I'm not sure why?

Can I set limits on this? The amount of data I was sending was definitely too much. BTW this is over, it stopped overnight.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rustyrussell @pabpas @hMsats