Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FloodWaitError #542

Closed
HeroadZ opened this issue Sep 30, 2024 · 16 comments
Closed

FloodWaitError #542

HeroadZ opened this issue Sep 30, 2024 · 16 comments

Comments

@HeroadZ
Copy link

HeroadZ commented Sep 30, 2024 •
edited
Loading

Describe the bug

Tip

A clear and concise description of what the bug is.

only the first command will be executed after restarting.

To Reproduce

Tip

Steps to reproduce the behavior.

  1. restart the docker container by docker compose up -d
  2. send several /list command in telegram
  3. command after second doesn't have any response

Screenshots

Tip

If applicable, upload screenshots to help explain your problem.

Feed URL

Tip

If applicable, provide the feed URL that causes the bug.

Expected behavior

Tip

A clear and concise description of what you expected to happen.

Important log

Important

Rerun RSStT with DEBUG=1 set in the environment variables, and paste the log below.
If you are using Docker, you can get the log by executing:

docker logs <container name>
$ docker logs 75ac51a451a1
2024-09-30-10:59:55:INFO:RSStT.env - Config folder: /app/config
2024-09-30-10:59:56:INFO:RSStT - API_ID and/or API_HASH not set, use sample APIs instead. API_ID_PUBLISHED_FLOOD_ERROR may occur.
2024-09-30-10:59:56:INFO:RSStT.db - Successfully connected to the DB
2024-09-30-10:59:57:INFO:RSStT - RSS-to-Telegram-Bot (v2.9.0-5-g5a67570@dev, build@2024-09-17T15:29:40+00:00) started!
SELF: rss2tg Bot @chance_rss2tg_bot (5694024327)
MANAGER: 880551570
ERROR_LOGGING_CHAT: 880551570
T_PROXY (for Telegram): not set
R_PROXY (for RSS): not set
DATABASE: sqlite:/app/config/db.sqlite3
TELEGRAPH: Disable
UVLOOP: Enable
MULTIUSER: Enable
CPU: 1 (usable) / 1 (available) / 1 (total)
2024-09-30-11:00:01:WARNING:RSStT.locks - Blocking any further messages for 861035684 due to flood control, 19152.00s left
2024-09-30-11:00:01:WARNING:RSStT.locks - Blocking any further messages for 1292505774 due to flood control, 19152.00s left
2024-09-30-11:00:08:INFO:RSStT.command - Allow blink (880551570) to use /export
2024-09-30-11:00:08:INFO:RSStT.command - Exported feed(s) for 880551570
2024-09-30-11:00:15:WARNING:RSStT.locks - Blocking any further messages for 118069398 due to flood control, 19138.00s left
2024-09-30-11:00:15:INFO:RSStT.command - Allow blink (880551570) to use /activate_subs
2024-09-30-11:00:15:ERROR:RSStT.command - Uncaught error occurred when blink (880551570) attempting to use /activate_subs
Traceback (most recent call last):
  File "/app/src/command/utils.py", line 505, in wrapper
    await execute()
  File "/app/src/command/utils.py", line 368, in execute
    await asyncio.wait_for(
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/app/src/command/customization.py", line 284, in cmd_activate_or_deactivate_subs
    await callback_get_activate_or_deactivate_page.__wrapped__(event, activate, lang=lang, chat_id=chat_id, page=1)
  File "/app/src/command/customization.py", line 334, in callback_get_activate_or_deactivate_page
    await (
  File "/opt/venv/lib/python3.12/site-packages/telethon/tl/custom/message.py", line 764, in respond
    return await self._client.send_message(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/telethon/client/messages.py", line 905, in send_message
    result = await self(request)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/telethon/client/users.py", line 30, in __call__
    return await self._call(self._sender, request, ordered=ordered)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/telethon/client/users.py", line 56, in _call
    raise errors.FloodWaitError(request=r, capture=diff)
telethon.errors.rpcerrorlist.FloodWaitError: A wait of 19137 seconds is required (caused by SendMessageRequest)
2024-09-30-11:00:15:WARNING:RSStT.locks - Blocking any further messages for 880551570 due to flood control, 19138.00s left
2024-09-30-11:00:42:WARNING:RSStT.locks - Blocking any further messages for 557382913 due to flood control, 19111.00s left
2024-09-30-11:00:45:WARNING:RSStT.locks - Blocking any further messages for -1001827528289 due to flood control, 19108.00s left
2024-09-30-11:01:12:WARNING:RSStT.locks - Blocking any further messages for -1001862440612 due to flood control, 19081.00s left
2024-09-30-11:01:43:WARNING:RSStT.locks - Blocking any further messages for -1002005713895 due to flood control, 19049.00s left
2024-09-30-11:01:54:WARNING:RSStT.locks - Blocking any further messages for -1001877989123 due to flood control, 19039.00s left
2024-09-30-11:03:26:INFO:RSStT.env - Config folder: /app/config
2024-09-30-11:03:27:INFO:RSStT - API_ID and/or API_HASH not set, use sample APIs instead. API_ID_PUBLISHED_FLOOD_ERROR may occur.
2024-09-30-11:03:27:INFO:RSStT.db - Successfully connected to the DB
2024-09-30-11:03:27:INFO:RSStT - RSS-to-Telegram-Bot (v2.9.0-5-g5a67570@dev, build@2024-09-17T15:29:40+00:00) started!
SELF: rss2tg Bot @chance_rss2tg_bot (5694024327)
MANAGER: 880551570
ERROR_LOGGING_CHAT: 880551570
T_PROXY (for Telegram): not set
R_PROXY (for RSS): not set
DATABASE: sqlite:/app/config/db.sqlite3
TELEGRAPH: Disable
UVLOOP: Enable
MULTIUSER: Enable
CPU: 1 (usable) / 1 (available) / 1 (total)
2024-09-30-11:03:57:INFO:RSStT.command - Allow blink (880551570) to use /list
2024-09-30-11:04:01:WARNING:RSStT.locks - Blocking any further messages for 861035684 due to flood control, 18912.00s left
2024-09-30-11:04:01:WARNING:RSStT.locks - Blocking any further messages for 1292505774 due to flood control, 18912.00s left
2024-09-30-11:04:02:INFO:RSStT.command - Allow blink (880551570) to use /list
2024-09-30-11:04:02:ERROR:RSStT.command - Uncaught error occurred when blink (880551570) attempting to use /list
Traceback (most recent call last):
  File "/app/src/command/utils.py", line 505, in wrapper
    await execute()
  File "/app/src/command/utils.py", line 368, in execute
    await asyncio.wait_for(
  File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
    return await fut
           ^^^^^^^^^
  File "/app/src/command/sub.py", line 204, in cmd_list_or_callback_get_list_page
    await (
  File "/opt/venv/lib/python3.12/site-packages/telethon/tl/custom/message.py", line 764, in respond
    return await self._client.send_message(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/telethon/client/messages.py", line 905, in send_message
    result = await self(request)
             ^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/telethon/client/users.py", line 30, in __call__
    return await self._call(self._sender, request, ordered=ordered)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/venv/lib/python3.12/site-packages/telethon/client/users.py", line 56, in _call
    raise errors.FloodWaitError(request=r, capture=diff)
telethon.errors.rpcerrorlist.FloodWaitError: A wait of 18910 seconds is required (caused by SendMessageRequest)
2024-09-30-11:04:02:WARNING:RSStT.locks - Blocking any further messages for 880551570 due to flood control, 18911.00s left
2024-09-30-11:04:09:WARNING:RSStT.locks - Blocking any further messages for 118069398 due to flood control, 18904.00s left
2024-09-30-11:04:19:WARNING:RSStT.locks - Blocking any further messages for -1001862440612 due to flood control, 18894.00s left
2024-09-30-11:04:26:WARNING:RSStT.locks - Blocking any further messages for -1001827528289 due to flood control, 18887.00s left
2024-09-30-11:04:32:WARNING:RSStT.locks - Blocking any further messages for 557382913 due to flood control, 18881.00s left
2024-09-30-11:04:52:WARNING:RSStT.locks - Blocking any further messages for -1001877989123 due to flood control, 18862.00s left
2024-09-30-11:06:27:WARNING:RSStT.locks - Blocking any further messages for 1754919950 due to flood control, 18766.00s left
@Rongronggg9
Copy link
Owner

Rongronggg9 commented Sep 30, 2024 •
edited
Loading

Messages from bots have a very low priority on the server side of Telegram (i.e. Telegram DCs). When a Telegram DC is under high server loads ("service degradation"), bot messages are dropped severely. As a result, the DC asks bots not to send more messages by returning FloodWaitError with an absurd delay.

We can do nothing but wait for the DC to recover from the service degradation.

Currently, DC5 is under service degradation and @RSStT_Bot is also suffering from the same error. Your bot also seems to belong to DC5.

@Rongronggg9 Rongronggg9 pinned this issue Sep 30, 2024
@HeroadZ
Copy link
Author

HeroadZ commented Oct 1, 2024

Thank you for detailed response!

Is it possible to solve this problem by recreating a bot of other DC(seems like assigned randomly)? This problem has existed for at least half year, I don't think it will be solved recently lol

@Rongronggg9
Copy link
Owner

Rongronggg9 commented Oct 1, 2024 •
edited
Loading

Is it possible to solve this problem by recreating a bot of other DC(seems like assigned randomly)?

AFAIK, a bot is created at the same DC as its creator, i.e., if your Telegram account belongs to DC5, all your bots will belong to DC5 too.

This problem has existed for at least half year, I don't think it will be solved recently lol

It has existed for several years. Generally, a service degradation will be resolved within a day, and @RSStT_Bot works all fine now.

@HeroadZ
Copy link
Author

HeroadZ commented Oct 2, 2024

Generally, a service degradation will be resolved within a day

But my bot won't respond to any command forever once the error occurred. What do I need to do to make sure it will be recovered?

@Rongronggg9
Copy link
Owner

What do I need to do to make sure it will be recovered?

Once the service degradation comes to an end, restart the bot.

@HeroadZ
Copy link
Author

HeroadZ commented Oct 2, 2024

restart the bot

Could you tell me the detailed steps to restart the bot? Do you mean restarting the docker container? Or restarting the telegram bot by some command within Telegram client?

@Rongronggg9
Copy link
Owner

restart RSStT, i.e., restart the docker container if it is deployed via docker.

@Rongronggg9 Rongronggg9 mentioned this issue Nov 11, 2024
Closed
@nageshuizeii
Copy link

能否参考此项目：https://github.com/iovxw/rssbot
我两个项目都部署了，设置的频率一样，但rssbot没有遇到FloodWaitError。
FloodWaitError导致RSS-to-Telegram-Bot其实已经不能使用了😭

@HeroadZ

This comment was marked as off-topic.

Sign in to view
@Rongronggg9
Copy link
Owner

能否参考此项目：https://github.com/iovxw/rssbot

@nageshuizeii
实际上 iovxw/rssbot 与 RSStT 唯一值得留意的不同只有：
前者使用 HTTP Bot API，后者使用 MTProto API。
实际上，HTTP Bot API 只是 MTProto API 的一个 wrapper，后者更接近底层。目前没有证据表明两者在被 rate limit 的概率上存在不同。
然而，RSStT 的一部分核心功能必须使用 MTProto API（HTTP Bot API 不提供对应的 method）。因此，将 RSStT 改为使用 HTTP Bot API 不是可行的。

我两个项目都部署了，设置的频率一样，但rssbot没有遇到FloodWaitError。
FloodWaitError导致RSS-to-Telegram-Bot其实已经不能使用了😭

我所知的所有先验知识都指出，一个从一开始就大得不合理的 flood wait time，只可能由 DC 服务降级引发，而不会由常规的 rate limit 引发。常规的 rate limit 所要求的 flood wait time 一开始很小，然后会逐步增长，但也不会增长到 DC 服务降级的时候的那种数量级，基本上不会大于四位数。实际上，在最严重的服务降级中，属于 DC5 的所有用户（不只是 bot）都会在进行任何操作时被要求一个大得不合理的 flood wait time（五位数或以上）。

目前所有的这类 issue 都有一个共同特征：bot 属于 DC5。本项目的公共 demo 也属于 DC5，且其上托管了非常多的订阅，发送消息的频次应该会比你的私有实例更频繁，然而公共 demo 却从未持续地遭遇过这样的问题，仅在 DC5 服务降级期间会一过性地以同样的症状被阻断几个小时。因此，我倾向于认为这个问题是罕有个例。更重要的是，因为我无法复现你所说的长时间内一直被要求不合理的 flood wait 延迟的情况，我也无法真正探寻其根本原因。

仅为毫无依据的猜测，为了搞清楚造成你的 RSStT 实例、公共实例、你的 iovxw/rssbot 实例三者之间的现象差异的原因，我有以下猜测或建议：

  1. Telegram 在服务降级时实施了 bot 评分，用户量少的 bot 更容易先被阻断服务以缓解 DC 的服务器压力。
  2. 官方的 HTTP Bot API 实际上在 DC4，不清楚它是如何管理位于 DC5 的 bot。但是，如果 Telegram 确实允许它部分地绕过 DC5 来工作，那就有可能能够绕过这个问题。
  3. 不明因素可能导致 Telegram shadow ban 了你的 bot session。你可以试着删除 config/bot.session 并重启 RSStT 以重新建立 session。如果还不行，试试在上一步之前 revoke 你的 bot token，删除 session 后用新的 token 重新启动 RSStT。
  4. 不明因素可能导致 Telegram shadow ban 了你的服务器 IP。但是 IP 匹配可能只实施在 MTProto API 上，而没有实施在 HTTP Bot API 上，因此，使用后者可能就绕过了 IP 匹配。你可以试着在别的服务器上部署 RSStT（提示：先停止原来的部署，并且不要使用同一个 session），看看有无不同。
  5. 不明因素可能导致 Telegram shadow ban 了你的 bot 本身（bot 账号）。那就只能重新申请一个 bot 来看看会不会有所不同了。你也可以试试将你用作运行 RSStT 与 iovxw/rssbot 的两个 bot 账号对调，看看情况会有什么不同。
  6. 由于这个问题目前只出现在 DC5，假使你有另外的 Telegram 账号且那个账号不属于 DC5，试着用那个账号申请一个 bot 来部署 RSStT，应该也可以帮助找出问题所在。
  7. HTTP Bot API 自身可能已经实现了 rate limit，因此，在触发 Telegram DC 的 rate limit 之前，HTTP Bot API 已经阻止了更多请求，从而防止了情况恶化。但我不是很相信这种可能性，因为 RSStT 本身就会在被要求初步的 flood wait 延迟的时候锁定对于目标用户的讯息发送尝试，本身已经可以阻止情况恶化。而且这也无法解释为什么目前只有极少量的 DC5 用户遇到了这个问题，更无法解释为什么你遇到的 FloodWaitError 从一开始就要求了五位数的等待时间。

@nageshuizeii
Copy link

3.尝试了，同样报错FloodWaitError，WARNING:RSStT.locks - Blocking any further messages for 【TGID】 due to flood control, 2379.00s left
4.部署在nas上，nas科学上网，变更过节点，同样报错FloodWaitError
5/6.尝试了，用其他账号的id和新的机器人bot，同样报错FloodWaitError，WARNING:RSStT.locks - Blocking any further messages for 【TGID】 due to flood control, 2379.00s left

@Rongronggg9
Copy link
Owner

4.部署在nas上，nas科学上网，变更过节点，同样报错FloodWaitError

绝大部分用作代理出口的 IP 都是容易辨识的（查询 ASN 可以知道 IP 归属于云服务商，且会有大量的用户同时使用该 IP 登录）。这很可能导致 IP 被 shadow ban。变更节点不改变这一事实。因此我认为很可能就是此原因。

5/6.尝试了，用其他账号的id和新的机器人bot，同样报错FloodWaitError，WARNING:RSStT.locks - Blocking any further messages for 【TGID】 due to flood control, 2379.00s left

这更加加重了 IP shadow ban 的嫌疑。

如果信任的话，你可以私底下将一个确定会引发该问题的 bot token 发给我（不要发在这里！），我看看能不能复现该问题。

还有一种可能性。RSStT 内置了数个网络上公开可知的 Telegram 客户端指纹，并在默认情况下随机选择一个用于 bot 的登录。由于对于 bot 来说，客户端指纹是不重要的，因此这并没有引起什么问题。但是，不清楚这是否也和我猜测的 IP shadow ban 相关。如果你兴趣的话，可以申请一个自己的客户端指纹并使用之。
https://github.com/Rongronggg9/RSS-to-Telegram-Bot/blob/dev/docs/advanced-settings.md#api-settings

@nageshuizeii
Copy link

已邮件反馈token，感谢！我再看一下指纹

@Rongronggg9
Copy link
Owner

已邮件反馈token，感谢！我再看一下指纹

测试了你的 token，可以正常使用，无法复现该问题。因此，可能确实为 IP 的问题。

在 Railway.app 上测试了，也没有问题，你可以自行测试。

@nageshuizeii
Copy link

观察了几天，如果是IP的问题的话，那么https://github.com/iovxw/rssbot应该是同样报错的，但是https://github.com/iovxw/rssbot一直都能正常使用。rssbot转RSS-to-Telegram-Bot就是因为本项目能够推送富媒体，是不是和富媒体有关系🥲

@Rongronggg9
Copy link
Owner

如果是IP的问题的话，那么 https://github.com/iovxw/rssbot 应该是同样报错的，但是 https://github.com/iovxw/rssbot 一直都能正常使用。

这个问题我已经回答过了：

实际上 iovxw/rssbot 与 RSStT 唯一值得留意的不同只有：
前者使用 HTTP Bot API，后者使用 MTProto API。
…但是 IP 匹配可能只实施在 MTProto API 上，而没有实施在 HTTP Bot API 上，因此，使用后者（HTTP Bot API）可能就绕过了 IP 匹配。

实际上，由于 HTTP Bot API 是开源的，可以知道，它本身就是个合法的 MTProto 客户端。尽管不清楚 Telegram 官方的 HTTP Bot API 是否有未知的独特的改动，但是可以合理猜测，在 MTProto API 层面，Telegram 服务器看到的是 HTTP Bot API 的托管服务器的 IP，而不是你的 IP。如果这个假设是真的，那么就可以绕过我所猜测的 IP shadow ban。

rssbot转RSS-to-Telegram-Bot就是因为本项目能够推送富媒体，是不是和富媒体有关系🥲

没有任何先验知识表明存在这样的关联性。况且，你先前在另一个 issue 中提供的 log 表明，纯文本消息也会被 FloodWaitError 阻断。

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Rongronggg9 @nageshuizeii @HeroadZ