Skip to content

goodBots DNS lookup is async, causing bots to be challenged on first request #81

@pavelforever

Description

@pavelforever

Problem

When rateLimit: 0 is set, the plugin challenges every new IP immediately before performing the reverse DNS lookup to check if it's a good bot.

This means that even correctly configured good bots (e.g. goodBots: yandex.com) get served a challenge page on their first request, because the DNS lookup happens asynchronously and the decision to challenge is made before the result is available.

Example

Config:
goodBots: google.com,yandex.ru,yandex.net,yandex.com

YandexBot IP 5.255.231.189 has PTR record 5-255-231-189.spider.yandex.com — which should match yandex.com in goodBots. However the bot still receives a challenge page because the DNS lookup hasn't completed yet when the request is processed.

Verified with:
host 5.255.231.189
→ 5-255-231-189.spider.yandex.com

Expected behavior

If a bot's IP resolves to a domain matching goodBots, it should never be challenged — including on the first request.

Suggested fix

Either:

  1. Perform DNS lookup synchronously before deciding whether to challenge (with a short timeout)
  2. Cache verified bot IPs after first DNS lookup so subsequent requests from the same IP are immediately exempt
  3. Add a config option like goodBotsWaitForDns: true to allow users to opt into synchronous lookup

Workaround

Currently the only reliable workaround is to add all known bot IP ranges to exemptIps, which requires manual maintenance as bot IPs change frequently.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions