Problem
When rateLimit: 0 is set, the plugin challenges every new IP immediately before performing the reverse DNS lookup to check if it's a good bot.
This means that even correctly configured good bots (e.g. goodBots: yandex.com) get served a challenge page on their first request, because the DNS lookup happens asynchronously and the decision to challenge is made before the result is available.
Example
Config:
goodBots: google.com,yandex.ru,yandex.net,yandex.com
YandexBot IP 5.255.231.189 has PTR record 5-255-231-189.spider.yandex.com — which should match yandex.com in goodBots. However the bot still receives a challenge page because the DNS lookup hasn't completed yet when the request is processed.
Verified with:
host 5.255.231.189
→ 5-255-231-189.spider.yandex.com
Expected behavior
If a bot's IP resolves to a domain matching goodBots, it should never be challenged — including on the first request.
Suggested fix
Either:
- Perform DNS lookup synchronously before deciding whether to challenge (with a short timeout)
- Cache verified bot IPs after first DNS lookup so subsequent requests from the same IP are immediately exempt
- Add a config option like
goodBotsWaitForDns: true to allow users to opt into synchronous lookup
Workaround
Currently the only reliable workaround is to add all known bot IP ranges to exemptIps, which requires manual maintenance as bot IPs change frequently.
Problem
When
rateLimit: 0is set, the plugin challenges every new IP immediately before performing the reverse DNS lookup to check if it's a good bot.This means that even correctly configured good bots (e.g.
goodBots: yandex.com) get served a challenge page on their first request, because the DNS lookup happens asynchronously and the decision to challenge is made before the result is available.Example
Config:
goodBots: google.com,yandex.ru,yandex.net,yandex.com
YandexBot IP
5.255.231.189has PTR record5-255-231-189.spider.yandex.com— which should matchyandex.comingoodBots. However the bot still receives a challenge page because the DNS lookup hasn't completed yet when the request is processed.Verified with:
host 5.255.231.189
→ 5-255-231-189.spider.yandex.com
Expected behavior
If a bot's IP resolves to a domain matching
goodBots, it should never be challenged — including on the first request.Suggested fix
Either:
goodBotsWaitForDns: trueto allow users to opt into synchronous lookupWorkaround
Currently the only reliable workaround is to add all known bot IP ranges to
exemptIps, which requires manual maintenance as bot IPs change frequently.