Skip to main content

Known Bots

This policy group functions uniquely in WAAP. Instead of challenging or blocking incoming traffic like other policy groups, it's designed to recognize and allow traffic from legitimate automated services.

These services include:

  • Common web crawlers like Google and Microsoft bots
  • Preview fetch bots like Slack, Facebook, or Twitter
  • SEO and monitoring tools
  • Other tools like Instant Payment Notification (IPN) requests from PayPal or HiPay

All these automation services often access your domain to gather some information and are considered beneficial or necessary for normal web activities. Note that some policies are set to Policy-based mode by default, and you can set them to Allow to fully permit traffic from the relevant services.

In some cases, you might want to restrict automation tools from accessing your domain. For instance, you can do so to enforce specific data protection requirements on a domain, protect it from potentially harmful services, or restrict paid content from unauthorized use. In this case, adjust the mode for particular bots in the Known Bots section.

Info

This policy group is available in the Pro and Enterprise plans.

Configure Known Bots

You can review and configure relevant bots in the CDB Technical Web Portal:

  1. Navigate to WAAP > Bot Management.

  2. In the domain dropdown at the top of the page, select the needed domain.

  3. Click the Known Bots tab to view and adjust the list of allowed bots.