Managing bot traffic

Web resources process a significant portion of bot traffic. Some of these bots are legitimate and essential for the operation of various web services. Among such bots, there are web crawlers tasked with content indexing to ensure search engines return relevant results, web page archiving bots, SEO optimization bots, etc. However, there is also a large number of malicious bots, e.g., DDoS botnets, vulnerability scanners, or bots used for fraud and data theft. To protect a web service, one has to carefully set up traffic filtering so as not to get in the way of helpful bots while minimizing the activity of malicious ones.

Smart Web Security offers flexible protection and bot traffic filtering tools:

• List of verified bot categories by intended use or type of activity (AcademicResearchBot, AISearchBot).
• Up-to-date lists of legitimate bots used by various services and companies (e.g., Yandex, Googlebot, or Bing).
• Specific attribute to distinguish a verified bot.
• Configurable bot score thresholds from 0 to 100 for rule customization.

You can use the above traffic conditions in security profile (basic, Smart Protection, or WAF) and ARL profile rules.

To further fine-tune your security rules, check logs for traffic filtering results and bot scores.

Verified bot categoriesVerified bot categories

• AcademicResearchBot: Collect web data used in academic research; these bots help automate gathering information for analysis and scientific study.
• AccessibilityBot: Improve web content accessibility for users with special needs, e.g., bots that help read out text on websites.
• AdvertisingAndMarketingBot: Support advertising and marketing campaigns.
• AggregatorBot: Gather and distribute information, e.g., bots for aggregating articles from news websites.
• AIAssistantBot: AI-based assistants involved in a wide range of tasks.
• AICrawlerBot: Help train and improve AI models, e.g., bots that collect data for training machine learning algorithms.
• AISearchBot: AI-driven bots optimized for interactive search and delivering information in response to user queries.
• ArchiverBot: Capture and archive snapshots of web pages and other web resources.
• FeedFetcherBot: Fetch data from various regularly updated sources, e.g., news feeds and blogs, helping users receive up-to-date information.
• MonitoringAndAnalyticsBot: Collect website analytics. These bots monitor website metrics, e.g., traffic, load speed, or error rates, to provide analytical data to improve site performance.
• PagePreviewBot: Generate page previews for links shared by users in messenger apps or social media.
• SearchEngineCrawlerBot: Scan the internet and index pages for search engines. They help systems, such as Yandex, update their databases and deliver up-to-date search results.
• SearchOptimizationBot: Help optimize websites and improve their visibility in search engines by analyzing pages for errors and suggesting changes to page content and structure.
• SecurityBot: Check websites for vulnerabilities and various security threats.
• SocialMediaMarketingBot: Help manage brand presence on social platforms by automating posting, moderation, and replies to users as well as collecting analytics for SMM campaigns and assessing their performance.
• WebhooksBot: Automate processes using real-time technologies that enable web applications to interact with each other.
• OthersBot: Other categories of verified bots.

List of legitimate botsList of legitimate bots

Below is a list of bots used by various businesses for content indexing, delivering information in response to user queries, SEO, and AI model training. The list is updated regularly. You can use this list to configure traffic conditions tailored to your particular service or application.

Bot scoreBot score

To configure custom rules for your traffic, you can define filtering rules based on bot score.

Smart Web Security assigns traffic a score from 0 (lowest probability, i.e., human) to 100 (highest probability, i.e., bot).

The following threshold ranges are used in decision-making:

• up to 20: Human
• 20–40: Likely human
• 40–60: Undetermined
• 60–80: Likely bot
• over 80: Bot

In request filtering conditions, specify threshold values using the >=, <=, =, or != operators and the AND logical operator.

For example, =10, >=20 AND <=40.

For final tuning, apply the rule in only logging mode and analyze logs to determine the optimal threshold.