Akamai Bot Manager

Manage and mitigate the impact of bot traffic with Akamai Bot Manager.

Akamai Bot Manager provides developers and admins with a flexible framework to manage interactions with bots. Bot Manager helps you identify all the bots that are accessing your site and categorizes them based on their business role or value. Bot Manager then provides advanced management actions that you can apply to any category of bots to generate the desired interaction.

How Bot Manager Works

Akamai Bot Manager gives you unparalleled bot protection via four powerful capabilities:

1. Edge deployment

Bot Manager runs on the Akamai Intelligent Edge Platform, a globally distributed content delivery network (CDN) comprising more than 240,000 servers worldwide. This means that Bot Manager is deployed at the edge, so you’re able to detect and mitigate bot traffic with Akamai edge servers rather than allowing that traffic to hit your origin. As a result, you get faster detection and mitigation capabilities, along with reduced stress on your origin infrastructure. In addition, Akamai edge servers allow for the automatic injection of JavaScript, which means you don’t need to worry about where to inject the JavaScript or making any configuration changes to do so. Finally, simplicity: if you're on the Akamai Intelligent Edge Platform, you can enable Bot Manager in just a few clicks.

2. Bot detection and categorization

Bot Manager enables you to take specific action on different types of bots based on your unique goals and requirements. To deliver this ability, Bot Manager must not only detect and identify different bots by name and type that are interacting with your website but also categorize different bot types in a manner that supports those requirements. Accordingly, Bot Manager provides a variety of methods to detect and identify bots, including known bot signatures, bot reputation, both active and transparent detections methods, and custom signatures. You can then assign the appropriate action to take on different types of bots, either through Akamai- or customer-defined bot categories, or based on the real-time bot detection method.

Below is an overview of five key pillars of Bot Manager’s detection and categorization capabilities (click to jump to any selection):

A. Akamai bot directory
B. Real-time bot detections
C. Web-scraper reputation
D. Machine learning
E. Custom bot signatures

a. Akamai bot directory

Bot Manager leverages Akamai’s exclusive visibility into global web traffic to maintain a bot directory containing signatures for more than 1,400 bots that commonly interact with Akamai customers. Bot Manager categorizes these bots into 17 predefined categories to help you manage the most commonly seen bots while minimizing any impact on performance. You can easily assign the desired management action on a per-category basis, which provides you with greater flexibility to manage different types of Akamai-defined bots in the most appropriate manner for your goals and requirements.

b. Real-time bot detections

Akamai-categorized bots provide a foundation for managing a broad spectrum of self-identifying bots employed by known third-party services. However, a variety of other external actors also employ bots for different purposes. Those with malicious intent or employed for less reputable purposes typically do not identify themselves to the website owner and instead attempt to either disguise themselves as human users or spoof other well-known legitimate bots. Bot Manager employs real-time bot detection capabilities to identify these unknown bots as they attempt to access protected websites, including:

• Behavior anomaly analysis: Collects telemetry from client input devices, such as mouse movements and keyboard strokes from a desktop, or gyroscope and accelerometer readings from a mobile device, to identify minute differences in behavior that distinguish between human and bot
• Browser fingerprinting: Collects identifying information about the client browser, such as screen resolution, browser plugins, and installed fonts. Bot Manager analyzes the browser fingerprint to identify automated or headless browsers and detect anomalies that indicate an automated bot
• HTTP anomaly detection: Employs a risk scoring model to inspect HTTP requests for patterns and anomalies that indicate they were generated by an automated bot attempting to disguise itself as a human user or legitimate bot, such as Googlebot
• Rate-based and session activity: Looks for differences in the behavior of a web client to that of human users, including the rate of requests, page access, and other behavioral indicators
• Workflow validation: Allows an organization to (i) define a workflow (for its website) that a human user would follow and (ii) take action on clients that deviate from the specified workflow

c. Web-scraper reputation

Bot Manager helps you automatically identify and manage web-scraping bots based on prior observed behavior against other Akamai customers. Akamai continuously monitors and tracks the behavior of malicious bots as they interact with websites operated by all Akamai customers applying machine learning to its Cloud Security Intelligence (CSI) big-data analysis engine. CSI then assigns a dynamic risk score (i.e., the bot’s “reputation”) based on a number of factors:

• Persistence: Frequency and length of time that the bot was observed to have performed web-scraping activities
• Distribution: Number of customer websites that the bot scraped
• Severity: Rate at which the bot was observed to have scraped targeted websites
• Magnitude: The number of attempts that the bot made to scrape targeted websites

Based on these dynamic risk scores maintained by Akamai, you can configure Bot Manager to automatically take action on any detected bots with specified reputation parameters. This allows you to simplify management of your bot traffic based on historical web-scraping reputation.

d. Machine learning

Bot Manager employs a variety of detection techniques—including pre-defined signatures, bot reputation, and real-time detections—to identify bots as they attempt to access protected websites. When Bot Manager identifies a request as generated by a bot, it sends details about that request to the Cloud Security Intelligence (CSI) big-data analysis engine. Within CSI, Akamai’s team of threat researchers refine the set of characteristics (features) that are used to identify bots, such as how quickly and for how long a client requests content, how many websites the client interacts with, where the client is coming from, and its responses to fingerprinting and other challenges.

Akamai employs machine learning to continuously analyze these large and dynamic datasets retained within CSI, identify bots interacting with Akamai customers, and detect anomalies in patterns that warrant further investigation. Bot Manager uses the output of this machine learning in several ways:

• Behavior anomaly analysis (available only in Bot Manager Premier—see Bot Manager Editions section below) employs machine learning to analyze the telemetry collected from client input devices and identify characteristics that accurately identify human behavior. The use of machine learning enables Bot Manager to detect minuscule differences in behavioral characteristics that distinguish between human and bot, even when bots attempt to mimic human behavior or replay previously validated telemetry.
• With web-scraper reputation, machine learning allows Bot Manager to not only identify a suspected bot but also provide a weighted risk score that reflects the characteristics that were used to identify it. The risk score reflects recent behavior that Akamai has observed in interactions with all Akamai customers, and will increase or decrease dynamically over time based on changes in observed behavior. You can tune your bot management policies using the reputational risk score to be more or less aggressive in order to achieve desired outcomes.
• Akamai’s Threat Research team investigates anomalies in the dataset flagged by the machine learning process. Where applicable, Akamai tunes the rules used by Bot Manager’s real-time detections to not only identify bots but also minimize false positives. Investigations into these anomalies help the Threat Research team refine existing rules and identify when new rules are required. Bot Manager automatically updates the rules used by its real-time detections whenever changes are made.

e. Custom bot signatures

Most organizations interact with specific, known bots as part of their normal business or IT functions. These can include:

• Internal bots: Automated bots created by IT or other internal groups to perform a repetitive interaction with their external websites
• Third-party services: External bots operated by third-parties contracted by the organization to perform a business or IT service
• Partners: External bots operated by business partners to stay informed of any relevant changes to the organization’s websites
• Malicious bots: External bots observed to be performing malicious activities against the organization’s websites

Bot Manager allows you to define custom bot categories in order to assign a desired management action to these bots. Within each category, you can create your own bot signatures based on multiple identifiers, including the IP address or subnet of the bot, AS number, cookie information, or specific information in the request header. Management actions for customer-defined categories override those for Akamai-defined categories, allowing you to provide differential treatment to known bots.

3. Management actions

Most traditional bot management solutions only provide the ability to alert on or block suspected bot traffic. Unfortunately, simply blocking all bot traffic impacts beneficial and harmful bots alike. Akamai Bot Manager gives you the flexibility to apply different management actions to different categories of bots in order to achieve superior business and IT outcomes.

There are two main categories of management actions:

A. Advanced actions
B. Conditional actions

Here’s an overview of each:

a. Advanced actions

Bot Manager provides a range of advanced actions to better manage the wide array of different bot types. You can specify the desired action to take for each existing bot category as well as for unknown bots. Bot Manager supports the following advanced management actions (in addition to basic actions that monitor or block traffic from identified bots):

• Tarpit: Stops requests from identified bots without responding to the request with a 403 Forbidden HTTP status code and without alerting the bot that it was blocked
• Silent deny: Inserts a delay equivalent to the HTTP timeout as the bot waits for a response from the website
• Delay: Delays requests from identified bots to reduce the rate at which they can extract content or limit their impact on the origin infrastructure. Delay offers an intermediate response that allows all requests to proceed normally after inserting a 1-3 second delay. Delay can be an appropriate action to take on bots that perform acceptable business functions but consume a disproportionate amount of origin resources
• Slow: Inserts an 8-10 second delay before responding to requests from identified bots. The Slow action is similar to Delay but allows a greater reduction in the rate at which a bot can extract content from the site as well as the origin load
• Serve alternate content: Serves identified bots with different designated content and prevent content theft. With the Serve Alternate Content action, organizations can create an alternate web page that resembles the original—but differing in sensitive information, such as price or inventory—and mislead the bot into believing that it has retrieved its targeted content
• Serve alternate origin: Directs web traffic from identified bots to an alternate origin location. Organizations can select this option to reduce the load on the primary origin infrastructure from bot traffic and maintain higher performance for real users. In addition, the Serve Alternate Origin action can also be used to serve bots with alternate content located at the alternate origin
• Serve cached: Instructs Bot Manager to always respond to requests from identified bots with cached content. Served Cached responds to bots with legitimate content but minimizes bot traffic reaching the origin as well as the performance impact on the origin infrastructure
• Signal origin: Adds information about the detected bot into the request header for organizations to take appropriate action at the origin

b. Conditional actions

In addition to the advanced actions noted above, you can also conditionally assign multiple actions to a bot category. Conditional actions give you the flexibility to adjust for changing requirements and allow you to introduce an element of unpredictability into the bot response to slow down the rate of bot evolution. Bot Manager supports the following conditional management actions:

• URL: Assigns a conditional action based on the hostname, path, or query parameter in order to take differential action on bot traffic to URLs that require additional flexibility or face different bot-related challenges
• Time: Allows different actions to be applied to bot traffic based on the time of day. For example, organizations suffering performance degradation from partners or other types of good bots can apply a Delay or Slow action to bot traffic during business hours to better manage the traffic impact
• Percentage of traffic: Takes different actions on identified bots based on percentage of traffic. For example, organizations with malicious bot traffic can allow a small percentage of traffic to access their website in order to minimize the negative impacts without alerting the bot operator

4. Visibility and reporting

In addition to categorizing and taking action on identified bots, Bot Manager gives you real-time visibility into the number and types of bots that are accessing your websites. Bot Manager integrates visualization and reporting of bot traffic into Akamai Security Center within Akamai’s Luna Control Center interface. Security Center displays overall bot traffic statistics along with those of other types of attack traffic, providing a simplified view of your organization’s overall security posture. Bot Manager’s visibility and reporting capabilities are based on five key features (click to jump to any selection):

A. Bot trends report
B. Web security analysis report
C. Bot endpoint protection report
D. Origin signaling
E. Event logging

Here’s an overview of each:

a. Bot trends report

Bot Manager’s bot trends report gives developers and administrators a high-level view into the various characteristics of bot traffic and how those characteristics change over time. The bot trends report gives you visibility into the different types of bots accessing your site. You can easily expand the date range to see long-term trends or drill down into smaller time frames to examine bot patterns around specific events. Statistical information provided by the report includes:

• Overall bot traffic: Percentages of edge hits, page views, and bandwidth attributed to identified bots over the selected date range
• Type of bots detected: Percentages of Akamai-categorized, customer-categorized, and unknown bots identified over the selected date range
• Action applied: Breakdown of management actions taken over the selected date range
• Bot trending: Visual representation of bot traffic compared to overall site traffic over the selected date range
• Bot categories: Breakdown of bots by Akamai- and customer-defined categories and detection method over the selected date range
• Triggered policies: Visual representation of bot traffic broken down by your website security policies
• Targeted hostnames: Breakdown of bot traffic to various protected hostnames over the selected date range
• Countries of origin: Breakdown of bot traffic by country of origin over the selected date range

b. Web security analysis report

The web security analysis report provides a detailed and granular analysis of your bot traffic. Developers and admins can easily view bot traffic across multiple dimensions over a specified or pre-defined date range, including:

• Botnet ID: Breakdown of bot traffic by botnet; the bot analysis report displays the bot operator for self-declared bots and a unique Akamai identifier for Akamai-identified botnets
• IP address: Breakdown of bot traffic by the IP address of individual bots
• IP subnet: Breakdown of bot traffic by the IP subnet of individual bots
• Country: Displays the countries from which bot traffic originated over the selected date range
• AS number: Displays the ASN from which bot traffic originated over the selected date range
• URL: Breakdown of bot traffic by the targeted URL

The Web Security Analysis report helps you better understand the behavior of individual bots and botnets, the impact of bot traffic on specific web properties, and the effectiveness of your configured management actions against different categories of bots. In addition, with the Web Security Analysis report you can view sampled logs for filtered or unfiltered bot traffic to further analyze bot traffic at the request level and formulate an appropriate strategy to manage it—such as creating new bot signatures and categories for identified bots.

c. Bot endpoint protection report

The bot endpoint protection report (available only in Bot Manager Premier—see Bot Manager Editions section below) helps you focus on the detection and management of bot activity in connection with specific, protected URLs. For the critical parts of your site, you can easily monitor human vs. bot activity and gain insights into the breadth of attacks as well as whether the attacks were alerted or mitigated. Additionally, for any given attack period, you can quickly identify the key attack characteristics. The report also provides key details like origin of the attack (top countries and AS Numbers) and characteristics of the attacker (top IP address, user agents, and botnet IDs). In summary, Akamai built this report to help save you time in understanding what your critical endpoints are experiencing. For further details, see the “Introducing the Bot Endpoint Protection Report” blog post.

d. Origin signaling

Bot Manager can further enhance your reporting capabilities with origin signaling. Origin signaling inserts identifying information into the HTTP header of any detected bot request before forwarding it to the application origin. This can help you in two ways:

• You can better report on—and gain improved insights into—your own internal marketing data. Website and page view statistics can be significantly skewed by bot traffic, which in turn skews your marketing data, making it difficult to understand the behavior of real users interacting with your site. By identifying requests generated by bots, Bot Manager can help you exclude bot traffic from your analysis.
• You can deploy Bot Manager alongside existing solutions or capabilities. In some cases, you may want to take action on identified bot traffic at the origin using a different mechanism or process. By signaling the origin of a bot-generated request, you can easily apply differential treatment to any identified bot traffic.

e. Event logging

Developers can integrate Bot Manager into existing reporting solutions through Akamai’s Log Delivery Service (LDS). LDS provides you with event logs generated from Bot Manager, including events from Akamai edge servers that process bot traffic. LDS delivers log lines on a predetermined schedule—typically within 24 hours. However, due to the globally distributed nature of Akamai’s intelligent edge platform, some log lines may be delayed and be part of a later delivery.

If you require real-time integration of security event data, you can choose Bot Manager’s optional SIEM Integration (see below).
 

Additional Options

Bot Manager can be extended and customized with these additional options:

SIEM Integration

Akamai SIEM Integration allows you to integrate Bot Manager into your existing security information and event management (SIEM) infrastructure. This enables you to correlate events from Bot Manager along with those from other security solutions for broader visibility across your entire security infrastructure. You can pull security events data directly from the Akamai platform in real time through an OPEN API. SIEM Integration includes connectors for Splunk, HP ArcSight, and IBM QRadar out of the box; you can also custom-develop connectors for other SIEM solutions.

Site Shield

Akamai Site Shield provides an additional layer of protection that helps to prevent bots from bypassing the cloud-based protections of Bot Manager and directly accessing the application origin. Site Shield gives you a list of IP addresses for Akamai edge servers that are allowed to communicate with the application origin, and through which Akamai will direct all traffic to the protected website or web application. You can then allowlist the Site Shield servers and block all other incoming connections on ports 80 and 443, either at the network firewall or by working with your Internet service provider (ISP). By restricting clients from directly accessing the origin, Site Shield forces bot traffic to go through Bot Manager for optimal bot management.
 

Bot Manager Editions

There are two available editions of Akamai Bot Manager, both based on the same bot management framework but with some important differences:

Bot Manager Premier

Bot Manager Premier includes the latest bot detection capabilities suitable for detecting the most sophisticated bots commonly seen committing credential abuse, engaging in web fraud, and other critical use cases. You can also pair Bot Manager Premier with the Bot Manager Premier SDK, which takes the fundamental technology of Bot Manager Premier and applies it to your native mobile apps.

Bot Manager Standard

Bot Manager Standard provides bot detection capabilities suitable for addressing a wide range of general use cases, including web scraping, content aggregation, and more.