Most commercial enterprise web applications are subjected to significant traffic from web crawling bots. A bot is a software agent that navigates websites to extract information about them. Some bots are used by search engines for indexing pages.
Before you begin, you must configure Tealeaf to detect user agents.
Through its user agent detection capabilities, Tealeaf enables you to closely monitor activities on your web application that are identified as bot traffic.
The number of mobile web visitors continues to expand at a rapid rate. Depending on your web application, these visitors might have special requirements and customer issues. The ability to track their activities independent of desktop and other types of visitors is essential in providing good service to this market.
The CX Mobile module enables insightful tracking and useful monitoring of mobile visitors and their devices.
A bot is a program that is designed to crawl pages of a website. A bot can be used for indexing a website for search engines, accumulating price information, and other organizational purposes. Tealeaf enables the monitoring of bot activities on your website. This type of activity can be tracked easily through RTV and Portal searches.
Valid bots identify themselves as user agents of bot type and can be tracked through Tealeaf. For example, when bots are detected, you can identify the pages on your site that have been crawled by specific search engine bots such as googlebot, msnbot, and more.
Self-reporting bots can be identified through the user-agent string value in the request. Tealeaf can match the values reported in this field to the values in the UserAgent field in the public standard Browscap.csv. This user agent information is posted to the [ExtendedUserAgent] area of the request.
Optionally, you can configure the Tealeaf Reference Session Agent to overload the TLT_APPLICATION_NAME with the browser type extracted from the user agent cache.
Complete the following steps.
- You must verify installation of the appropriate user agent detection files. For best results, you should update your local copies of these files from the available public standards.
- After you have acquired and installed the user agent files, you need to enable user agent detection through the Tealeaf Reference session agent, which must be included in your Windows™ processing pipeline.
When the above steps are completed, Tealeaf can perform the basic detection of user agents, including bots, utilizing the public standards.
Masked bots
Some bots do not report themselves as legitimate bots. These bots may attempt to crawl your application for illegitimate purposes or competitive advantages. Tealeaf does not provide direct mechanisms for identifying masked bot activity.
All known and self-reporting bots are maintained in Browscap.csv, which Tealeaf uses to identify user agents.
If you detect a user agent that is not properly captured by Tealeaf, you can add it to UserCap.csv.
Searching for bot sessions in RTV
In the CX RealiTea Viewer, you can search for bot traffic.
- In RTV, click Search in the RTV toolbar.
- At the bottom of the Search Builder, events available for search are listed as folders, with report groups and dimensions listed beneath them. Find an event that occurs in every session and contains the Traffic Type dimension.
Note: You may have to create this event with
Traffic Typeas an included dimension. - Select the
Traffic Typedimension. - In the right side of the panel, for the dimensional value, enter BOT.
- Specify the other parameters of your search.
- To begin the search, click Search.
Searching for bot sessions through the Portal
Through the Portal, you can search for and replay sessions from bots.
- In the Portal, select Search > Completed Sessions.
- In the Search Fields panel, remove all default options.
- In the Basic Search Fields panel, click Event.
- In the Event Values area, click Select an event. Select an event that contains the Traffic Type dimension.
- In the Event Values area, click Any Dimension. Select
Traffic Type. - In the Search Scope drop-down, select
OR - Any Session. - Specify the other parameters of your search.
- To begin the search, click Search.
To replay a found search, click the results. Then, click the Replay icon for a specific session. To replay through Browser-Based Replay, click Browser.
Reporting on bots
For most reporting situations, you can use the Traffic Type dimension to segment your report to display only bot-related activities.
- You may add this dimension to your X-axis to show bot traffic side-by-side with other types of traffic
- You may add this dimension to the Y-axis to create stacked charts showing how bot traffic compares to other types of traffic in metric calculation.
- You may add this dimension as a Segment to show bot traffic only.