If you experience long search times, search timeouts, or increases in search times, you should try the tests listed here to see if you can accelerate searches.
- Obtain a disk I/O benchmarking utility.
Microsoft™ SQLIO is a useful benchmarking utility. For more information, search the Microsoft website for "SQLIO Disk Subsystem Benchmark Tool".
- Stop all Experience Analytics services.
- Run SQLIO against the Experience Analytics disk(s).
- You should observe transfer rates of higher than 7 megabytes per second. If you are experiencing slower transfer rates, there are issues with your storage device(s). Consult the appropriate administrator for details and further diagnostics.
Another area to check is the size of the indexes that are compared to the LSSN files in the
Long-Term Canister. In the Portal, navigate to System Status > Storage. Select an individual server from the CX drop-down.
- If the index sizes are larger than the size of the LTC files, then Experience Analytics may be indexing unnecessary data.
In several captured hits, you should examine the contents of the request and the response to
ensure that all data that is being indexed is needed. Remove from indexing any unneeded data.
You can create privacy rules to strip out unwanted content so it is not indexed.
Unable to perform searches of completed sessions
If you are performing a search of completed sessions from the Portal, you might receive an error indicating that there are no indexes to search.
You can perform the following steps to troubleshoot the issue.
- Try searching through RTV. If you are able to search using RTV, then the Search Server is operational.
- You may want to examine the state of individual indexes in the Search Server. To do so,
you must configure RTV to use the same Search Server.
- If you are able to successfully connect to the same Search Server, the list of available indexes is displayed beneath the server node in the Search Setup window.
- To review a summary of the indexes for the Search Server, select the server node and click Summary.
- In the Summary page that is generated in your browser, you can review the status of each index that is managed by the search server. Review the From and To timestamps and the values for Index, Size, and Session Indexes. If these fields have meaningful values, you should be able to search for session data between the From and To dates.
- If Search Server appears to be operating properly through RTV, the issue may be with the
Data Service. The Portal acquires its set of available indexes from the Data Service. Complete the
following steps to verify that the Data Service is operational.
- In the Portal, navigate Portal Management.
- Click the Manage Servers link.
- Select the Data Service server.
- If the Data Service server does not exist in the list of servers, click the Display Inactive check box. If the server now appears, you must activate it.
- If the Data Service server still fails to appear, you must create an entry for it.
- When the Data Service server is selected, in the toolbar above the list of servers, click the Ping tool in the toolbar. Version and request information should be displayed in the Data Service Server Status panel below the list of servers.
- If the Data Service server does not respond, try restarting the Data Service. Note: While the Data Service is restarting, the Portal is unable to connect to other Search Servers and the Experience Analytics databases. Typically, restart takes only a few seconds.
- In the Portal, navigate to TMS.
- In TMS, select the WorldView tab.
- From the View drop-down, select Servers.
- Click the Data Service node.
- In the Server Actions, click Restart.
- If the restart does not resolve the issue, the problem may be related to any of the
- Inadequate permissions
- Special configuration for the Portal in IIS or ASP.NET required to connect to Search Server must also be applied to the Data Service.
- Use of raw IP addresses in the server addresses of the Portal Management page. Use server names instead.
- Proxy server between the client browser and the Portal. Disable the proxy if possible or configure it to manually connect to the Search Server.
Status Code 400 errors
If completed searches are failing with Status Code 400 errors, you might have a poorly specified path value in Search Server configuration.
The temp path must be a fully qualified path. If the path is not fully qualified, the path is treated as a relative path, and the resulting behavior depends on the current drive directory:
- Search server might refuse to start, and an event log error might indicate a problem with the temp path.
- Search server might return Status Code 400 errors on commands that use the temp drive, such as searching or retrieving the list of indexes.
The fix is to use full path names for values in Search Server configuration.
Using the "All text" search to examine the request and the response
An "all text" search examines the request and the response for every term that was indexed.
The all text search of the response includes every word that is considered "meaningful".
In a request, specific items are included by default.
- To see approximately what is indexed and thus searchable in a response, open the CX RealiTea Viewer and examine the response view of a typical page.
- Right-click the response view
- Select the Indexed view or use the drop-down menu on the right side of the RSP button on the toolbar.
- This view indicates the words for which you can search in the response to find a specific page.
Can't include the word NOT in a search
In the dtSearch Indexing engine, the word
NOT is a reserved operator for
Boolean expressions used to construct complex search terms.
Experience Analytics uses two different search engines:
- dtSearch Indexer is the more powerful one and is invoked when searching completed sessions through RTV or the Portal.
- string comparer is a simple string comparison search facility, which is invoked when you search for active sessions.
NOT is not indexed, which means it cannot be searched for. Likewise,
hyphens and other punctuation characters are not indexed, as they are considered word
You want to search for the URL
slp-system-not-available.html. Entering this
string results in few if any matches. From Portal Search, search instead for
available", including the double quotation marks.
html was left off at the end of the search to widen the net on the
first search. If you find the three-word search is returning incorrect hits, then you can add the
html to the phrase to narrow the results set.
Can't search on fields with ~ or other punctuation characters in the field names
The Session Indexer service breaks words apart on special characters as defined in
name=value fields would be broken up if
they contained any names with special characters, making it impossible to search for the
To avoid this situation, the Indexer replaces any special punctuation characters in the name
(left of the
= sign) with underscores (
_). For example, to search
~cc_num, enter the field name as
Session fragments and search results
Sessions can become fragmented under multiple conditions. Fragmentation affects searching and some event triggers.
Suppose the web pages for a business process consist of the
add an item,
start checkout, and
complete checkout events. Assume that the
website allows for a long application timeout and that the Experience Analytics STC session
timeout is configured to be less than the application timeout.
This discrepancy is often necessary, as the Short-Term Canister usage directly affects RAM consumption in the Experience Analytics server.
A visitor to the site could add an item, start checkout, go have dinner, and return to finish the
checkout an hour later. This behavior results in one session fragment containing the first two
events, and a second, later session fragment containing the final
Experience Analytics search looks for session fragments. A search for
and 2 and not event 3 would find session fragment 1 and not fragment 2. When replaying
session fragment 1, the user can select
Find all fragments of this session, which
replays a single logical session consisting of the contents of both session fragments. The merged
session has all three events, which are not what the original search requested.
For most sites, it is difficult to completely avoid fragmentation. To reduce fragmentation, you can increase the duration of the STC timeout, which may impact STC RAM consumption. Over-consumption of the STC RAM leads to spooling, which may cause fragmentation. Spooling should be avoided wherever possible. You can experiment with gradually increasing the STC timeout until fragmentation is reduced to a tolerable level without increasing spooling.
Another approach is to find a representative sample of the fragmented sessions, analyze them for the time duration that contains all events of interest, and set the STC timeout such that 95% or other desired percentage of the user sessions are not fragmented for the events of interest.
How to tell if the disks are I/O limited
Suboptimal disk layout and allocation can lead to poor performance of the Experience Analytics components. Use the information in this topic to determine if your disks are creating a performance bottleneck.
This solution explains how to determine if your disks are creating a performance bottleneck:
- Run the Windows™ PerfMon utility.
- If the
Average Disk Queue Lengthcounter is not already displayed, right-click the right pane and select Add counters.
- In the Performance Object drop-down, select Physical Disk.
- In the rightmost set of radio buttons, select All instances.
- In the left pair of radio buttons, select Select counters from list.
- Select the following three counters from the list box:
Avg Disk Queue Length
Avg Disk Read Queue Length
Avg Disk Write Queue Length
- Click Add.
- Look at the results in the perfmon right pane. Focus on the disks
that house the CANISTER, dbs directory, and the Indexes directory.
- Average queue lengths should be less than 1.
- Queue lengths of 4 or greater mean the disk is I/O bound.
Average queue lengths of 4 or more are a strong indication that the system's disk drives need repartitioning along the guidelines of Solution 65.
If your system is configured according to Solution 65 standards, does not use RAID, and has high average queue lengths, the solution is to move to a faster disk subsystem or add additional Experience Analytics servers. The current system does not have the disk I/O bandwidth to handle the data processing load that is presented to it.
Repeated restarts after dtSearch hang
When running searches you might receive entries in the application event log indicating multiple restarts.
For example, you might receive the following entries in the application event log:
(18:49 Search Server) - TeaLeaf Search Server Ver: 126.96.36.19950 - RestartOnDtSearchHang (18:01 Search Server) - TeaLeaf Search Server Ver: 188.8.131.5250 - RestartOnDtSearchHang (16:21 Search Server) - TeaLeaf Search Server Ver: 184.108.40.20650 - RestartOnDtSearchHang
Search Server is unable to complete the requested search due to disk performance issues or insufficient time to complete the search.
This problem may be the result of performance issues with the disk hardware. Perform the following checks:
- Run the Windows PerfMon utility.
- Check the average disk queue length of the disk where the Experience Analytics
Indexes directory resides. If this value is typically larger than
1, the disk cannot keep up with I/O requests.
- Check the average disk queue length of the disk where the Experience Analytics Indexes directory resides. If this value is typically larger than
- Use Microsoft's SQLIO benchmark utility to
assess the disk speed.
If the disk's throughput is less than 50 megabytes/sec, it is not fast enough for the processing volume that is required by the Experience Analytics server. Consult with your IT department for ways to improve the disk I/O performance.
If the disk hardware performance cannot be improved, change the Search Server Watchdog Timeout setting:
- In the Portal, navigate to TMS.
- From the View drop-down, select
- Click the Search Server node.
- Click Search Server configuration.
- Click View/Edit (raw).
- In the window, click the Default group.
- Change the value for
WatchDogIndexSearchSecondsto double or quadruple the current value.
If necessary, you can set the value to
0to disable this feature entirely.
- Click Save.
- Assign and push the configuration.
- To apply the new value, restart Search Server.