There is a delay processing data acquired after <timestamp>. Reports, alerts and indexes can not have complete data until the processing completes.

• Severity: Major to Minor.
• Cause: The most likely cause is that one or more of your Canisters is spooling data. The appearance of this warning message in the Portal is enabled by a setting.

  If a Canister is unable to process incoming hits in a timely manner, the system begins writing those hits to the hard disk, from which it pulls hits when more processing resources become available.

• What to do: If the spooling message disappears within an hour, it can temporary surge in traffic volume. If the message redisplayed consistently or displayed for significant periods during a workday, the conditions that lead to spooling must be investigated further.
  You can use Health-based Routing (HBR) to manage load balancing between multiple Canisters. You can verify if the system is spooling by checking to see if the DecoupleEx session agent, which sits at the top of each pipeline, is spooling hits to disk.

  1. In the Portal, navigate to TMS.
  2. Click the Pipeline Status tab.
  3. Select a Windows™ pipeline.
  4. Find the DecoupleEx row. Review the value in the Queued column.
  5. If this value is not zero, the pipeline is spooling hits to disk.
  6. Perform the check for each Windows pipeline in your system.

  Additional information is available in the DecoupleEx report and the Status report.

  The root issue is that the Canister is unable to process hits as fast as they are coming in, due to any of the following factors:

  • Spikes in volume - Depending on your web application, there can be periodic spikes in volume due to expected or unexpected changes in site traffic. For example, if a marketing campaign is released, which directs traffic to the site, there can be a spike in traffic soon after. If the only effect on the Canister is a temporary spooling of hits due to these spikes, then the issue can not be significant.
  • Other processes taking up Processing Server resources - If other processes are consuming system resources on the server hosting the Processing Server, overall throughput of hits can be impacted.
    • Wherever possible, you must minimize the other applications or services in use on the Processing Server.
    • Periodic maintenance tasks must be scheduled during off-peak hours for all Tealeaf servers.
  • Poorly designed events - Events that perform inefficient searches, use regular expressions too liberally, or fire too frequently can affect Canister throughput.
  • Poorly designed pipelines - The Windows pipeline is intended to be a sequence of efficient, high-performance processing agents that drop unnecessary data and change the captured data to meet the search, reporting, and replay requirements of your enterprise. It is a good idea to periodically review the configuration of a spooling pipeline to verify the following conditions:
    • All included session agents are required.
    • Data that is not necessary is removed early in the pipeline, so that it is not processed needlessly by other session agents.
    • If the Privacy session agent is deployed, verify that all privacy rules are needed and are efficiently designed. Poor design of regular expressions in privacy rules can have a large impact on Canister throughput.
  • Insufficient hardware - If your traffic volumes are changed significantly, the hosting system can be inadequate to handle the change.
    • For more information about the recommended sizing, contact Professional Services.

Report groups on one or more events have been disabled because they exceeded the safety limit of 500,000 new occurrences within a single hour.

Report groups on one or more events are disabled because they exceeded the safety limit of 500,000 new occurrences within a single hour. Make the necessary changes to the report group and/or event to ensure that the safety limit is not exceeded before manually reactivating the report group.

• Severity: Minor to Major.
• Cause: This error message indicates that the number of instances that are recorded of one or more report groups exceeded 500,000 within a single hour. Tealeaf automatically disables the report group so that the database does not become burdened with data. The appearance of this warning message in the Portal is enabled by a setting.
• What to do: Typically, this situation is caused by a poorly constructed event. In many cases, the event is configured to be recorded on every hit, which means that the event value and all related dimensional values are being recorded on each detected hit.
  Note: In some cases, the source of the problem can be constrained to a single dimension within the report group. To reduce the number of recorded instances for the dimension, you might be able to constrain the dimension to use a whitelist of recorded values.

To locate the sources of these messages, complete the following steps:

1. In the Portal menu, select Configure > Event Manager.
2. Click the Events tab.
3. From the filter drop-down at the top of the page, select Events with Disabled Report Groups.
4. The list of events that have one or more disabled report groups is displayed.
5. If possible, you must change the evaluation of the item to a different trigger. If the event data is available on the first hit or last hit and is not required for other evaluations during the active period of the session, then you must change the event trigger.

   You can record the event data to a session attribute, which is accessible throughout the life of the session.

6. Additionally, you must consider whether the report group is necessary for recording with each instance of the event. If possible, remove the report group from the event.
7. If you believe that the data is generated by an unusual situation in the traffic in the web application, you can re-enable the recording of the report group data.