If an incomplete set of hits is arriving into the Canister, then there may be a problem in how the Portal application is communicating with the Canister machine or the database.

• Data from the Canister may not be surfacing in the Portal reports.
• Data in the Canister may not be collected, aggregated, and inserted into the Experience Analytics databases.

There might something wrong with the Portal application talking to either the Canister machine or the database.

In most cases, this issue is caused by a basic I/O throughput problem. The incoming session count is higher than the outgoing session count. You can verify this issue through the DecoupleEx status report.

To fix this issue, you should identify if there is excess I/O capacity available using a tool such as SQLIO while Experience Analytics is running.

• If there is excess capacity, you can experiment with increasing the number of Session Processors, whose primary function is to compress sessions for storage on disk. By increasing the processing units, more sessions can be written to disk at a faster rate. After raising this number, you may notice that the rate at which sessions are processed is increasing.
• If there is no excess capacity, there is no software solution for this issue. Review your storage configuration and capacity.

• Review your STC memory settings.
• Identify if this issue has occurred a peak period or is a normal load
• Identify if there are any reasons why the load may have recently changed. For example, if recent changes to the web application have resulted in large average page sizes or more hits per session, STC processing capabilities may be affected.
• It may be that there are a high number of sessions in the STC waiting for writing to LTC.

To resolve high ctree memory issues:

• Review your session timeout settings. The value Session Timeout in the Canister report indicates the current session timeout.
• Review your session size settings. Three parameters in the Services Controls tab define the limits in size of a session in terms of hits, time, or kilobytes of data.
• You should consider configuring Session Close events to terminate sessions earlier.
• Acquire more memory for the STC server.

• In most Experience Analytics deployments, the STC begins spooling hits when Memory In Use reaches 80% of the total available memory, as defined in Canister Memory Configuration.
• Sessions waiting for evaluation (STC) indicates the number of sessions that have been spooled.
• Freeing up memory: You can review the following configuration settings to make more memory available in the STC for processing hits:
  • Max Ctree Bytes In the Services Controls tab, this setting defines the amount of RAM memory available for the Short Term Canister. It should be evaluated based on other required resources on the Processing Server.
  • Session Size Limits In the Services Controls tab, the following settings define the maximum permitted sizes of sessions, in terms of hits, data, and time. Reducing these size settings may increase the amount of free memory available for processing other hits.
    • Max Hits Per Session
    • Max Size Per Session
    • Max Time Per Session
      Note: The above settings may not be available in standard format. To change them, you may have to edit the Canister configuration in raw format.
  • Number of Hits Processors In the Services Controls tab, you may choose to increase the number of Hit Processors. Some considerations apply.

• Compare the Canister Memory Current Size and Canister Memory Max values to the Canister Memory Configuration value, which indicates the maximum available memory for the STC. If the first two values are significantly less than the latter one, there should be available RAM to add another processing thread.
• Review the CPU Utilization statistics to identify if there is processing capability available for adding another thread.
• If resources are available, you can add a Hit Processor through the Services Controls tab in the Canister configuration in TMS.
• After adding the thread, you should continue to monitor the effects through this report.

When the STC begins spooling, all hits from new sessions are spooled. Hits from sessions currently stored in memory continue to be processed through memory. For this reason, the percentage at which spooling begins should always be set below 100%.

Spooling information is reported in the DecoupleEx System Status report. To access the DecoupleEx report, navigate to System Status > DecoupleEx.

For each processing server, check Canister Status.

• If it is not listed as Healthy, check the Canister Message options:
  • High ctree Memory - The currently specified memory settings are inadequate for the load.
  • Non Evaluated Hits - If this figure is high, then the STC may be having difficulty processing events.
• A message of Spooling indicates that the Canister is spooling.

  The count of hits that have been queued is listed for each Canister:

  • Queued Hits in Memory - Queued hits in memory are waiting for evaluation. A high number may indicate CPU constraints.
  • Queued Hits On Disk - Queued hits stored on disk waiting for the opportunity to be evaluated.
  • Queued Hits Total - Sum of the two values above. This figure should be divided by the Canister Hit/Sec In value for the Canister to determine the number of seconds of queued data.
  • Hit/Sec In versus Hit/Sec Out - Difference between these two numbers indicates the rate of change in queuing.
    • If this value is > 0, then additional hits are being spooled.
    • If this value is < 0, then there is capacity to begin processing queued hits.

Spooling is initiated and managed by adding the Extended Decoupler Session Agent to each pipeline. The following checklist identifies the settings that enable and configure spooling behavior:

• Spool Triggers: Canister Max % Memory Used - When this percentage is reached, the session agent begins spooling hits to disk for the Short Term Canister and does not stop spooling until Canister Min % Memory Used is reached. The following settings can also trigger spooling:
  • Max Unevaluated Canister Hits - When this number of hits is in the STC waiting for evaluation, spooling begins. Spooling does not stop until Min Unevaluated Canister Hits is reached in the STC.
  • Max Sessions Waiting for LTC - When this number of sessions is in the STC waiting for writing to the LTC, spooling begins. Spooling does not stop until Min Sessions Waiting for LTC is reached in the STC.
  • Max Sessions Waiting for IDX - When this number of sessions is waiting for indexing, spooling begins. Spooling does not stop until Min Sessions Waiting for IDX is reached.
• Spool Drain: The rate at which the spool is emptied is determined by the following configuration settings for the Extended Decoupler Session Agent.

  Control Outflow Rate - When set to ON, the flow of hits from the queue back to the Canister is moderated based on the following settings:

  • Outflow Control Mode - Defines the method of how to control outflow of hits from the spool.
  • Max Outflow Hits Per Second - Depending on the Mode above, this setting specifies the maximum outflow in hits per second.
  • Max Outflow Bytes Per Second - Depending on the Mode above, this setting specifies the maximum outflow in bytes per second.

If you do not wish to enable disk queuing, you can use the lightweight predecessor to the above session agent. When memory limits are exceeded with the Decoupler session agent, the hits are simply dropped.

• Review the number of Non Evaluated Hits.
• Compare the above value to the rate at which hits are entering the Canister. This information is in the DecoupleEx report.
• If the non-evaluated hits constitutes more than a few seconds of data entering the Canister, the problem is usually associated with event processing. In most cases, the source of the issue is that events have been added or changed in the system to cause the processing backlog.
• Increase the number of Hit Processor or Session Processor threads if resources are available.