One component of a Canister, the Short Term Canister is an in-memory database used for storing active sessions.
To review information on an STC, in the Canister Status report select the individual Canister that you wish to review from the Experience Analytics CX drop-down. The following areas of the report can be reviewed to check STC health.
Determine if hits are even arriving in the Canister. If some hits are arriving, continue with these steps:
Session Timeout: This value in seconds defines how long idle sessions are permitted to remain open in the Short Term Canister without updates or changes applied to them.
Sessions waiting for archive to disk: This value indicates the number of sessions that have been marked for closing but have not been written to disk in the Long Term Canister.
- A high number for this value typically indicates disk storage problems, I/O problems with the
storage media, and processing resource issues.
- Disk storage: Verify the amount of free disk space available under Disk Usage:
Server Free Disk Size.
- I/O problems: To verify I/O problems, use your external disk monitoring tools.
- Disk storage: Verify the amount of free disk space available under Disk Usage:
- Processing resource issues: It's possible to increase the number of session processors in the STC. However, adding these threads does cost available CPU and RAM resources.
- A high number for this value typically indicates disk storage problems, I/O problems with the storage media, and processing resource issues.
Memory In Use: If this figure is greater than 80% in most deployments, hits submitted to the STC for evaluation are spooled to local disk for later processing.
Non Evaluated Hits: A non-zero figure indicates that the Short Term Canister is having difficulty processing events.
Verifying that no hits are arriving in Canister
To verify that there are no hits arriving in the Canister, do the following:
Hits/Sec In Current
MB/Sec In Current
If both of the above values are 0 and unchanging over a period of a few seconds, then there are no hits arriving in the Canister, and the system is spooling.
Some hits are arriving in the Canister
The Canister Status report provides detailed information on the active and completed sessions stored in the Canister, as well as indexing operations.
If an incomplete set of hits is arriving into the Canister, then there may be a problem in how the Portal application is communicating with the Canister machine or the database.
- Data from the Canister may not be surfacing in the Portal reports.
- Data in the Canister may not be collected, aggregated, and inserted into the Experience Analytics databases.
There might something wrong with the Portal application talking to either the Canister machine or the database.
Sessions waiting for archive to disk
If the Canister Status report indicates that there are a high number of
waiting for archive to disk, then the Short Term Canister is unable to write to the Long
Term Canister at a sufficient rate.
In most cases, this issue is caused by a basic I/O throughput problem. The incoming session count is higher than the outgoing session count. You can verify this issue through the DecoupleEx status report.
To fix this issue, you should identify if there is excess I/O capacity available using a tool such as SQLIO while Experience Analytics is running.
- If there is excess capacity, you can experiment with increasing the number of Session Processors, whose primary function is to compress sessions for storage on disk. By increasing the processing units, more sessions can be written to disk at a faster rate. After raising this number, you may notice that the rate at which sessions are processed is increasing.
- If there is no excess capacity, there is no software solution for this issue. Review your storage configuration and capacity.
High ctree Memory
If the Canister message indicates
High ctree Memory, the configured RAM
aside for the STC is insufficient for current processing load. Do the following:
- Review your STC memory settings.
- Identify if this issue has occurred a peak period or is a normal load
- Identify if there are any reasons why the load may have recently changed. For example, if recent changes to the web application have resulted in large average page sizes or more hits per session, STC processing capabilities may be affected.
- It may be that there are a high number of sessions in the STC waiting for writing to LTC.
To resolve high ctree memory issues:
- Review your session timeout settings. The value
Session Timeoutin the Canister report indicates the current session timeout.
- Review your session size settings. Three parameters in the Services Controls tab define the limits in size of a session in terms of hits, time, or kilobytes of data.
- You should consider configuring Session Close events to terminate sessions earlier.
- Acquire more memory for the STC server.
- In most Experience Analytics deployments, the STC begins spooling hits when
Memory In Usereaches 80% of the total available memory, as defined in
Canister Memory Configuration.
Sessions waiting for evaluation (STC)indicates the number of sessions that have been spooled.
- Freeing up memory: You can review the following configuration settings to make more memory
available in the STC for processing hits:
Max Ctree BytesIn the Services Controls tab, this setting defines the amount of RAM memory available for the Short Term Canister. It should be evaluated based on other required resources on the Processing Server.
- Session Size Limits In the Services Controls tab, the following settings define the
maximum permitted sizes of sessions, in terms of hits, data, and time. Reducing these size settings
may increase the amount of free memory available for processing other hits.
Max Hits Per Session
Max Size Per Session
Max Time Per SessionNote: The above settings may not be available in standard format. To change them, you may have to edit the Canister configuration in raw format.
- Number of Hits Processors In the Services Controls tab, you may choose to increase the number of Hit Processors. Some considerations apply.
In most deployments, the Short Term Canister utilizes a single thread for processing hits. Depending on traffic volumes and available system resources, adding one or more Hit Processors can improve STC performance.
- Compare the
Canister Memory Current Sizeand
Canister Memory Maxvalues to the
Canister Memory Configurationvalue, which indicates the maximum available memory for the STC. If the first two values are significantly less than the latter one, there should be available RAM to add another processing thread.
- Review the CPU Utilization statistics to identify if there is processing capability available for adding another thread.
- If resources are available, you can add a Hit Processor through the Services Controls tab in the Canister configuration in TMS.
- After adding the thread, you should continue to monitor the effects through this report.
When the STC cannot process enough hits to match the incoming rate of hits, all new hits are written to a temporary file on the local disk until the memory usage has dropped to a sufficient level that both incoming and spooled hits can be safely processed.
When the STC begins spooling, all hits from new sessions are spooled. Hits from sessions currently stored in memory continue to be processed through memory. For this reason, the percentage at which spooling begins should always be set below 100%.
Spooling information is reported in the DecoupleEx System Status report. To access the DecoupleEx report, navigate to System Status > DecoupleEx.
For each processing server, check
- If it is not listed as
Healthy, check the
High ctree Memory- The currently specified memory settings are inadequate for the load.
Non Evaluated Hits- If this figure is high, then the STC may be having difficulty processing events.
- A message of
Spoolingindicates that the Canister is spooling.
The count of hits that have been queued is listed for each Canister:
Queued Hits in Memory- Queued hits in memory are waiting for evaluation. A high number may indicate CPU constraints.
Queued Hits On Disk- Queued hits stored on disk waiting for the opportunity to be evaluated.
Queued Hits Total- Sum of the two values above. This figure should be divided by the
Canister Hit/Sec Invalue for the Canister to determine the number of seconds of queued data.
Hit/Sec Out- Difference between these two numbers indicates the rate of change in queuing.
- If this value is > 0, then additional hits are being spooled.
- If this value is < 0, then there is capacity to begin processing queued hits.
Spooling is initiated and managed by adding the Extended Decoupler Session Agent to each pipeline. The following checklist identifies the settings that enable and configure spooling behavior:
- Spool Triggers:
Canister Max % Memory Used- When this percentage is reached, the session agent begins spooling hits to disk for the Short Term Canister and does not stop spooling until
Canister Min % Memory Usedis reached. The following settings can also trigger spooling:
Max Unevaluated Canister Hits- When this number of hits is in the STC waiting for evaluation, spooling begins. Spooling does not stop until
Min Unevaluated Canister Hitsis reached in the STC.
Max Sessions Waiting for LTC- When this number of sessions is in the STC waiting for writing to the LTC, spooling begins. Spooling does not stop until
Min Sessions Waiting for LTCis reached in the STC.
Max Sessions Waiting for IDX- When this number of sessions is waiting for indexing, spooling begins. Spooling does not stop until
Min Sessions Waiting for IDXis reached.
- Spool Drain: The rate at which the spool is emptied is determined by the following
configuration settings for the Extended Decoupler Session Agent.
Control Outflow Rate- When set to
ON, the flow of hits from the queue back to the Canister is moderated based on the following settings:
Outflow Control Mode- Defines the method of how to control outflow of hits from the spool.
Max Outflow Hits Per Second- Depending on the Mode above, this setting specifies the maximum outflow in hits per second.
Max Outflow Bytes Per Second- Depending on the Mode above, this setting specifies the maximum outflow in bytes per second.
If you do not wish to enable disk queuing, you can use the lightweight predecessor to the above session agent. When memory limits are exceeded with the Decoupler session agent, the hits are simply dropped.
When hits arrive in the Short Term Canister, events are evaluated, and any resulting actions are taken. Review the Canister report for detailed information.
- Review the number of
Non Evaluated Hits.
- Compare the above value to the rate at which hits are entering the Canister. This information is in the DecoupleEx report.
- If the non-evaluated hits constitutes more than a few seconds of data entering the Canister, the problem is usually associated with event processing. In most cases, the source of the issue is that events have been added or changed in the system to cause the processing backlog.
- Increase the number of Hit Processor or Session Processor threads if resources are available.