Depending on system hardware, web traffic, and configuration options, the STC can become backlogged, use up all permitted memory, and begin spooling incoming hits to disk via the DecoupleEx pipeline agent for later processing. This section provides some guidelines for handling high memory usage conditions.
The Short-Term Canister (STC) is an in-memory datastore for processing session data from active sessions on your web application. When a session ends or is timed out, its data are moved from the Short-Term Canister to the Long-Term Canister, which is stored on disk.
Monitoring canister status
Through the Portal, you can monitor Canister status in the DecoupleEx System Status report.
- Log in to the Portal as an administrator.
- Under System Reports, you can view the following status reports:
- Canister Status: Provides information about the current web traffic as well as STC memory/disk usage.
- DecoupleEx Status: Provides information about current DecoupleEx status and spooling.
In the System Status report for DecoupleEx, look for the Canister Status
value for each processing server.
Canister spooling
During spooling, the Canister Manager service and its child processes continue to process the data that is inserted into the Short-Term Canister.cWhen DecoupleEx stops the flow of hits into the Canister, the c-tree memory use gradually decreases as sessions time out and are moved out of the Short-Term Canister.
When the c-tree memory usage level drops below the predefined threshold, DecoupleEx begins inserting data into the Short-Term Canister again. This condition is displayed as Canister:Real-Time
on the DecoupleEx System Status report.
The canister oscillates between Real-Time
and Spooling
states as long as the volume of spooled data exceeds the Short-Term Canister memory capacity. Additionally, the Portal activity reports indicate that website activity is occurring later than the actual time of capture, because these statistics are recorded after insertion into the Canister by the STC processes. It is common for some hours on these activity reports to indicate lower than expected counts due to spooling. Hours after these artificially low-activity periods on the reports may show above normal counts because spooled hits were being re-inserted into the Canister from the spool at a faster rate, generally at the maximum safe rate that is configured in the Extended Decoupler session agent.
When spooling does occur, you should observe the Canister status for a sufficient period to identify behavior during periods of low web traffic. During these periods, the Canister should be able to catch up. New spool files will continue to be created as old spool files are processed, as spooling does not stop until the consumption rate of spool data exceeds the creation rate for long enough that the last spool file is consumed before a new one can be created.
If spooling is becoming a chronic condition, then one of the following solutions may alleviate the problem depending on the root cause:
- Reducing the workload
- Throughput optimizations
- Creating a faster and/or larger Tealeaf hardware cluster
Configuring c-tree memory use
In the configuration for the Extended Decoupler session agent, you can define the Canister memory threshold percentages at which the session agent begins spooling incoming hits to disk and at which the session agent resumes sending hits to the Canister after spooling began.
- These settings can be configured through the Pipeline Editor in TMS. The parameters for the DecoupleEx session agent to modify are listed in the Display Name column below.
- You can configure these parameters in the
[DecoupleEx]
section of theTealeafCaptureSocket.cfg
file. The parameters to change are listed in the Internal Name column.
Display Name | Internal Name | Description |
---|---|---|
Canister Max % Memory Used |
CanCheckMaxCtreeMemUsedPct |
Maximum allowed percentage of memory that is allocated by the Canister server. When set to 0, the cache value is ignored, and the CanCheckMinCtreeMemUsedPct setting is also ignored.
The default value is |
Canister Min % Memory Used |
CanCheckMinCtreeMemUsedPct |
Specifies the minimum percentage of FairCom Cache in use.
|
Applying Canister settings
After modifying Canister settings and restarting the Transport Service, hits may be spooled until the Canister is up and running. In some situations, the Extended Decoupler session agent does not receive notification that the Canister was restarted, and hits continue to be spooled even though the Canister is ready to receive them. In this case, a restart of all Tealeaf services is required.
Canister spooling status
This status indicates that the Canister exceeded one or more of the performance thresholds, and DecoupleEx began spooling incoming hits.
In this state, the Canister is processing the data already in memory, and new data arriving to the Canister is written to the disk spool to be read into the Canister later.
- If its
Persistence
setting was enabled for the DecoupleEx session agent, restarting services or rebooting retains the spooled data on disk. - If
Persistence
is disabled, the disk spool is deleted when the Transport Service running DecoupleEx is started.
If the DecoupleEx System Status report indicates high memory usage for Canister
Reason
, the likely explanation is that there was a surge in captured web traffic. While you can adjust system settings to increase the data throughput in the Canister, making these changes provides sufficient increases only if the spooled backlog represents a time period considerably longer than the specified session timeout. For example, if the Canister is configured to have a Session Idle Seconds
setting of 300 seconds, the spooled backlog must be over one hour old to be worth reconfiguring the Canister settings.
Metrics for sizing calculations
You can use metrics to help calculate the required sizing for the Canisters to process your web traffic without significant spooling.
The following metrics help calculate the required sizing.
Peak values can be calculated based on the DecoupleEx logs. These files are stored in the following files:
<logs_directory>\CSS_*
Where:
<logs_directory>
is specified in the DecoupleEx configuration.
To assess your system requirements, the absolute minimum information that is required for sizing calculations are the first two items; the other information provides a more accurate assessment.
Metric | Data source |
---|---|
Average Number of Sessions (or visits) Per Day |
See the Session Count report in the Portal Activity Reports. Average this value over as many days as makes sense. |
Average Number of Page Views Per Day |
See the Page View Count report in the Portal Activity Reports. Average this value over as many days as makes sense. |
Average Number of Hits Per Day |
See the Hit Count report in the Portal Activity Reports. Average this value over as many days as makes sense. |
Average Session Duration |
See the Session Duration report in the Portal Activity Reports. Average this value over as many days as makes sense. |
Average Page Size |
See the Page Size report in the Portal Activity Reports. Average this value over as many days as makes sense.
Note: If possible, filter on the Tealeaf reference dimension values to remove image files and other static content from the report.
|
Average number of hits/pages per session |
See the Session Avg Hits report in the Portal Activity Reports. Average this value over as many days as makes sense. |
Number of Sessions during Peak Hour |
See the Session Count report in the Portal Activity Reports. Use the peak hour from the peak day. As an alternative, you can use the DecoupleEx log files, whose location is listed above. |
Number of Page Views during Peak Hour |
See the Page View Count report in the Portal Activity Reports. Use the peak hour from the peak day. As an alternative, you can use the DecoupleEx log files, whose location is listed above. |
Number of Hits during Peak Hour |
See the Hit Count report in the Portal Activity Reports. Use the peak hour from the peak day. As an alternative, you can use the DecoupleEx log files, whose location is listed above. |
Number of Days Persisted |
This setting is defined in the Canister configuration. |
Session Timeout |
This setting is defined in the Canister configuration. |
Growth Factor |
You can use either expected or measured growth of the data volume as the metric. |
Setting the session timeout
The Short-Term Canister determines when to end a session that is based on the Session
Idle Seconds
setting for the Canister.
When no new hit was inserted into the Canister for a session for a period greater than the value of Session Idle Seconds
, the STC closes the session and flags it for downstream processing. This timeout scheme parallels standard web application practices.
The following secondary settings may contribute to deciding when to end a session:
- Events may be defined to close sessions that are based on the occurrence of the event. For example, if there is a logout link or button in your web application, you might define a close session event that is triggered when that link or button is pressed.
- To protect the STC, the Canister pipeline agent in the Transport Service imposes limits on session size and duration. By default, these limits are set to 2048 total hits, 5 MB of total memory, and 1 hour of total session duration. After changes any of the above settings, the Transport Service must be restarted.
Assessing canister storage allocation
If you are monitoring multiple sites or areas of your website using a single canister, you can use the following process to assess the canister storage requirements for each site.
- On the canister storage device, find out how much total storage space is used per day.
- In the Portal, navigate to System Status > Canister.
- The ratio of the session counts for each host that is compared to the total session count should give a rough estimation of how much space is used.
If the sessions from a specific host are much bigger than average, then this estimate may not be accurate. It should provide a good estimate.
Assessing Canister session agent performance
The Canister session agent supports multiple threads, which enable you to increase the hit insert rate by routing hits through multiple pipelines into the short-term canister. Before adding pipelines, you should assess the current Canister insertion performance.
Typically, bottlenecks in the Canister session agent occur while compressing the hit to better use STC memory. By enabling the display of the session agent performance statistics, you can monitor the hit insertion and compression rates.
In the Canister session agent configuration section in TealeafCapturesocket.cfg
, set the following property value:
DisplayPerf=true
This option enables the output of the maximum processing rate for the session agent to the Queued field of the Pipeline Status report.
When enabled, the Queued field of the Canister session agent in the report displays a number like 30360558
.
You can use this figure to determine if you have a bottleneck in the Canister session agent and the number of pipelines that are required to service the wanted hit rate. In the number 30360558
, the last four digits indicate the maximum compression rate into the STC, and the first four or five digits indicate the maximum insertion rate into the session agent. In the above example:
- Maximum insertion rate =
3036
hits per second - Maximum compression rate =
558
hits per second