Canister Services continue to restart

If you disabled Canister Services through the Windows Services Control Panel and the service continues to restart, you should verify that the network monitoring and infrastructure diagnostic tools such as Big Brother are not performing these restarts.

Canister spacer file does not exist

The Windows Application Event Log may contain a warning message from the TeaLeaf Pipeline indicating the Canister spacer file does not exist.

SADecoupleEx (TeaLeafCSS_1966): Canister spacer file does not exist
(C:\Program Files (x86)\TeaLeaf\Canister\Canister.spacer).

The Canister space file is used to reserve disk space for storage of sessions in the Canister. The above message indicates that this file does not exist. Without it, the Canister cannot save the processed sessions when disk space is low.

The Extended Decoupler session agent normally does not interact with this spacer file. However, when a low disk space condition is detected, the session agent deletes it and frees up disk space. When the message appears, DecoupleEx was unable to find the file.

The spacer file is created when the TLTMaint utility is run. To correct this issue, you can run this utility to create the spacer file.

The causes of session fragmentation

Tealeaf sessions can become fragmented. Fragmented sessions can be caused by the following conditions:

• A visitor's session is fragmented when the Tealeaf inactivity timeout period is exceeded. For example, if a visitor leaves a web page open during lunchtime, browsing again after lunch in the same browser window may cause a fragmented session. Because of the long period of inactivity, the first few pages are moved out of the Short-Term Canister (STC) into the Long-Term Canister (LTC). The next pages after lunch appear in the STC and are given a unique session fragment ID. However, since the new session is the same browser window, the TLTSID session cookie value does not change.
• A visitor session that exceeds any of the three 'safety' limits (time duration, number of hits, or bytes) is moved out of STC into LTC to prevent unnecessary consumption of STC space. After the session is moved to LTC, the next hit with the same TLTSID generates a new session fragment in the STC.
• A session opened anytime the Canister services are shut down for maintenance is moved to the LTC. Subsequent hits with the same TLTSID create a new session fragment in the STC.

Index error recovery

When an index job creates a new index or performs a merge, a new index is created and added to the TLL file immediately to ensure that the index name is reserved, but the index is marked as not valid. After the index or merge operation is complete, the index is marked as valid.

If the index or merge crashes, then the index remains marked as not valid and remains locked. A lock is broken if it is in place for more than two hours. If during the TLL check, Index Check finds an index with no lock that is marked as not valid, the index is deleted.

If an error is returned during indexing, the index is marked as requiring an index check. This index is not used for indexing until the flag is removed. If the index check fails, the index is marked as corrupt and removed from the TLL. If the check succeeds, then the check required flag is cleared and the index is returned to service.

Corrupted index directories are renamed with a .CORRUPT file extension for later inspection.

Rebuilding base canister files so that Tealeaf services can connect to canister as TLUSER or ADMIN

If the TLUSER and/or ADMIN userids are not present in the Canister datastore, the other Tealeaf services are prevented from connecting to the Canister. In such a scenario you must rebuild the base Canister files, saving the existing Canister session and search index data.

The following steps rebuild the base Canister files, saving the existing Canister session and search index data:

1. Stop all Tealeaf services.
2. Rename CANISTER.dbs directory to CANISTER.dbs.old.
3. Rename Indexes directory to Indexes.old.
4. Run CanRebuild.exe, choosing to rebuild only the full Canister and nothing else.
5. Rename Indexes.old directory back to Indexes.
6. Move (do not copy) the following files from CANISTER.dbs.old to the new CANISTER.dbs directory:

   
   LSSN * PEVT * NDLY.dat LDLY.dat PATH.dat SRVR.dat EVNT.dat SEVT.dat
   

7. Start all Tealeaf services.

Changing the logging level to troubleshoot exceptions

When an exception is detected in the Indexer log, raise the logging level for the indexer to 4 or 5, which enables the capture of a wider set of debugging data.

When reporting an error to Tealeaf, submit a log file that is set to level 4 or higher.

Complete the following steps to change the logging level.

1. Log in to the Portal as an administrator.
2. From the Portal menu, navigate to TMS.
3. From the View drop-down, select Servers.
4. Click the Session Indexer node.
5. Click Index Service configuration. In the Config Actions panel, click View/Edit.
6. In Index configuration, click the Scheduling/Diagnostic tab.
7. In the Diagnostics group, increase the level to 4 or 5.
8. Click Save.
9. Push the configuration to each server. A service restart should not be performed currently.

Using "Check and Fix" feature to add missing sessions to the index

If the Session Indexer service is in the stopped state, sessions are not indexed when they move to the Long-Term Canister. If you suspect some sessions were not indexed or that the Indexes may not be up to date, you can use the Search Server "Check and Fix Indexes" feature.

To run the Check and Fix feature:

1. In a web browser, enter the following URL: http://<hostname or IP address of server>:19000/.
2. Click the Canister/Indexer Check link. The table on the resulting page shows how many sessions exist in each day's data and how many of those sessions were indexed.
3. Click Check and Fix.

   The system indicates the number of sessions that are needed to be indexed and begins the process. The process continues in the background even if you close the browser window.

4. To check the status of the process after the browser window is closed, open a new browser window and enter the following URL: http://<hostname/IP addr of server>:19000/CanisterIndexCheckerStatus .