client_random missing

If the capture.log contains the message "client_random missing", it is an indication that the PCA corrupted SSL traffic. The source may be many possible issues. There may be duplicate SSL keys that the PCA is using.

Failed to write ReportQueryLogEntity

The Failed to write ReportQueryLogEntity error message is a recorded transaction issue with an internal logging table. It does not impact users or performance in any way and can be ignored.

Here is an example of the message in the Data Service log:

Failed to write ReportQueryLogEntity

FetchNSAT failed in WriteInteraction with code -160

To address FetchNSAT failed in WriteInteraction with code -160, consider increasing the session timeout by a minute, especially if this error occurs repeatedly and frequently.

Session timeouts can also be changed by triggered events.

The most likely cause of this error is when a session times out just as the next hit in the session is being added. The result is that the session is fragmented, and the hit is placed in a newly created session.

FetchNSAT failed in WriteInteraction with code -127

Typically, the FetchNSAT failed in WriteInteraction with code -127 message occurs when the system is completing a cycle services or a CanTrim operation currently.

If the Canister Server service fails while a hit is in the pipeline (a rare occurrence), the hit could be dropped with this message.

If this message occurs only when the Canister service is being restarted, it can be ignored.

Invalid Deflate Data

There might be instances when the IIS 6 Capture filter for Siebel application does not capture correctly.

The error as shown by CX RealiTea Viewer is "Invalid Deflate Data".

The Siebel ISAPI filters are performing compression. For the IIS capture filters to handle this compression, add the following two lines to the [Globals] section of the TeaLeafIIS6.cfg file, on all the Siebel Web servers where the IIS capture filter is installed:

IgnoreChunked=False
IgnoreGZip=False

After making this change, you must restart IIS on those Web servers.

Missing anchor attributes in the <A> link HTML tag

During replay you might see several highlighted links to the same page.

Missing anchor attributes in the <A> link HTML tag

Anchor attributes that link to items on the same HTML page are not downloaded with the pages in the session. These items are a client-side tag feature, so you may see several highlighted links to the same page.

System.Web.HttpRequestValidationException

You might encounter an Error System.Web.HttpRequestValidationException message.

Symptom:
ErrorSystem.Web.HttpRequestValidationException: A potentially dangerous
Request.Cookies value was detected from the client (ErrorId=
Resolution:

In IIS Manager, verify that the Tealeaf IIS capture filter is installed at the global websites level rather than in a specific website.

SQL exception "Connection reset by peer: socket write error"

This error may occur when the Report Data Collector service is trying to communicate with SQL Server running on a remote server. It indicates that the remote SQL Server did not respond to the request by the Report Data Collector service to insert data. The error condition could be either network or SQL Server related.

Restarting the Report Data Collector service should clear up the problem, if it was a temporary issue with the remote SQL Server. If the error persists after restarting the service, contact the DBA responsible for the remote SQL Server or the network staff responsible for the connection between the Tealeaf server and the remote SQL Server. The root cause is probably found in one of these two areas.

Failed to get number of sessions to be indexed from Canister

This error indicates that the Session Indexer service cannot connect to the Canister Server datastore service. This situation may occur if the Canister Server service is in the stopped state or in maintenance mode.

When the Canister Server service is started, it begins maintenance mode by running runs the TLTMaint program. TLTMaint does an exhaustive check of the Canister data files. Although TLTMaint usually completes in less than a minute, it can sometimes take quite a long time to complete if there was an abnormal shutdown that invalidated any of the Canister's .dat or .idx files.

When the other Tealeaf services start, they try to open communications with the Canister Server. The services are well-behaved. If they cannot achieve communication right away after starting, they retry every 10 seconds or so and wait at least five minutes before reporting any connection errors to the Event Log.

If you look at the actual Event Log, you should see messages after the Session Indexer service error message that TLTMaint completed without errors, and the Session Indexer error messages should stop being logged.

Query Server fails to shut down and restart properly

Upon restart of the Query Server, you may see errors like the following in the event log:

Errors:
7/28/2008 10:50:00 AM QueryServer ERROR An error occurred in the listener for 
10.10.32.252:21000 : Only one usage of each socket address (protocol/network 
address/port) is normally permitted
7/28/2008 10:50:00 AM QueryServer ERROR An error occurred in the listener for 
127.0.0.1:21000 : Only one usage of each socket address (protocol/network 
address/port) is normally permitted
7/28/2008 10:50:00 AM QueryServer ERROR at 
System.Net.Sockets.Socket.DoBind(EndPoint endPointSnapshot, SocketAddress 
socketAddress)
at System.Net.Sockets.Socket.Bind(EndPoint localEP)
at System.Net.Sockets.TcpListener.Start(Int32 backlog)
at TeaLeaf.Reporting.Server.TLHttpServer.Listen(Object obj)

If you see the above error messages, the issue might be caused by a failure in the RSE Service to shut down properly. In some cases, the service remains operational for a few seconds after the shutdown command is issued.

When a restart of the RSE Service is attempted, the Service fails to bind to the assigned port. When this error occurs, the service must be killed manually through the Windows™ Task Manager (or equivalent tool). Then, the RSE Service can be restarted as part of normal Tealeaf startup procedures.

reassd: Couldn't find SSL session cache entry

Every SSL session is unique, and after a fully negotiated session occurs, its session key is cached for reuse during a long session. The reassd: Couldn't find SSL session cache entry message can occur in two cases: as described here.

• (More likely) The SSL session cache table size is too small. The default is 10,000. The PCA statistics page shows the maximum number of entries used. If it reaches the configured maximum size and you are getting this message, then there was probably much SSL activity for a given time period that caused the still-needed session key to be dropped to make room for new entries.

  To fix the issue, try increasing the maximum SSL session limit to 20,000 on the PCA Web Console interface, on the Interface page, under the entry "Max SSL sessions to cache".

  Note: If the issue isn't resolved after setting the limit to 20,000, then another issue is likely the cause and should be investigated. Do not continually increase the session cache limit beyond the maximum of 20,000.
• (Less likely) A PCA restart occurred, which caused the SSL session entry cache table to be cleared. Any SSL transactions that were in progress then would cause the "not found" message to be generated after the PCA software was reinitialized. Restarts occur when saving Web Console configuration changes or when the health/maintenance script issues a restart instruction based on its assessment of the PCA health.

Max SYN/WAIT limit reached

The PCA logs this message as a result of processing the raw TCP/IP data.

Initiating a TCP connection requires three handshake packets:

• The source host sends a sync packet (SYN)
• The destination host then responds with its own sync packet (SYN+ACK)
• The source host sends an acknowledgment packet (ACK).

The PCA only attempts to reassemble a TCP connection into HTTP hit data if it sees all thee above packets. PCA maintains a finite buffer for caching a TCP connection's packets until all three handshakes are seen. If the PCA doesn't see all three before the buffer becomes full, those packets is flushed from the buffer and the above message is logged.

The three handshakes may not be seen for the following reasons:

• Packets lost on the network
• Packets corrupted
• Packets missing from the data stream that is fed to the PCA
• Packets arriving too far out of order
• Malicious SYN attacks

If you are seeing this message regularly, it could mean that the TCP connection cache is not big enough, and you can increase the cache size. However, over time, these messages may appear in any Tealeaf system. If the messages occur with only sporadic frequency, the cache may be sized appropriately; the issue may be simple network "noise".

The incomplete TCP connections could was introduced at the network switch SPAN monitor session without affecting you. It is also possible that you may have broken connections, which the SPAN session would pass on exactly as it was transmitted. From the point of view of a passive observer, there is no easy way to determine which of these occurred. However, visitors were probably not affected as far as browsing was concerned; if the connection was lost (no ACK packet received), then the browser automatically sends another request to the Web server for the wanted data, causing at worst nothing to happen. It should not be the source of an HTTP 500 error.

Other common sources of these errors are requests to an IP address that is not active. For example, suppose the PCA is configured to listen to IP port 8080, and there is a keep-alive service trying to get a page from each of 8 Web servers on port 8080. If only six Web servers are currently installed, the failure of the other two servers to respond may cause the "SYN/WAIT limit reached" message to appear.

When debugging related problems, the PCA Max SynWait limit can be increased. For sites with many high-traffic spikes, this limit may need to be increased during installation/configuration. After increasing the maximum limit, monitor the capture log.

• If the Syn/Wait buffer fills up rapidly, and there are many dropped connections every minute, the problem is probably in the SPAN port configuration.
• If there are only a few missing Syn/Wait packets each hour, it is probably normal network behavior.

Search server log error "Auth failed, no Authorization header on cmd"

When Portal or NT authentication is enabled for Search server, you might get errors in the Search Server log indicating a failed authentication. For example, you might see the following error in the Search Server log:

Auth failed, no Authorization header on cmd: 
   /EventDefinitions</Authentication>
Auth failed, no Authorization header on cmd: 
   /IndexSearchParallelStart</Authentication>
Auth failed, no Authorization header on cmd: /SessionInfo</Authentication>
Auth failed, no Authorization header on cmd: /ResultSet</Authentication>
Auth failed, no Authorization header on cmd: /TLLList</Authentication>

Some number of these messages is expected during normal use of the product.

Under NT authentication or Portal authentication, each time that a user logged in and attempts to query Search Server for data that is protected by this authentication, one instance of this message is generated.

When Search Server is configured for NT or Portal Authentication, a series of headers is used to validate requests made by Tealeaf services on behalf users.

An Authentication header cannot be added by the Tealeaf service calling to Search Server until the server responds with the challenge string. This string is then used in the header of the subsequent query to Search Server. Since Search Server cannot anticipate whether the initial query is followed by the subsequent query, the initial request is rejected, and these messages are recorded in the log, if they are, in fact, errors.

The reject response from Search Server contains a set of challenge headers that are used to generate valid subsequent requests with the proper authentication information.

Under normal conditions, one of these entries is almost always followed by a normal log entry for the same command, which indicates that the client successfully supplied the correct authentication header on the second try.

DecoupleEx warning occurs when starting services

The Windows Application Event Log contains a warning message from the Pipeline, which indicates that the Canister has been stopped.

SADecoupleEx: Canister shared memory is invalid. Flow of hits to 
the canister has been stopped. 

If the Application Event Log contains many other messages that are clustered around this same time, you might see that all Tealeaf services are shutting down and restarting. In most cases, the "TLBackup -CycleServices" Windows Scheduled Task is run during the early morning hours, typically at 2:05am. Part of the start of CycleServices is running the TLTMaint program, which is executed whenever the Canister Server service is started. TLTMaint examines the entire Long-Term Canister for data consistency and can run from 2 seconds to 200 seconds or more. In most cases, it completes in less than a minute. While TLTMaint is running, no other Tealeaf services can connect to the LTC.

The warning message you see from SADecoupleEx occurs when the Pipeline is attempting to connect to the Canister and is being informed that the Canister is not available because TLTMaint is still running.

A little later in the startup messages in the Event Log, you should see that TLTMaint completed with no errors and another message from the Pipeline, indicating that the flow of hits to the Canister resumed.

After Tealeaf is in place for a few weeks, the TLTMaint program takes longer to complete because the LTC is bigger. Therefore, the warning message might not appear until a few weeks after the initial installation yet becomes a standard entry in the Event Log thereafter.

If you see this warning in the Event Log and it is not part of a startup sequence, log a support case.