Troubleshooting failover

The PCA can be configured to failover from a primary CX Passive Capture Application to a secondary machine. In the event of network outage, system failover, or other interruption on the primary machine, the secondary machine becomes the active machine and begins capturing traffic.

If you configured failover and are experiencing difficulties, you may find troubleshooting tips and steps in this section to help you resolve the issue.

If you do not specify a port number when you configure a failover primary or subordinate, then failover uses port 9866.

PCA failover mode requires two PCAs, designated Primary and Secondary. On the Failover tab in the PCA Web Console, define the Primary/Secondary PCA IP/Port Addresses.

Note: Use their respective physical IP addresses instead of their host names.

Note: Both the Primary and Secondary PCAs must be running the same PCA build.

For proper identification of which is Primary or Secondary, the PCA examines the /etc/hosts file to find its IP address based on its local hostname. It then matches that IP address to the failover Primary/Secondary IP entries for assignment.

By running the Linux™ command hostname, the host name that is listed must be in the /etc/hosts file with its corresponding IP address. For example, executing hostname returns the pca01machine, then an entry similar to the following should appear in the hosts file.


10.10.100.1    pca01machine

Note: The wanted hosts file entry should be the first entry in the file.

If no match is found in the hosts file, then Failover mode fails to start. An error message appears in the capture.log file:


Both PrimaryAddress and SecondaryAddress must be specified in configuration file.

Determining the failover state

The PCA publishes statistics for the primary and secondary failover servers in various states. You can review the statistics to examine the failover states and determine the cause of failover issues.

Primary PCA stats

You can review the statistics that are published by the PCA for the primary failover server in various failover states. These statistics are published in the failover section of the Statistics tab.

When the Node state statistic is set to active, the server is delivering hits to other Tealeaf servers.

The Node state is active, meaning it is delivering hits:


Value	Statistic
primary	Node role
active	Node state
running	Capture state
yes	Failover active

The Primary was forced to failover to the secondary. Primary was stopped:


Value	Statistic
primary	Node role
passive	Node state
stopped	Capture state
yes	Failover active

Primary PCA failover log messages

Below you can review log messages for various failover conditions for the failover primary PCA server:

These log messages appear in capture.log.
Any capture.log messages that refer to peer is a reference to the other PCA, not the local one.

Failover is disabled:


TLAPI: Failover is disabled. Delivery is always enabled.

Failover Primary is in active delivery state:


TLAPI: Failover is enabled. Delivery is currently enabled.

Failover service falls over to the PCA secondary machine:


failoverd: Peer node is down (connection refused).
failoverd: Peer node is alive.
failoverd: Capture has stopped. Initiating failover to peer.
failoverd: Delivery stopped.

Failover failed back to the PCA primary machine:


failoverd: Requesting failback from peer.
failoverd: Delivery started.

Secondary PCA statistics

You can review the statistics that are published by the PCA for the secondary failover server in various failover states. These statistics are published in the failover section of the Statistics tab.

When the Node state statistic is set to active, the server is delivering hits to other Tealeaf servers.

Primary is running in active Node state. Secondary is in passive state (no delivery):


Value	Statistic
secondary	Node role
passive	Node state
running	Capture state
yes	Failover active

Secondary PCA is running, and Primary PCA is stopped or non-existent (Failed over to Secondary PCA):


Value	Statistic
secondary	Node role
passive	Node state
running	Capture state
yes	Failover active

When Secondary fails back to Primary, the Node state returns to passive again.

Secondary PCA failover log messages

Below you can review log messages for various failover conditions for the failover primary PCA server:

These log messages appear in capture.log.
Any capture.log messages that refer to peer is a reference to the other PCA, not the local one.

Failover is disabled:


TLAPI: Failover is disabled. Delivery is always enabled.

Failover Secondary is in passive Node state:


TLAPI: Failover is enabled. Delivery is currently disabled.

Failover Secondary took control:


failoverd: Received TakeControl request from peer. Taking control.
failoverd: Delivery started.

Secondary failed back to Primary server:


failoverd: Received ReleaseControl request from peer. Releasing control.
failoverd: Delivery stopped.

Proper sequence for starting, stopping, and restarting failover

There is a proper sequence for the Primary-Secondary failover PCAs.

Start Primary PCA first.
After the Primary PCA successfully started, start the Secondary PCA.
Note: If you are experiencing issues with correct operation of failover, you should always use the command line to manually stop and start it.

Note: The PCA may not correctly indicate the failover active state if the service was enabled or disabled multiple times. This is a known issue affecting all PCA builds.

Restarting Failover through the Web Console

The Failover service can be started and stopped through the Failover tab in the CX PCA Web Console.

To start and stop the Failover service through the Failover tab in the CX PCA Web Console:

Open the CX PCA Web Console on the secondary CX PCA server.
Click the Failover tab.
Click Restart failover.
Click Save changes to apply your changes.
Repeat this procedure on the primary CX PCA server.

After the failover status is reset, the CX PCA Web Console displays the new failover status. For example, the primary CX PCA reports the status as Failover is active (primary) and the secondary CX PCA reports a status of Failover is active (secondary).

Stopping and starting the Failover service manually from the command line

Older versions of the PCA Web Console do not correctly accept changes to failover. When in doubt, you must manually stop and start the failover service from the command line.

To stop and stop the Failover service from the command line:

Stop the failoverd service:
```
tealeaf stop failoverd
```
Confirm failoverd was stopped and removed by running the following command:
```
tealeaf ps
```
Review capture.log messages for more information.
Make changes as necessary.
To start the failoverd service:
```
tealeaf start failoverd
```

Determining the failover state

Primary PCA stats

Primary PCA failover log messages

Secondary PCA statistics

Secondary PCA failover log messages

Proper sequence for starting, stopping, and restarting failover

Restarting Failover through the Web Console

Stopping and starting the Failover service manually from the command line

Join the community

Academy

Resolving problems with PCA

Determining the failover state

Primary PCA stats

Primary PCA failover log messages

Secondary PCA statistics

Secondary PCA failover log messages

Proper sequence for starting, stopping, and restarting failover

Restarting Failover through the Web Console

Stopping and starting the Failover service manually from the command line

Related articles

Join the community

Academy