Through the Pipeline Editor, you can create or edit the pipelines for the current server.
To open the Pipeline Editor, select one of the following options:
- Transport Service > Transport Service configuration
- Canister > Event Bus configuration
In the Pipeline Editor, the pipelines defined for the current TMS server are displayed in the left panel. Beneath each pipeline, you can review the session agents in the pipeline. From top to bottom, they are displayed in the order of execution in the pipeline.
You can add and remove session agents from pipelines by dragging and dropping them.
The Pipeline Editor is displayed whenever a pipeline configuration is edited. Pipeline Editor manages the Transport Server configuration (TealeafCaptureSocket.cfg
), the Event Bus configuration (TealeafEventBus.cfg
), as well as the various child pipeline configurations used by some of the Session Agents.
Note: The Event Bus can be used to deliver Tealeaf events to third-party systems for additional analysis and reporting. The Event Bus is a component of cxConnect for Data Analysis, a separately licensed product for the Tealeaf CX platform.
How to monitor capture and Transport Service pipelines
Through the Pipeline Status tab you can monitor Tealeaf capture and Transport Service pipelines, including the processing by each agent in the pipeline, for all Windows™-based Tealeaf servers in your environment. Throughput performance is also available for individual session agents and for the entire pipeline.
By default, the selected server is polled for stats every two seconds through TMS.
The Connections table shows active connections to the Transport Service through the protocol in use, and the name of the machine from which the connection originates.
- All current connections are permanently displayed and cannot be added, removed or selected.
- The
slot#
column contains the memory address in the Tealeaf shared memory occupied by the connection.
The Pipeline table shows the different session agents in the order in which they appear in the currently selected pipeline. In the Pipelines table, you can review each session agent in the pipeline in the order of execution.
Editing configuration settings
Through the Pipeline Editor, you can edit global settings that apply to all pipelines on the selected server. This option is only available for configurations that can have multiple pipelines (TeaLeafCaptureSocket.cfg).
Firewall settings only apply to Transport Service pipelines.
Additional documentation on all settings is available in the TealeafCaptureSocket.cfg
file stored in the Tealeaf install directory on the server.
Complete the following steps to edit configuration settings.
- Select the head node containing the configuration description and click Edit at the bottom of the editor, or double-click the head node.
- Click the
+
next to a section to expand it. - Click a configuration setting, change the values as desired, and click Apply.
- When done, click OK.
Pipeline global settings
The following settings can be changed.
Setting | Description |
---|---|
Application Name |
The name of the application.
|
Setting | Description |
---|---|
Maintenance Interval |
The frequency in seconds that a maintenance scan is performed. |
Max Data Size |
The maximum size in KB of any data accepted by the Capture Socket Service. Packets larger than this value are ignored. By default, this value is set to 2048 kilobytes.
Note: This value should not be less than the values of any pipelines sending data, such as the CX Passive Capture Application. |
Max Memory Size |
The maximum size in KB reserved on the server for the Transport Service. When the amount of memory used by the process exceeds this value, the service is automatically restarted. By default, this value is set to 1900 kilobytes and should not be increased.
|
Max Restart Count |
The number of times that the pipeline is permitted to restart within the number of seconds defined for Restart Reset Interval .
If the pipeline restarts more times than this value within one |
Restart Interval |
The interval in seconds at which the pipeline checks to see if a restart is necessary. |
Restart Reset Interval |
The period of time in seconds over which the pipeline can attempt restarts. If the pipeline restarts more than MaxRestart times during this period, it is shut down. |
VM Free Disable Threshold |
The threshold percentage of free virtual memory below which the pipeline automatically shuts down.
To disable this check, set this value to 0. |
VM Free Enable Threshold |
After a shutdown, the pipeline restarts when the percentage of free virtual memory exceeds this value.
This value should be greater than the value set for |
Adding and editing pipelines
Through the Pipeline Editor, you can set up and edit individual pipelines for each port to which the CX Passive Capture Application forwards captured data.
By default, a single pipeline is created for you, which monitors port 1966
using standard HTTP traffic. A set of pipeline session agents is added to the pipeline based on the installed components in your Tealeaf solution. Additional details can be managed through manual editing of the appropriate configuration file.
You can continue to build the session agents of your pipeline. In this manner, you can build your pipeline until you reach one of the terminal pipeline session agents:
- The Canister session agent forwards the session data to a Tealeaf Short Term Canister.
- The Archive session agent forwards the session data to a Tealeaf Archive.
- The Null session agent discards the data (for debugging purposes).
- You can also point the end of a pipeline to the start of another pipeline.
- The Transport Service is a 32-bit application. Creating and using more than 20 pipelines in an individual Canister at one time might exceed the maximum available memory, which is up to 3GB.
- Depending on the objectives of the pipeline you are creating, there may be dependencies on the order in which the session agents are listed. Some session agents can only terminate a pipeline.
- This option is only available for configurations that can have multiple pipelines (
TealeafCaptureSocket.cfg
).
Complete the following steps to add a pipeline.
- To create a new pipeline, click the topmost node in the left panel.
- Select Add Pipeline.
- Enter a port number and select whether to use SSL in the pipeline.
Use port
1966
for standard HTML (non-SSL) traffic and1967
for SSL traffic.An SSL pipeline and a non-SSL pipeline should not both reference the same downstream session agent, especially if it is a canister. This situation results in multiple threads.
Note: Use of SSL can impact pipeline throughput.
- Click Apply.
- To edit the pipeline, select it and click Edit.
- After the pipeline has been added and edited, you can begin adding session agents to it.
- If your pipeline is receiving data, you can iterate on the configuration through TMS to tune performance.
Adding a session agent to a pipeline
Session agents can be dragged from the Available Session Agents panel to the pipeline name to add them.
Complete the following steps to add a session agent to a pipeline.
- Select a session agent to add from either:
- Available session agents, which includes the following options:
Custom
- Custom native (C++) session agentManaged
- C# or VB.NET session agent hosted by session agentCLRRulesEngine
- TCL script hosted by session agentRulesEngine
- Saved session agents list
- Available session agents, which includes the following options:
- Drag the session agent to the desired position in the pipeline tree.
A line appears when you hover over the pipeline showing where the session agent is to be placed.
Note: You may be required to rename the session agent if an instance of the same session agent is already in the Saved session agents pane.After the session agent is dropped in the pipeline, the Edit session agent dialog is displayed, allowing you to change the settings for the session agent.
- Click Add Config Items to add custom configuration items for the SessionAgent.
- Click OK to keep any changes.
After multiple session agents have been added, you can reorder them within the pipeline or move them between pipelines using drag and drop.
Editing session agent settings
In the Edit session agent dialog, you can change the properties for the selected session agent.
Complete the following steps to edit session agent settings.
- Double-click the session agent node.
The Edit session agent dialog is displayed.
- Click the
+
next to a group to expand it, if the session agent has configuration item groups. - Click a configuration setting to change it.
The Edit Config Item dialog is displayed.
- You can add configuration items to the session agent.
- To add a custom configuration item, click Add Custom Config Item. Enter a name for the custom configuration item and click OK.
- To add a custom configuration item for the session agent, click Add Config Items.
Note: Adding a custom configuration item only inserts the item into the configuration data. Customization of the affected
.DLL
is required. No data validation is performed on customer configuration items through TMS.
- Change the value as desired and click Apply to save the change.
- When done, click OK.
Extended HBR session agent
The Extended HBR session agent (HBREx) implements multiple child pipelines for an individual pipeline in the canister to better use memory. This improves the hits processing performance of the pipeline.
The difference between HBR and Extended HBR is that child pipelines are separate processes in HBREx, and are no longer part of the parent pipeline process. By running the child pipelines as separate processes, each pipeline has extra memory space, which improves the performance of HBR.
Example of HBREx with two child pipelines:
[HBREx]
TypeName=HBREx
CanisterAppName1 = [Canister_App_Name_For_1st_Child_Pipeline]
CanisterAppName2 = [Canister_App_Name_For_2nd_Child_Pipeline]
CanisterOffline1=Daily at 2300 for 10 minutes winddown 20 minutes
CanisterOffline2=Daily at 2300 for 10 minutes winddown 20 minutes
CanisterName1= [Canister_Name_1]
CanisterName2= [Canister_Name_1]
CanisterPipeID1=1
CanisterPipeID2=2
DLL=SessionAgentPipelineSplitterEx.dll
DownStreamConfigSection=Null
DefaultPerSecMax=200
InitTimeOut=1800
ActionScript=HBR.Action.tcl
MainScript=HBR.Main.tcl
PreProcScript=HBR.PreProc.tcl
RoutingMethod=Equal
MaxSessions=100000
PipelineConfig1=HBR_Pipeline1.cfg
PipelineConfig2=HBR_Pipeline2.cfg
ScriptTrace=OFF
CanisterSSPort1=19000
CanisterSSPort2=19000
You can access the Extended HBR session agent by navigating to the available session agents section within the pipeline editor tab of TMS.
- In the Portal, go to TMS > Worldview.
- Expand the node for the HBR server.
- Expand the node for Transport Service.
- Select Transport Service configuration > Config Actions > View/Edit.
In the Pipeline Editor window, you can see the HBR parent pipeline configured with other session agents, as well as the HBR session agent. Make note of the values specified in the HBR session agent configuration, as these can be used as a reference when configuring HBREx.
Remove the HBR session agent from the pipeline by dragging and dropping it into the Available SessionAgents box to the right. Replace the HBR session agent with HBREx by dragging and dropping into the pipeline.
The Edit SessionAgent box appears. Here you can view all of the fields of HBREx. HBREx has all the same fields as HBR, plus the “Child pipeline port range start”. This comes with the default value of 2000. You can edit the value by clicking Child pipeline port range start, and applying your changes. Child pipelines are separate processes and need to listen on individual ports. The ports that will be used by child pipelines start from the value specified in Child pipeline port range start (with a default of 2000). All parameters specified in HBREx will be the same as those specified in HBR.
The CanisterAppName for child pipelines must be specified as TealeafCSS_<port_number_assigned_to_child_pipeline>
. If any other value is specified as the CanisterAppName, it will be overridden as TealeafCSS_<Port_Number>
.
Note: When HBREx session agent is configured, PipelineConfig DS and DL log files for child pipelines are saved as as CSS_child_pipeline_port_number DS and DL.
Note: If you attempt to stop the Transport Service when HBREx is configured, the parent Capture Socket process sends a signal to its child processes to exit. This halts the process. If you do not exit child processes before this time, child processes are terminated to avoid orphan child processes.
HBRexHBR_PipeLine1.cfg
(and subsequent configuration files) for the child pipelines utilized by HBRex ensure the following values are set:Within [StatusMaster]:
Set
AppName
to the corresponding names CanisterAppName
from the master pipeline. (for example: TealeafCSS_2001
).This name must be the same, to allow these processes to communicate properly. as that child pipeline’s
CanisterAppName
as set in the [HBREx] section of the TealeafCaptureSocket.cfg
file.Within [Globals]:
Add
MaxDataSize=nnnnn
to match the size in the master pipeline.This section does not normally exist in the child pipeline configuration, but in the multi-process HBREx architecture this limit is not defaulted from the master pipeline.
When large hits are dropped by the child pipelines, a warning is reported in the Windows Application log. If the warning states the limit is at 2048kb, it’s likely due to the child pipelines not having a specified limit, and defaulting.
Changing session agents within pipelines
Within a pipeline, you can move and remove session agents.
- Complete the following steps to move a session agent within a pipeline.
- Select the session agent to move.
- Drag the session agent to the desired position in the pipeline tree. A line appears when you hover over the pipeline showing where the session agent is to be placed.
- Complete the following steps to remove a session agent from a pipeline.
- Select the session agent to remove.
- Drag the session agent to the Available session agents list to remove the session agent from the pipeline or to the Saved session agents list to save session agent instance for future use in this configuration.
Pipeline monitoring workflow
After your pipeline starts receiving data, you can iterate on it through TMS to monitor performance.
- Make changes to the pipeline through the Pipeline Editor.
- In the Jobs tab, schedule a job to push the changes immediately to all servers.
- Check the Pipeline Status tab to see if data is flowing through all session agents in the pipeline.
- Review the results in the destination system:
- If the pipeline ends at a Canister session agent, you can search in the Portal for active sessions or recently completed sessions to review the captured and processed data.
- For Event Bus pipelines, you must review results in the destination system.
- Repeat these steps until you are satisfied with your pipeline configuration.
Removing a pipeline
This option is only available for configurations that can have multiple pipelines (TealeafCaptureSocket.cfg
).
When a pipeline is removed, all of its session agents are discarded as well. To preserve the configurations for session agents in a pipeline, drag them to the Saved session agents panel first.
Note: There must be at least one pipeline to save the configuration.
Complete the following steps to remove a pipeline.
- Select the pipeline to remove.
- Click Remove Pipeline at the bottom of the editor.
- Click OK to confirm the removal of the pipeline.