cxConnect for Data Analysis enables you to export configured session data to a series of data files, which can be inserted into database tables of your own configuration. To use the data files options, you must customize a set of provided scripts to create a set of database tables to receive the output database files and to insert data into the databases.
As part of the software distribution, sample scripts are provided on the server where cxConnect for Data Analysis is installed in the <install_directory>\DataExtractor\Scripts directory:
You can customize these scripts to upload a batch ETL to any staging or master integration database. Uploading operations must be scheduled on an interval such that the number of concurrent log files in the export directory does not exceed the value that is defined for the
Concurrent Logs setting.
In the directory, the two SQL files, CreateDB.sql and CreateSchema.sql are used to create the database.
CreateDB.sqlto set the database file location.
CreateDB.sqlto create the DB data files.
CreateSchema.sqlto create the schema.The database is created.
- Configure the insert scripts.
Note: After cxConnect for Data Analysis exports the data files to the server, you must move them to their final destination, which requires more configuration and external scripts.
Using the OnClose text box, you can specify a batch file command that is applied to every data file after it was written and closed. The closed file is passed to the batch file as an argument.
For example, Hadoop functions well with files compressed using the LZO algorithm. Below is a batch file that compresses the closed data file using LZO.
echo off echo file: %1 :: call lzop to compress file lzop %1 if errorlevel 0 ( delete %1 goto all_is_good ) exit /b 1 :all_is_good exit /b 0
cxConnect for Data Analysis exports session data into a set of denormalized flat data files, which can be easily searched and rapidly bulk-loaded into the destination database.