Skip to main content

Google Cloud Storage as an event publisher

Exchange provides an integration with Google Cloud Storage as an event publisher to share event data with any application in the Exchange Ecosystem.

Google Cloud Storage allows worldwide storage and retrieval of any amount of data at any time. You can use Cloud Storage to serve website content, store data for recovery, distribute large data objects to users via direct download, and many more. When combined with the Exchange ecosystem, you can address various business needs, such as:

  • Store offline customer purchase data in Google Cloud Storage.
  • Gain insight into your online and off line customer journey.
  • Share customer data to an analytics platform like Journey Analytics.

How it works

  1. Exchange pulls event data stored in Google Cloud Storage as batch JSON payloads, TSV, or CSV files.
  2. Exchange uses address and account credential information to share the event data with any application in the Exchange ecosystem. If Google Cloud Storage finds new event data, it downloads it, maps the data by using a mapping file that you provide, and sends the data to Exchange.
  3. The event files are moved to a processed folder Google Cloud Storage.
  4. Exchange generates a report file for each JSON, TSV, or CSV file that was processed.
  5. Exchange shares the data as events to an application endpoint that you designate.

Before you begin

Exchange

  • You must have an Exchange account.
  • You must be a licensed user of Exchange or an Exchange Business Partner solution.

Google Cloud Storage

Ensure that you have a Google Cloud Storage account and that you have configured your account according to Google Cloud Storage requirements and your business needs.

Configure your Google Cloud Storage bucket

For this integration, it is best practice that you create a new Project and bucket.

To create and configure your Google Cloud Storage Project and bucket:

  1. Open the Cloud Storage browser in the Google Cloud Platform Console.
  2. Create a New Project.
    1. Go to your Projects, and create a New Project.
    2. Enter a Project name and select the Location, then click Create. For more information, see Projects.
    3. Go into the Project that you created.
  3. Now that you're in your new Project, you must configure the Project's Service Account. Service accounts allow applications to authenticate and access Google Cloud Platform resources and services.
    1. Open the Service Accounts page in the Google Cloud Platform Console.
    2. Click Select a project. Select your project and click Open. Then select Create Service Account.
    3. In Service account details, enter in a Service account name, Service account ID, and optionally a Service account description. Then click Create.
    4. Designate a Role to the Service Account that will give it read and write permissions for the Google Cloud Storage bucket and its objects.
    5. Click Continue.
    6. Under Grant users access to this service account, select the Service account user role you created. Then select Create Key
    7. Indicate that the Service Key file type should be JSON and click Create. The Service Key JSON file is downloaded to your local computer.

      It is essential that you save the JSON file. From this point on we will refer to this file as the Service Credentials file. You will need this file later in the integration during the endpoint registration process in Exchange.

      Note: The Google Service Credentials Key has a 10 minute activation period. After generating the key, wait 10 minutes before registering your Google Cloud Storage endpoint in Exchange.

  4. Create a bucket
    1. Open the Cloud Storage browser in the Google Cloud Platform Console.
    2. Ensure that you are in Project that you created for this integration.
    3. Click Create bucket.
    4. Specify a Name, subject to the bucket name requirements. Make sure to name it something memorable or significant. You will need the bucket name during the endpoint registration process.
    5. Select a Default storage class for the bucket. The default storage class will be assigned by default to all objects uploaded to the bucket.
    6. Enter the Location where the bucket data will be stored.
    7. Click Create.

Configure your mapping file (Optional)

Exchange can consume TSV, CSV, or batch JSON files. Other files in your Google Cloud Storage bucket are ignored. For Exchange to be able to read and share the contents of those files, you must create a mapping file to map the data to a naming convention and syntax that Exchange can read. There can only be one mapping file per bucket. However, if you need to change the type of data Exchange pulls from your bucket, you can edit the mapping file at any time.

Note: If you do not create a mapping file, then you can upload batch event JSON files only. The events in the JSON file must be recognized Exchange events.

Create your mapping file

Your mapping file must be in XML format and must be host it in your GCS bucket, the same location as the data files that you want to share with Exchange. When you create the mapping file, make sure to name it something memorable or significant. You will need the mapping file name during the endpoint registration process.

When you create a mapping file to map event data from a CSV file, is can look similar to this example: <?xml version="1.0" encoding="UTF-8"?> <UBX_EVENT_MAPPING dataFileFormat="CSV" sourceEvent="offerAccepted"> <UBX_EVENT sourceEvent="offerAccepted"> <code>offerAccepted</code> <channel>Paid Media / AdTech</channel> <timestamp sourceIndex="1" sourceType="timestamp" sourceFormat="yyyy-MM-dd HH:mm:ss"></timestamp> <attributes> <attribute> <name>eventName</name> <value>Accepted Offer</value> <type>String</type> </attribute> <attribute sourceIndex="3"> <name>USERAGENT_TYPE</name> <type>String</type> </attribute> <attribute sourceIndex="4"> <name>USERAGENT_FAMILY</name> <type>String</type> </attribute> <attribute sourceIndex="5"> <name>USERAGENT_DEVICETYPE</name> <type>String</type> </attribute> <attribute sourceIndex="6"> <name>USERAGENT_OS</name> <type>String</type> </attribute> <attribute sourceIndex="7"> <name>LANGUAGE</name> <type>String</type> </attribute> <attribute sourceIndex="8"> <name>GEO_CITY</name> <type>String</type> </attribute> <attribute sourceIndex="9"> <name>GEO_CITYCONF</name> <type>String</type> </attribute> <attribute sourceIndex="10"> <name>GEO_POSTAL</name> <type>String</type> </attribute> <attribute sourceIndex="11"> <name>GEO_POSTALCONF</name> <type>String</type> </attribute> <attribute sourceIndex="12"> <name>GEO_AREACODE</name> <type>String</type> </attribute> <attribute sourceIndex="13"> <name>GEO_REGION</name> <type>String</type> </attribute> <attribute sourceIndex="14"> <name>GEO_COUNTRY</name> <type>String</type> </attribute> <attribute sourceIndex="15"> <name>GEO_GMTOFFSET</name> <type>String</type> </attribute> <attribute sourceIndex="16"> <name>GEO_DMA</name> <type>String</type> </attribute> <attribute sourceIndex="17"> <name>QUERYSTRING</name> <type>String</type> </attribute> </attributes> <identifiers> <identifier sourceIndex="2"> <name>USERID</name> <type>String</type> </identifier> </identifiers> </UBX_EVENT> </UBX_EVENT_MAPPING>

In the root element <UBX_EVENT_MAPPING>, you define the input data file format and where the source event type can be found. Because the event mapping is based on the source event type, Exchange needs to know where to get the source event type. If the data file format is UBXEventBatchJSON (Batch event JSON payloads), then UBX reads the JSON field. If the data file format is TSV or CSV, the source event type can come from bucket name, file name, column name or column index.

Each <UBX_EVENT> element represents the mapping from a source event type to an Exchange event. It contains an <attributes> block which represents mapping for the attributes and an <identifiers> block which represents mapping for the identifiers.

Table 1. <UBX_EVENT_MAPPING>
Attribute Description Accepted Value Example
dataFileFormat Defines the input data file format.

UBXEventBatchJSON

TSV

CSV

 <UBX_EVENT_MAPPING dataFileFormat="CSV">
dataFileHeader Defines whether the CSV or TSV file contains a column header. True | False dataFileHeader=”true”
sourceEvent Defines where Exchange can get the event type in the source data.

{BUCKET_NAME}: Indicates that the bucket name is the source of the event type.

{FILE_NAME}: Indicates that the file name is the source of the event type, up to the first “_” character if present.

{COLUMN_INDEX}: Indicates that the column index number is the source of the event type. The column index number is defined in the attribute sourceEventColumnIndex. Use this value for CSV and TSV files where the column is referenced by index number.

{COLUMN_NAME}: Indicates that the column name is the source of the event type. The column name is defined in the attribute sourceEventColumnName. Use this value for CSV and TSV files where the column is referenced by column name.

String: It can also be a string value. There can be one String value only.

 sourceEvent="{FILE_NAME}"
sourceEventColumnIndex Defines the index number of the column for the source. <Number> sourceEventColumnIndex=”3”
sourceEventColumnName Defines the name of the column for the source event type value. <String> sourceEventColumnName=”event”
Table 2. <UBX_EVENT>
Attribute Description Accepted Value Example
sourceEvent Defines the source event name. This attribute looks up the event that is being mapped for the source event. <String> <UBX_EVENT sourceEvent="offerAccepted">
Table 3. <UBX_EVENT> child notes
Attribute Description Accepted Value Example
<code> Defines the Exchange event code. <String> <code>adImpression</code>
<channel> Defines the Exchange event channel <String>

<channel>Paid</channel>
<timestamp> Defines the mapping of the timestamp value. It converts the timestamp from the source format into format consumable by Exchange. Attributes:

sourceName: Defines the source timestamp field or column name.

sourceIndexDefines the source column index name.

sourceType: Defines the source type. Use timestamp.

sourceFormat: Defines the source timestamp format. For example “yyyy-MM-dd HH:mm:ss”, or “MM/dd/yyyy HH:mm:ss”.

 <timestamp sourceIndex="1" sourceType="timestamp" sourceFormat="yyyy-MM-dd HH:mm:ss"></timestamp>
<attributes> Defines the attribute mapping. It contains a list of <attribute> blocks. Each represents the mapping from a source attribute to an UBX attribute. Attributes:

sourceName: Defines the source attribute or column name.

sourceIndex: Defines the source column index.

Elements:

<name>: Defines the Exchange attribute name.

<type>: Defines the Exchange attribute type. The value options are String, Number, Boolean and Time (in ISO-8601 format).

<valueMap>: Defines the value mapping. It contains a list of <entry> blocks, each represents the mapping from source value to Exchange value.

Entry:

sourceValue defines the source value;

The node value defines the Exchange value.

 Maps the source attribute aa.productId to the Exchange attribute productid: 
<attribute sourceIndex="10">
  <name>subChannel</name>
  <valueMap>
    <entry sourceValue="1">web</entry>
    <entry sourceValue="4">web</entry>
    <entry sourceValue="2">video</entry>
    <entry sourceValue="3">social</entry>
    <entry sourceValue="10">social</entry>
    <entry sourceValue="6">search</entry>
    <entry sourceValue="7">email</entry>
    <entry sourceValue="8">app</entry>
  </valueMap>
</attribute>
<identifiers> Defines the identifier mapping. It contains a list of <identifier> blocks. Each represents the mapping from a source identifier to an UBX identifier. Attributes

sourceName: Defines the source identifier or column name.

sourceIndex: Defines the source column index.

sourceMinLengt: Defines the minimum length for the source identifier value. If the value doesn’t match this length, the identifier is not be included in Exchange events. This is to prevent identities from merging.

Element

<name>: Defines the Exchange identifier name.

<valueMap>: Defines the value mapping. It contains a list of <entry> blocks, each represents the mapping from source value to Exchange value.

Entry

sourceValue defines the source value.

The node value defines the Exchange value.

 Maps from source identifier cid to the UBX identifier connected_id: 

<identifier sourceName="cid" sourceMinLength="5">
  <name>connected_id</name>
</identifier>

Aggregated Events

For Exchange, aggregated events, such as cart purchase and cart abandonment events, are required to be sent with their corresponding item events (e.g. cart purchase item and cart abandonment item events) in a single event batch. Since in TSV/CSV files, one line represents a single event, we need a way to group multiple lines together to support the aggregated events. Such event groups can appear anywhere in the data file, however, events that belong to the same group must be together (sequential lines). Within the group, the first line should be the aggregated event, followed by the member (item) events. The member event line should have a dedicated column (defined by mapping header attribute memberEventColumn) to indicate this line is a member event (when the column value equals to what's defined by mapping header attribute memberEventColumnValue). 

For example, in the following TSV file. we use the column "memberFlag" to indicate the member event line. The line is considered a member event line when the value is set to "M".

  Identifier column Other attributes column sourceEventType memberflag
1     productView  
2     abandonment  
3     abandonmentItem M
4     abandonmentItem M
5     abandonmentItem M
6     abandonment  
7     abandonmentItem M
8     pageView  
9     productView  
10     abandonment  
11     abandonmentItem M

In the mapping file, we set the following in the <UBX_EVENT_MAPPING> element:

<UBX_EVENT_MAPPING dataFileFormat="TSV" sourceEvent="{COLUMN_NAME}" sourceEventColumnName="sourceEventType" memberEventColumn="{COLUMN_NAME}" memberEventColumnName="memberFlag" memberEventColumnValue="M" dataFileHeader="true">

When this data file is processed, line 2-5, line 6-7, line 10-11, each will be grouped together in separated event batches.


Upload data to the bucket

Now that you've created a bucket, and if you've created your mapping file, you're ready to add the mapping file and other objects to your bucket. An object can be batch JSON, TSV, or CSV files.

If you did not create a mapping file, then you can upload batch event JSON files only.

  1. Open the Cloud Storage browser in the Google Cloud Platform Console.
  2. In the list of buckets, click on the name of the bucket that you want to upload an object to.
  3. In the Objects tab for the bucket, either:
    • Drag and drop the desired files from your desktop or file manager to the main pane in the GCP Console.
    • Click the Upload Files button, select the files you want to upload in the dialog that appears, and click Open.

For more information on adding objects to your bucket, see Uploading objects.

Batch event JSON files

A batch event JSON payload is multiple events in a JSON payload as a series of event descriptions. Each event description must define the identifiers that Exchange can use to identify the specific individual who is associated with the event. For example, an email address or cookieId is a typical identifier. Each line must contain one complete event batch JSON payload.

Each event description must also specify the event code that indicates the type of event, a time stamp, and various attributes that describe the event. The event description can also identify the channel in which you detected the event. You can also define the channel at the batch level if all of the events in the batch are observed in the same channel.

Batch event JSON payload example:

This code cannot be formatted, you must use raw JSON.

TSV and CSV files

TSV and CSV files can be configured as they would normally, no special configuration is required withthe TSV or CSV files themselves. However, as stated above, you will need to create a mapping file for the content of the TSV or CSV files.

Example of a CSV file:

"DATETIME","USERID","USERAGENT_TYPE","USERAGENT_FAMILY","USERAGENT_DEVICETYPE","USERAGENT_OS","LANGUAGE","GEO_CITY","GEO_CITYCONF","GEO_POSTAL","GEO_POSTALCONF","GEO_AREACODE","GEO_REGION","GEO_COUNTRY","GEO_GMTOFFSET","GEO_DMA","QUERYSTRING" "2017-10-16 22:29:26","452fa06fd0bf55f4777f2c2033525eb7","SM-J700T","Chrome Mobile","Smartphone","Android","","los angeles","65","90009","35","310/424","CA","US","-25200","803","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:29:26","36d17521808c1452a7de76c5e38b2421","SM-G955U","Chrome Mobile","Smartphone","Android","","los angeles","60","90044","30","323","CA","US","-25200","803","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:31:9","f2743713a4f97075e6ea55ad755d7f28","Browser","Chrome","Personal computer","Mac OS X","","ft worth","65","76137","30","817/682","TX","US","-18000","623","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:32:12","5712ee8ecee794715a7bd3e9c3cbed1a","Browser","Chrome","Personal computer","Windows","","college station","70","77840","40","979","TX","US","-18000","625","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:31:10","362e5148306a6294ae92fc4f07ae16a4","SAMSUNG SM-G930A","Chrome Mobile","Smartphone","Android","","san marcos","60","78666","30","512/737","TX","US","-18000","635","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:32:14","14e6e35bf5c47fd540d9041f6dac9732","iPhone","Mobile Safari","Smartphone","iOS","","coral gables","65","33134","35","305/786","FL","US","-14400","528","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:29:28","cedb350ba0f334eeeba416156e03f196","iPhone","Mobile Safari","Smartphone","iOS","","van nuys","95","91405","90","818","CA","US","-25200","803","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:31:13","85331dbfbb736a9ffc84fb3d99531d2f","SM-G930P","Chrome Mobile","Smartphone","Android","","jarrettsville","90","21084","90","410/443/667","MD","US","-14400","512","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0" "2017-10-16 22:29:29","dc19b0b414773aa1aa5d2cd6e0236c0f","SAMSUNG SM-G890A","Chrome Mobile","Smartphone","Android","","moreno valley","65","92555","35","951","CA","US","-25200","803","p=XXX&g=Y&cpid=123&plid=456&stid=789&crid=101112&j=0"

Register your endpoint

To exchange data from Google Cloud Storage through Exchange, you must register Google Cloud Storage as an endpoint with Exchange for use in your Exchange account. You must have a valid license and access credentials for products and solutions that you register.

Register the Google Cloud Storage endpoint with the endpoint registration wizard on the Endpoints tab or through the navigation menu.

Google Cloud Storage can deploy business solutions on one or more servers or data centers. You must specify a deployment when you register Google Cloud Storage as an Exchange endpoint for your account.

Use the registration wizard to provide required address and credential information about your organization and Exchange account.

Exchange accepts the address and credential information as permission from you that the endpoint provider can interact with Exchange on your behalf. In turn, the endpoint provider requires credentials from you as permission that Exchange can interact with the endpoint on your behalf. During the registration process, you must provide the user credentials that the endpoint provider requires.

  1. On the Endpoints tab in Exchange, select Register new endpoint.
  2. Select Google Cloud Storage as the endpoint that you want to register and click Next.
  3. Provide the information that the endpoint provider requires as part of the registration request. To register the endpoint your will need to provide your:
    • Deployment
    • Service Account Credentials JSON

      Copy and paste the contents of the Service Account Credentials JSON file into this section.

    • Bucket name.
    • Mapping file name.
    • Optional: If your business needs require it, enter an Endpoint alias for your endpoint.

      An endpoint alias is a designation that differentiates one endpoint from another. Defining an endpoint alias is useful in cases where you have multiple endpoints with the same configuration but have different purposes. If you do not define an endpoint alias, one is generated for you.

    • Optional: Enter a description for the endpoint.

      By entering a description for the endpoint, you can provide better insight to the purpose or goal of the endpoint. Again, this is useful when you have multiple endpoints of similar configuration, but different purpose. If you do not define an endpoint description, then Exchange uses the default endpoint description.

  4. Click Register.

If you entered registration information into the registration wizard, Exchange transmits the information to the endpoint provider so that the provider can complete the registration. The endpoint displays on the Endpoints tab as Pending. When Exchange and the endpoint provider completes the registration, the endpoint status changes to Active.

If the registration did not complete successfully, the endpoint status is Failed. Contact Exchange Support for assistance.

Subscribe to events in Exchange

You must register the publishing and subscribing endpoints. As part of the endpoint registration, the endpoint provider registers the events that it can provide. Depending on the information that the provider supplies, Exchange can display a description of the event and related event attributes.

When you subscribe to events, you select an event and a destination for the event. You can select multiple events and multiple destinations.

  1. On the Events tab, click Subscribe to events. The Event subscription window displays.
  2. In the Select events column, select a provider > endpoint > event. Click details about the event.
  3. In the Select destinations column, select one or more endpoints to receive the event notification. The endpoint displays in the Pending subscriptions column.
  4. Review the pending subscriptions. You can change the subscriptions as necessary.
  5. Click Subscribe. Refresh the list of available event syndications.

When will Exchange pull event data from Google Cloud Storage?

Every 8 hours, or three times over a 24 cycle, Exchange looks into your Google Cloud Storage bucket to see if there is any new event data that has not been previously shared. If it finds new event data, it downloads it, maps the data by using a mapping file that you provide, and sends the data to Exchange. The event files are then moved to a processed folder in Google Cloud Storage. Exchange generates a report file for each JSON, TSV, or CSV file that was processed.

Related articles

community icon

Join the community

Ask questions, share best practices, and connect with fellow marketers.

Ask a question

Academy

Read how-to articles, guides, and more to help you on the path to success.

Learn more