Programs are here to help you maintain hands-free customer engagement. However, you still need to keep an eye on the program's performance and be aware of any unexpected events.
Will you be able to react in time if things don’t go as you planned?
On the other hand, what if things go better than expected? Can you spot an opportunity quickly enough?
It's hardly possible to spot unusual events in your program instantly. Anomaly detection relieves you from this necessity and notifies you whenever a program requires special attention.
What is anomaly detection
Anomaly detection in programs is based on an AI-powered model that monitors multiple program metrics to tell you when the unexpected events happen and show you where to look. There's no need to run a report or wade through message logs. Plus, the longer your program runs, the more the model learns about what to expect and when to grab your attention. Everything fits into the Metrics tab in the Program insights panel.
From the initial summary view, you can dive deeper. Each metric summary links to an expanded view that reveals additional detail about the performance of that metric over time. Any unexpected program behaviors – anomalies – are highlighted in charts that plot the history of each metric. The Metrics tab displays results up to the previous 450 days (30 days is a default).
How anomaly detection works
The program anomalies evaluation model reviews program activity nightly. The model calculates expected results, compares them to the actual results, and highlights unusual behavior.
As a program runs, the anomaly detection model continually defines a range of expected values for each metric, based on an analysis of previous program performance and customer responses. The upper and lower expected values define the bounds of the metric. Within these bounds, the model defines a specific expected value that is most likely to occur. Trends for the bounds and expected values become apparent over time.
The model displays dates and times in GMT.
Enable anomaly detection notifications
You can automatically receive a notification when an anomaly is detected. Go a step further and also enable email notifications to receive anomaly notifications by email.
Note: You must enable anomaly detection emails for each program individually.
- Open a program and click Metrics.
- Turn on Notify me about anomalies. You'll start receiving in-app notifications.
- Enable email notifications if you want to stay in the loop outside Campaign.
- Go to Notifications > Settings.
- Turn on Email notifications. We'll use the email address associated with your profile to alert you to anomalies that occur.
Availability and stored data
To display program metrics and take advantage of anomaly detection modeling, your organization must meet the following criteria:
- Provisioned for Performance Insights
- Not provisioned for Transact
- Uses the new navigational experience
- Actively using the latest programs canvas with programs that contain email steps only
Contact client support to request access to program anomaly insights for your organization.
Anomaly identification and notification data is hosted exclusively in a United States data center. The stored information consists of user notification preferences and aggregate history for program transmission and response activity. No personal data is ever required or stored.
The programs anomaly detection model watches several key metrics that are reliable measures of customer engagement. Each metric type indicates summary results observed for all emails currently running in the program.
- Emails sent is the sum of all email messages that were sent during the previous day by all emails currently running in the program.
- Opens (unique) identifies the total number of emails associated with the program that have been opened at least once per email recipient. If a recipient opens an email more than once, the system records a single unique open.
- Clicks (unique) identifies the total number of email recipients who have clicked at least one link in any of the emails sent in the program. If a single recipient clicks on more than one link in the same email, the system records one unique click. If the same individual clicks one link in three different emails, the system records three unique clicks.
- Hard bounces refers to emails that are undeliverable because of a permanent characteristic of the message or delivery address, such as a misspelled or non-existent email address.
- Soft bounces refers to emails that are currently undeliverable because of a temporary characteristic of the message or delivery address, such as a full inbox or system delays.
- Unsubscribes indicates the total number of requests to stop receiving emails sent by any of the contacts currently running in the program.
Monitor email deliverability
Your program sends an email with weekly promotions to all new customers and other customers who qualified for a Preferred account in the past week. Audience updates and new email content are loaded every Saturday night. The program starts sending emails on Sunday to have emails in the inboxes by Monday morning. This week, the update runs as scheduled.
On Monday morning, you receive a system notification that there’s been an anomaly in your weekly promotion program. Better check this out.
When you open the program and scan the Metrics tab, the Hard bounces metric indicates an anomaly. There’s been a spike in the number of hard bounces, beyond the highest expected response.
Click the metrics to dive deeper. The diagram compares the current metrics to expected values.
The alert makes it clear that this week’s emails aren't getting through.
After reviewing the database, you’ve identified the culprit. The audience data entered on Saturday contained a high number of invalid email addresses. Many of them contained typos and random characters. It will take you a couple of hours to fix. Later, you’ve fixed the addresses, updated the audience database, and the emails are going out again.
Without those analytics proactively telling about unexpected behaviors, it would be less likely to notice the bad signs until it was too late to resolve the issue. Thanks to the analytics and anomaly detection, you can spot problems and address them instantly.
Demonstrate ROI with anomalies
We often think of anomalies as bad. In fact, sometimes anomalies are a good thing.
You have been running an online promotion for a while, which has been generally successful. Related sales have been good but have started to trend down in the past few months. You’re worried that the promotion might be getting stale, so you try investing in a major on-site event for the brand at its flagship store.
A couple of weeks before the event, you begin a program that targets customers that qualify as loyalty members but have not signed up yet. A big jump in customer interest will justify the cost. Emails to this audience promoting the event have been flowing steadily.
The event goes off on Saturday without a hitch. Multiple potential loyalty club members attended and are now entered in the audience database. Your program is configured to send a follow-up email to everyone who registered.
On Monday, you receive a system notification that an anomaly has been detected in the event follow-up program. You open the Metrics tab in your program to see what’s happening. The Clicks (unique) metric grabs your attention.
There is a big spike in the number of clicks in the email that the program sent. You click Clicks (unique) to dig a little deeper into the results.
In the metric detail, the anomaly is even more striking. Last night’s email clearly exceeded the bounds expected by the program analytics. The anomaly detail reveals that nearly everyone who registered at the event also engaged by responding to the follow up email.
The anomaly detected in your program will help you demonstrate a positive ROI by clearly showing that the money spent on the event directly resulted in a massive increase in customer interest and engagement.
Uncover contact fatigue
If you are running a program that's working well, it’s natural to want to tweak it to build on the success. Maybe you can change the greeting, try a new subject line, configure the program to send email more frequently, or add a new email step.
How much is too much? The analytics and anomaly detection gives you a way to quickly and easily see the results of your changes. The Unsubscribes metric can be a particularly effective indicator.
Let’s say that you run a program that sends monthly emails to your loyalty club customers. Response rates have been good, but they don’t quite measure up to what your company is looking for. You try a bolder subject line to boost the open rate. Within a couple of days, unique clicks go up a bit, so it appears that your customers are paying attention.
Next, you try switching from monthly emails to weekly emails, expecting that more is better. However, soon after you make the change, you get a system notification that an anomaly has been detected in the Unsubscribes metric for your loyalty program. You open the Metrics tab in the program to learn more.
The metrics indicate an unexpected spike in unsubscribes.
Looking at the Unsubscribe details shows the effect of the change in email frequency. Lots of your loyalty customers are suddenly opting out of future communications. It inhibits the success of future campaigns and you might even be damaging your sender reputation. Now that you’ve seen the effect of your recent change, you determine that it would be a good idea to go back to contacting your loyalty customers less frequently.
Fortunately, you discovered this problem in time. Previously, you might have relied upon weekly or month-end reports. Now, you can depend on anomaly detection to provide more timely alerts and proactively respond to issues on the go.