Implementing a chatbot isn’t a quick-fire “set and forget” task. A chatbot is an essential part of the contact centre, working both with and alongside your help desk operators to deliver a smooth support experience.
Like human employees, then, a chatbot should be trained. Its performance should be monitored. It should have goals, and regular evaluations against those goals.
But it’s not a good idea to use the exact same metrics for your chatbot that you do for your team. Customers have different expectations of chatbots, and so should you.
So, what metrics should you apply as you seek to assess your chatbot? Here’s a list of the best chatbot metrics to get a true performance measure.
FAQs
Chatbots are to FAQ handling as bread is to butter; they just go together.
As such, the metrics around the FAQs your chatbot handles provide a great starting point for getting a clear idea of performance.
For example:
- What are the most common questions?
If a particular FAQ is being asked even more frequently than the others, it can point to an opportunity for improvement. For instance, maybe information needs to be more clearly displayed on your website, or there’s a noticeable minor issue with a product.
- Which FAQs lead to bounce or transfer?
If a question regularly leads to users bouncing or transferring away from your chatbot, it connotes a problem. Perhaps it’s a question you need to teach your chatbot to better handle, or it’s indicative of a larger, more complex problem that’s occurring too frequently.
Goal completion rate
Goal completion rate measures how often your chatbot succeeds in the core goal(s) you’ve set for it. It’s shown as a percentage — how many chats out of all / a set sample were concluded successfully.
Chatbots have varying criteria for success depending on deployment type. If you’ve opted for an FAQ bot, for example, ‘success’ is answering questions. Routing bots are successful by quick and efficient transfers, guide bots by finding key pages and information, and so on.
Measuring goal completion rate as one of your chatbot metrics is the quickest way to see whether your chatbot is working properly or not.
Bounce rate
The bounce rate for a chatbot measures how often customers quit the chat session.
A customer might bounce for any number of reasons, so this should be measured against other chatbot metrics such as goal completion rate.
At its simplest, a low bounce rate tells you that the chatbot is performing well. A high bounce rate suggests the opposite.
If customers are clicking away from the bot at a high rate — without the answers they wanted —it suggests customers are getting frustrated and the chatbot isn’t helping them.
Fallback rate
Your fallback rate is essentially a measure of every time the chatbot fails. It’s a count of any time a chatbot doesn’t understand a message or question, and it ‘falls back’ to the previous message it sent.
Regular chatbot fallbacks demonstrate that customers are asking for something your chatbot hasn’t yet been equipped to handle.
Fallback rate can be split into three separate categories/chatbot metrics based on how or why the bot fails:
· Confusion rate
Confusion rate refers to times when the chatbot doesn’t understand or have the ability to answer a message from a customer.
· Reset rate
Reset rate refers to fallbacks caused by the customer wanting to go back in the conversation to change their answer or mind about a choice they’ve made.
· Human takeover rate
The human takeover rate is a measure of how many chatbot fallbacks lead to humans taking over the chat. It’s also helpful to look at why a human agent needed to step in.
A high fallback rate means your chatbot needs more training or tuning.
Chatbot accuracy
Closely related to fallback rate is chatbot accuracy. This is one of the chatbot metrics specifically for AI/NLP/intent-based chatbots. These are the bots that use natural language programming and machine learning to work, rather than pre-defined rules and conversation paths.
Accuracy tracks how often a chatbot misunderstands a message (or indeed, fails and falls back.)
These types of chatbots work through machine learning and training. So, a low accuracy will show you that the chatbot needs more training data to work effectively. High accuracy, meanwhile, means it’s working well.
Interaction volume
Chat volume, or interaction volume, is a measure of the popularity of your chatbot. The more helpful the bot, the more popular it will be.
Often, interaction-based chatbot metrics are broken down into several separate measures.
· Total users
How many users in total has the chatbot helped overall or over a set period? You can track this over time to get a view of the growth of your chatbot adoption. This metric will also help you see the reach of your chatbot.
Additionally, in the case of AI-based chatbots, it gives insight into how much data the chatbot is being fed.
· New users
How many of the chatbot users over a set period are new (have never used the bot before)?
This is another metric that points to the growth of your chatbot and the reach of your business marketing efforts.
· Target audience session volume
Chatbot metrics aren’t all about the ability of your bot, but also the success of your marketing.
Target audience session volume is a measure of how many of your chatbot users are part of your target audience/market. So, you can tell if your chatbot is appealing to the customers you’re targeting.
· Retention rate
How many chatbot users have returned to the chatbot after the first conversation?
A successful, helpful chatbot will not repel customers. A good experience is one a returning customer may seek to repeat. So, measuring the retention rate of your chatbot is another way to ensure it is providing good experiences.
Starter message rate
The starter message rate is an entry on the chatbot metrics list that is best used at the start of your chatbot journey — and slowly given less importance.
It’s a measure of how many chatbot conversations/interactions are started by a message from the bot, rather than a user.
This is to give you an early idea of how receptive your online visitors are to interacting with a chatbot. Having a chatbot reach out is a good way to get users accustomed to the bot.
Over time, you’ll likely still use proactive messaging through your bot, and it’s still a useful metric to measure. However, user-initiated conversations should also grow — and they’ll be important to monitor using the other chatbot metrics.
Measure the right chatbot metrics moving forwards
Measuring chatbot performance is all about ensuring your chatbot is working well and adding value to your business. It’s about making sure it’s reaching the right people and it’s able to answer the questions it faces.
The right chatbot metrics for your chatbot use depend on the type of chatbot you have and the goals you set for it.
But this list is a comprehensive starting point for selecting the best metrics to measure chatbot performance. Whether you have an NLP bot or a flow-based bot, a sales bot or an FAQ bot, these metrics will give you a clear view of how well your chatbot is working.
Useful links
Which chat KPIs should you monitor?