In a startup, we wear many hats. Operations does a little bit of sales, marketing does a little bit of quality assurance, engineering contributes to the blog…and everyone takes pride in helping out with customer service. For many months, we had one person managing questions we received through our little Intercom chat window (you know, the one in the bottom right hand corner of your screen), connecting customers with the appropriate team member to solve their problems. For a small team like ours though, time is a limited and valuable resource. It isn’t always possible to connect people immediately because our Intercom facilitator also has to take meetings with clients, manage financials, and so forth (many hats!). Plus, manually keeping track of Intercom has become very time-consuming as we continue to grow. So with a little bit of nifty machine learning, we built a bot to manage Intercom more efficiently and improve the timeliness of our responses! In this tutorial, we’ll show you how to train your own IntercomBot using our customizable machine learning API, Custom Collections.
Data
As you might have guessed, the data we used to train this model consists of Intercom conversations. More specifically, we combined all the messages sent by the customer into a single piece of text and used it as the datapoint. We then labeled it with the name of the indico employee who was assigned to the chat at the close of conversation. To download your history of Intercom conversations, use the code shown here.
Training Your Bot
Before we go any further, have you set up your indico account yet? If you haven’t, follow our Quickstart Guide. It will walk you through the process of getting your API key and installing the indicoio Python library. If you run into any problems, check the Installation section of the docs. You can also reach out to us through that little chat bubble if all else fails. Assuming your account is all set up and you’ve installed everything, let’s dive in.
Go to the top of your file and import indicoio. Don’t forget to set your API key. There are a number of ways you can do it; I like to put mine in a configuration file.
import indicoio from indicoio.custom import Collection indicoio.config.api_key = 'YOUR_API_KEY'
Let’s define our training function. Note that I’ve skipped over some preprocessing, which you’ll find in the _reformat_example function (used to combine all messages sent by the customer into a single piece of text). When specifying the model domain, be sure to set it to topics — this will use a text feature representation from our topic classification algorithms that helps improve the quality of your Custom Collection for this task ______. Set batch_size to a small number so you can catch errors more easily.
def train_custom_collection(data_file, collection_name):
data = json.load(open(data_file))
# Join a list of messages into a single example
data = map(_reformat_example, data)
batch_size = 20
c = Collection(collection_name, domain='topics')
try:
c.clear()
except indicoio.IndicoError:
# model doesn’t exist, so we already have a clean slate
pass
for start_idx in tqdm(range(0, len(data), batch_size)):
messages = data[start_idx:start_idx+batch_size]
messages = filter(lambda x: x[0].strip() != "", messages)
c.add_data(messages)
c.train()
c.wait()
print "%s: %s" % (collection_name, str(c.info()))
return c
Adding .wait() will block until the training is complete, and .info() will check the status of the collection when training is complete.
Deployment
In order for our bot to work effectively, we need to feed it real time information. The most efficient way to do so is with a webhook — each time a new Intercom message is received, we run webhook.py. Let’s walk through each of the functions in the script.
Note that there are three different situations we need to deal with.
- New conversations (
_predict). For a new message from a new customer, we make a prediction as to who would be the appropriate team member to aid the customer with his or her enquiry. If the prediction meets our confidence threshold, the bot autoassigns the conversation. Otherwise, it just makes a suggestion as to who might be a good person to answer the question. - Manual assignment (
_add_data_to_collection). For those conversations where the prediction falls below the confidence threshold, one of our team members needs to manually assign the conversation to someone. We automatically add this datapoint to the training data, so we know that the person the conversation was assigned to is good at answering similar questions. This means that even when the bot fails, it allows opportunity for improvement — whenever we have to manually assign a conversation to someone, Custom Collection receives new data with which to retrain the model. - A reply from a customer (
_predict). For customers we’ve already spoken with, we combine their new message with all previous messages. The bot then makes a prediction on the combined text, like it does for new conversations.
Sometimes messages come in chat-style, like so:
Hello
First of all, I want to thank indico.io team much for providing great product.
I try to use the product using PHP but I always get error below: Parse error: syntax error, unexpected '[' in D:RMhtdocsindico.iovendorindicoioindicoio-phpIndicoIoIndicoIo.php on line 338
If a customer sends separate messages like this, the bot evaluates after each message. So when it sees “hello”, it’s unlikely to assign the conversation to a team member as it’s not confident. When it sees the next one, it may assign to someone on our Ops team, or it may still not feel confident enough to make a decision. Finally, after seeing the third message, it will assign to someone on our Engineering team.
Behind the Scenes: Transfer Learning
You might be wondering how it’s possible to create a customized machine learning model without having to build a neural network from scratch. Using a technique called transfer learning, we can take a deep neural network trained to solve one task (like topic classification or recognizing emotion in text), and tune it to analyze customer questions using limited training data, like we did in this exercise. The key benefit of transfer learning is that it enables you to enjoy the flexibility and accuracy of deep learning without having to pay the high upfront training costs.
Next Steps
So today we’ve got a bot that’s quite effective at directing customers to the correct team member best suited to assist them. This has improved our response times as the customer is no longer dependent on the availability of our human facilitator. Encouraged by these results, we see opportunity to go further. Today we’re generally only able to respond to enquiries between 11am-7pm EST (our office hours). That means there’s a lag in response time particularly for our international customers. While some questions require our engineers to look at specific pieces of code and therefore responses can’t be automated (yet), there are other common questions that we feel we can handle more efficiently. Our next phase will be to train our bot to take care of such questions so that at least some folks in other time zones will receive immediate responses. Stay tuned!
If you’re interested in other tools built with Custom Collections, check out these tutorials:
- Customized Topic Tags
- Fashion Matching
- Smart Security Cameras
- Image Similarity for Product Recommendations
Questions about indico’s Custom Collection API, got feedback on how to improve it, or looking to integrate it into your business? Reach out to us at contact@indico.io.
