Building a Claims Agent using n8n, OpenAI, and Outlook (with vision and parallel tool calling)
Your employee of the month is a robot, Part 2
With companies like Dropbox and UPS laying off workers citing efficiency gains from AI, the trend has accelerated since the introduction of advanced Large-Language Models (LLMs) last year. Microsoft announced earlier this month their plan to invest $2.9 billion over the next two years. Sam Altman, CEO of OpenAI, is reportedly seeking $7 trillion worth of capital for his ventures.
To ensure we’re not left behind, let’s explore how we can put these advanced language models to practical use. In this article, I’ll guide you through building a customer support robot using n8n, Microsoft Outlook, and OpenAI, using multi-modal and parallel tool calling features.
If you missed Part 1 of this series, I use n8n with OpenAI and ElevenLabs to build a talking enrollment agent robot. You can find that article here.
Identifying the problem
I’d be a naughty product manager if I didn’t start with a customer pain point. So, before I get ahead of myself, let’s come up with a hypothetical scenario for an e-commerce business:
You work on a product team for an online business selling a variety of books. The CEO has set a company-wide goal to automate away all repetitive and manual tasks. At an all-hands, you throw out a suggestion to hire your first robot! The CEO agrees to your suggestion and asks for a proof-of-concept.
After a few weeks of user interviews, process mapping, and discussions with your team you’ve picked a narrow use case as a candidate for your first automation:
- Despite stellar customer reviews, 1% of orders arrive damaged. As the business grows, this 1% has become a back-office nightmare. The process involves handling inbound customer emails, verifying damage from photos, checking warranties, making decisions, and responding to customers.
Building the workflow
N8n is an open-source low-code workflow automation tool that connects nearly 1000 services and features via a drag-and-drop canvas. It’s ideal for rapid prototyping and systems thinkers.
As we set up our customer support robot, n8n will orchestrate the entire process, including:
- Collecting order information from customers
- Analyzing photos for defects
- Checking warranties
- Responding to customers
We’ll leverage OpenAI’s capability for calling multiple tools in parallel, guided by a detailed ‘process manual’ in the form of a system prompt.
Here’s what the entire workflow looks like:
Fetching mail from Outlook
Let’s use Microsoft Graph to access and send emails through Outlook, supported out-of-the-box by n8n.
Connecting to Outlook
To allow your n8n instance to connect to Outlook, we’ll need to set up a credential within n8n. I won’t cover this in detail since you can find those instructions on the n8n website.
On a high level we’ll need to:
- Set up an Azure account (if you don’t have one)
- Register your application in the Azure portal
- Connect n8n Microsoft Outlook Oauth2 API credential to your Azure application
Email fetching branch
At the start of our workflow, we use a Schedule Trigger that will fetch emails from Outlook on a regular cadence. We also apply a filter on the query so that we’re only fetching relevant emails.
To apply the filters in the Outlook node, add two filters:
- Folders to include: Inbox
- Filter query: `isRead eq false and (categories/any(c:c eq ‘robot’))`
The “eq ‘robot’” portion of filter 2 will only fetch emails that have been categorized as ‘robot’. This means we can manually categorize inbound emails for the robot to pick up, which is useful during testing.
Here’s our Outlook node in detail:
{
"meta": {
"templateCredsSetupCompleted": true,
"instanceId": "{instance_id}"
},
"nodes": [
{
"parameters": {
"operation": "getAll",
"returnAll": true,
"output": "raw",
"filtersUI": {
"values": {
"filters": {
"custom": "isRead eq false and (categories/any(c:c eq 'robot'))",
"foldersToInclude": [
"{id_of_inbox}"
]
}
}
},
"options": {}
},
"id": "{node_id}",
"name": "[MS] Get Messages",
"type": "n8n-nodes-base.microsoftOutlook",
"typeVersion": 2,
"position": [
-5040,
880
],
"credentials": {
"microsoftOutlookOAuth2Api": {
"id": "{credential_id}",
"name": "Outlook Account"
}
}
}
],
"connections": {},
"pinData": {}
}Reasoning with tools
To set up the LLM we need to define the tools and the process manual to pass to gpt-4-turbo.
Defining the tools (functions)
As emails arrive, we pass the content of the email along with the tool definitions so that our robot can reason which functions to call in parallel to get the job done.
Let’s define three functions for our robot:
- review_photo-reviews the customer’s photo for defects
- verify_warranty-checks to see if the product is still under warranty.
- email_customer-sends a reply email to the customer
const tools = [
{
"type": "function",
"function": {
"name": "review_photo",
"description": "Reviews the customer's photo for defects.",
"parameters": {
"type": "object",
"properties": {
"photo_link": {
"type": "string",
"description": "URL of the customer's uploaded photo."
},
},
"required": [
"photo_link"
]
}
},
},
{
"type": "function",
"function": {
"name": "verify_warranty",
"description": "Checks to see if the product is still under warranty.",
"parameters": {
"type": "object",
"properties": {
"order_id": {
"type": "string",
"description": "The customer's order ID."
},
},
"required": [
"order_id"
]
}
},
},
{
"type": "function",
"function": {
"name": "email_customer",
"description": "Sends a reply email to the customer.",
"parameters": {
"type": "object",
"properties": {
"message": {
"type": "string",
"description": "Message to the customer."
},
},
"required": [
"message"
]
}
}
}
];
return {tools};Process manual
Next, we need a manual detailing the steps for the process and some rules to keep the LLM on-task.
Act as a Customer Support Agent. The customer has sent you their order ID,
shipping address, and a photo of a defective product they recently purchased.
Your goal is to walk through this entire process with the customer.
Steps to accomplish your task:
1. Verify that the photo shows defects (function: review_photo)
2. Verify that the product is still under warranty. Products are under
warranty if purchased within the past 14 days (function: verify_warranty)
3. Decide if a replacement should be shipped to the customer - replacements
are offered if a valid order ID is given, it's covered under warranty,
and the photo review shows damage to the product
4. Email your decision to the customer (function: email_customer)
5. Terminate
Rules of engagement:
- Stick to the procedure and don't invent your own steps
- Stick to your role, and politely decline off-topic questions
- Do not output messages, only call functionsBuilding the functions
Now that we’ve prepped the LLM with the necessary definitions, it’s time to build out our functions.
function 1: review_photo
The goal of this function is to use gpt-4-turbo with vision to check if defects are detected in the photo uploaded by the customer. First, we download the image, then we encode it into base64 and finally, we pass it to OpenAI.
The system message to guide the vision model:
Act as a product defect detector for an e-commerce company that sells a
variety of books. The user has sent you a photo of a defective product.
Respond in JSON, referring to the following example:
{
'isDefective': true
'reason': 'The book shows visible damage on the cover, it seems to be torn.'
}After running a function the response needs to be passed back to OpenAI to help it decide if it should call additional tools. I’ll cover this in a later section.
function 2: verify_warranty
This function will pull a record from the database and calculate the order age. Later, our robot will use this value to determine if the order is still covered under warranty based on the instructions in the process manual.
We use an HTTP request to call our orders database. This could be an API hosted by Amazon, Shopify, or a proprietary data source.
function 3: email_customer
For our last function, our robot sends an HTML email response to the customer. We do this with an HTML node to format the content and an Outlook reply: message node to send the reply email.
Recursive tool calling
Calling tools in parallel is a recursive process. After we call a function we need to pass the result back to the LLM so that it can decide if it needs to call additional tools.
It’s important to take precautions here as there is a chance we get caught in a loop, driving up our OpenAI bill. You may want to consider adding a counter to break the loop. In my testing, an improperly written process manual was enough to confuse our robot into a loop.
For a detailed overview of parallel tool calling visit the OpenAI website.
Example execution
Now that our robot is constructed let’s take a look a an example execution of our workflow:
inbound customer email:
Hi there,
I recently purchased a card book and noticed it’s water damaged.
order ID: 102–8434057
address: 1165 Angelina St, Austin, TX 78702, United States
photo link: https://drive.usercontent.google.com/download?id=1BuhAlLRHV6RHLAwq_-o2uRykYR4u8Txd&authuser=0
Matt
robot’s response:
Dear Matt,
Thank you for providing the details and photo of your card book. Upon reviewing the photo, we confirmed that the book is indeed water damaged. However, our records indicate that this product was purchased a significant time ago and is no longer under warranty.
Unfortunately, as it falls outside our warranty period, we are unable to offer a replacement or refund for this item.
We apologize for any inconvenience this may cause and appreciate your understanding.
Best regards,
Customer Support
Closing thoughts
This article shares a very basic version of a customer support robot where I focus on building a recursive tool calling workflow with n8n that uses the multi-modal capabilities of gpt-4-turbo. If you plan to implement this in production, you’ll need to set up a memory system so that the robot can maintain conversation threads with customers. If you’re adventurous enough, you can add additional tools and build out a longer process manual to suit your needs.
I’ve been running a more complex robot in production and so far it’s been highly reliable and nothing short of amazing. I plan to continue to expand the toolset to handle other back-office automation tasks and will continue to share my journey here.
Now that you’ve created your first robot employee, it’s time to show the CEO what it’s made of!
Project Resources
If you’re inclined to test or improve on this workflow, here’s the project GitHub repo. You’ll need to set up the scaffolding around this workflow like adding your keys and credentials.
Follow me!
If you’ve made it this far, why not give this article a few claps 👏, comment, and follow? You can also join me on this journey at these coordinates:
