A large insurance company recently came to Indico with a story that is all-too-familiar among firms trying to implement rule-based approaches to handle unstructured documents. The company was looking to automate a process that involved all sorts of financial documents. It partnered with a consulting firm, ultimately paying millions for a small army of engineers to spend months writing countless rules in an attempt to account for every imaginable variation in the documents.
It worked – briefly. But not long after the consulting engagement ended, the automated process broke down. Why? Because it’s virtually impossible to account for every variation you may encounter when dealing with unstructured content–no matter how hard you try, or how much you spend.
That is the unvarnished truth when it comes to purely rule-based process automation software, and it’s a key reason templating tools will only get you so far. They are great at processing highly structured documents, where the same information is in the exact same place every time. But they don’t fare well with unstructured content – such as the various statements and financial documents inherent in the aforementioned insurance process automation example.
Optical character recognition: no panacea
You may hear that optical character recognition (OCR) technology is a solution to this issue. That’s not actually the case. OCR is simply machine learning technology trained to convert scanned documents, often in PDF format, into machine-readable text.
That’s all well and good, but what happens once the scan has been converted into a readable document–how do you extract the information you’re looking for? Most OCR products out there offer to produce rule-based templates built from a sample set of your documents. That’s known as an OCR templating solution. For that to work, you need to know exactly where the text you’re after is located within each document. In that respect, it’s hardly automation at all, because it requires enormous manual effort to come up with the hundreds or thousands of rules required to make it work.
Let’s say you want to automate a process that involves taking financial data from a PDF and putting it into a spreadsheet. If all the statements are from the same financial institution and all the data you’re after is in the same place on every single statement, then an OCR templating approach may be a viable solution.
But as soon as you introduce statements from another institution, or even statements from the same institution that vary from the norm, you now need a new set of rules or templates to handle those situations. It’s easy to see how coming up with all the required templates can quickly become unwieldy – and costly.
RPA challenge: accounting for variation
On top of the requirement that the data be in the same place each time, you’ve also got to consider how the same information may be presented in different ways. Consider the date “Jan. 1, 2020.” That could be rendered in multiple formats, including:
- 1/1/20
- 01/01/20
- 01/01/2020
- 1-1-20
- 01-01-20
- January 1, 2020
- And on and on
To successfully automate a process using a templated approach means coming up with rules that account for each of those possibilities. Even if you succeed in getting a system working initially, chances are it won’t be long until something new comes along that throws a wrench into the works. Or maybe the automated process works on a small scale, such as in a proof-of-concept test, but quickly unravels in production.
Benefits of intelligent process automation
A better approach is to apply natural language processing (NLP) combined with transfer learning to enable intelligent process automation (IPA). Because NLP models are self-learning, they are able to understand context. That means the model looks at surrounding information to predict that a date format it hasn’t seen before is still most likely a date, no matter where it appears in the document.
When you apply NLP tools after OCR, you can now create an automated process that really can “read” text even from unstructured documents and make sense of it – without requiring anyone to write hundreds of rules. That’s truly scalable, intelligent process automation.
To learn more about how IPA works in practice, and how it differs from templated approaches, check out the Everest Group white paper, Intelligent Document Processing for Unstructured Documents.
