The media is full of information about the use of Machine Learning in supporting implementations of any information systems. It seems that Machine Learning has turned out to be the solution to all the problems of traditional implementations and that it can significantly speed up implementations and increase their efficiency without engaging large amounts of internal resources.
Meanwhile, the truth is somewhat different. Depending on the application, Machine Learning works better or worse. There are areas in which Machine Learning is perfect for such areas as pattern detection: repeating customer behavior, facial image recognition, fingerprint recognition, speech recognition, analysis of frequently repeated actions performed by employees, etc.
Chatbot inquiries often ask about using Machine Learning to create a knowledge base. The idea roughly is that the customer provides records of conversations (in text or audio form) between call center agents and callers. Based on these records, the solution “learns” how to respond to particular questions from customers.
Machine Learning – requirements and limitations
Let us begin by explaining what is needed to make Machine Learning work well. First, it is necessary to prepare the right sets of information:
- training information set
- test information set
Based on the first set, the system will be “taught”. The second will be used to verify the correctness and additional tuning of the solution.
What should such sets contain? When creating a knowledge base, at least three groups of information must be included:
- obviously, the statements made by users and the answers given by the agent
- time
- interactions performed by the call center agent in external information systems
Let’s start with the first one – the set should be prepared in text form. In the case of hotline recordings, it is necessary to provide, apart from the voice, a transcription of the conversations. This is necessary if specific nomenclature, phrases, proper names are used in the industry jargon. Personal data remains a debatable issue – it is both a legal and practical aspect – the system “must” know what is the ontology of name, surname, place name or address data. Failure to properly label these elements can cause incorrect “learning” effects.
Time is also of the essence – some data changes over time (prices, commissions, conditions of services provided, etc.). Failure to take these changing factors into account can result in incorrect results. Suppose that in the training file 30 times the information appears that the commission on granting a mortgage credit is 3%. And the information that it is 2% appears only 2 times. Without considering the time function, Machine Learning will teach the system that the commission is 3% (statistically more frequent answers with this commission). Meanwhile, the correct answer is, of course, 2% (because the change occurred yesterday, for example, and is effective today).
And most importantly – what does an agent do while servicing a customer? Our analysis shows that up to 85% of customer queries require agent interaction with external IT systems (billing systems, e-banking systems, ERP/CRM systems, etc.). In practice, no one collects such information (i.e., what the agent did, what system they queried, what function they used to answer the customer’s question). It is therefore necessary to prepare an appropriate matrix:
user statement – agent response – activities in IT systems – time.
It is possible to use appropriate software tracking the user’s actions (the question is, however, who will allow it to be installed, for example, in a bank?). Manually describing hundreds of thousands of interactions is also pointless, because at the same time we will build a knowledge base “from scratch” and the lack of the above information will cause the chatbot to handle at most 15% of interactions correctly. Therefore, it seems pointless to create such a solution, especially since you will have to clean manually the remaining 85% of the “automatically” created knowledge base.
Experience shows that the process of cleaning the knowledge base created on the basis of Machine Learning is as time-consuming (and often even more) as preparing a knowledge base from scratch based on rules. In addition, such a knowledge base (based on rules) is fully deterministic and gives 100% certainty – what answer will be given when a business rule is met. It should also be remembered that solutions based on Machine Learning never give 100% certainty about the answers given. Their operation is somewhat unpredictable. Of course, after a longer period of time the solution works well even in 99% of cases, however, until this time and proper training of the solution the answers will be a bit chaotic as given by an untrained agent.
Therefore, can Machine learning be well suited to the implementation and creation of knowledge base? In our opinion the rule-based approach is much more efficient (in terms of time and quality).
Machine Learning – usage
What can Machine Learning be used for when it comes to chatbot technology?
Speech recognition
The first area, of course, is speech recognition. Here it is practically impossible to make a good speech recognition system without Machine Learning. Even in existing speech recognition systems there are always areas that can be improved:
- adaptation of the system to the specific industry
- adaptation of the system to the industry-specific nomenclature, especially the proper names used
- adaptation to the language of interlocutors (e.g. introduction of expressions in dialects)
In the above cases, it is useful to use recordings from the Call Center. Obviously, the recording alone is not enough; what is more, it needs to be transcribed into text so that the system can “learn” the new vocabulary. Such transcription may be done partially automatically, but it must be verified by a human.
Chatbots testing
Machine Learning is well suited for automatic testing of the knowledge base. Editing and configuration of the knowledge base is a process that consists in such a parameterization of the system that it is able to respond to any formulated question taking into account the context. When asked about the cost of a mortgage loan, a user may as well ask “how much will the mortgage cost me” as ask about “the APR for a home loan”. Expanding and updating the knowledge base exposes us to configuration errors and knowledge base conflicts. Manual search for errors in a database with thousands of facts and tens of thousands of rules is almost impossible. Therefore, the ideal solution is to run an automatic tester, which, based on an appropriate test set, will verify the correctness of the solution operation and indicate the places where corrections should be made.
When using Machine Learning to test automated systems, the files to be prepared are limited to test sets and can be simpler in structure (just sample questions and the correct answer plus possibly context).
Detecting Trends
Another area in which Machine Learning can be used are statistics and reports. Solutions that predict trends and detect deviations from the norm are perfect here. For example – we have an appropriate extensive knowledge base that is divided into relevant categories (e.g., products, service, complaints, online, etc.). Machine Learning can “discover” trends (and more importantly deviations from them) regarding the number of queries from users about specific categories. Imagine that we are a provider of certain products. One of the categories in the knowledge base we have created is complaints. Suppose that the system “discovered” that every weekend the system receives 10 complaints related to our products. However, one Saturday the number of complaints exceeds the number 100 is much more than the trend, because the trend says 10 on the weekend. In this case Machine Learning can detect a deviation from the norm and alert the appropriate services that something bad is happening. Even more interesting are the analyses when we combine the results of conversations conducted by the automated system with other systems of the enterprise.
Summary
Machine Learning is useful but also requires work in preparing the batch files to work. You need to consider well whether the effort to prepare training and test sets will exceed the time needed to configure the solution manually. In the case of chatbots, it seems that the greatest benefits lie in areas of:
- improving speech recognition
- automating knowledge base testing processes