Blog
Make New Friends, but Keep the Old: There’s More to NLP Methods Than Generative AI
Large language models (LLMs) have become the star of the natural language processing (NLP) world. Their ease of use enables businesses to enhance efficiency and make faster strategic decisions. However, NLP methods are more than just LLMs—algorithms for topic modeling, language parsing, entity recognition, sentiment analysis, and other techniques have a rich history before the advent of generative AI.
So as the newest and most robust addition to NLP, is an LLM always the best tool for the job at hand? To fully leverage generative AI, it’s essential to understand the NLP technologies—like text extraction—that enhance it, and recognize when less costly NLP techniques are best used as a stand-alone technology. Traditional NLP and generative AI each have unique strengths and appropriate use cases. This article explores scenarios where more traditional NLP methods offer efficient, cost-effective solutions, and instances where combining NLP with generative AI platforms like iX Hello™ is optimal.
Traditional NLP Use Cases
1. High Precision and Accuracy Tasks:
Traditional NLP techniques, such as rule-based classification systems, deliver precise and accurate results for well-defined tasks like named entity recognition, part-of-speech tagging, and sentiment analysis.
2. When Repeatability Is Important:
Older existing NLP methods and algorithms often behave more predictably in regard to the targets of their tasks, while the output from LLM prompts may vary unexpectedly. When ongoing reporting and trending are a key requirement of the task, using traditional NLP algorithms often leads to more stable outputs.
3. When Interpretability Is Important:
Traditional NLP methods are more interpretable and easier to understand than complex generative AI models, crucial in sectors like healthcare and finance where decisions must be explainable and auditable.
4. For Tasks Requiring Structured Output:
Tasks that necessitate consistent structured output—such as extracting specific information or classifying text into predefined categories—often are better suited to traditional NLP techniques like regular expressions, rule-based systems, or machine learning classifiers.
5. When Computational Resources Are Limited:
Generative AI models, especially LLMs, are resource intensive and often costly to place into production. When resources are limited, traditional NLP methods are more practical and cost effective.
Traditional NLP and Generative AI Working Together
Generative AI excels in natural language generation tasks, such as text completion, summarization, translation, and open-ended question answering. It also handles complex and nuanced language understanding tasks, like capturing context and generating human-like responses.
Recent Concentrix research revealed that despite interest in generative AI, only a quarter of enterprises with captive CX operations had already begun implementing the technology. Most enterprise organizations are still using older AI technologies (e.g., machine learning), and many industries, like banking, finance, healthcare, are slow to invest in tech advancements.
Organizations need to develop a strategy for evolving their AI mix over time. A good start is looking at how traditional NLP and generative AI can work together.
1. Preprocessing and Analysis:
Analyzing and parsing text data before it goes into generative AI models can be done using traditional NLP techniques. This improves the quality and consistency of the text output.
2. Combining Context and Classification:
For use cases such as using generative models to classify unstructured text, combining traditional NLP and generative AI provides an approach that leverages the strengths of both. Traditional NLP excels at repeatable classification, while LLMs often provide a deeper understanding of context than is typical with historical approaches.
3. Evaluation and Validation:
Businesses often need to ensure content generation follows their brand rules and regulations. Traditional NLP can evaluate and validate generative AI outputs to ensure the output meets the intended purpose for the specific business. Metrics such as perplexity, semantic similarity, BLEU (Bilingual Evaluation Understudy) score, along with sentiment analysis or named entity recognition provide the best results.
4. Textual Extraction:
Unstructured data, from sources such as social media feeds, online customer reviews, or legal documents are use cases where rich text data exists but often needs to be extracted to be useful in LLMs. Deploying traditional NLP extraction techniques allows for consistent, accurate, and cost-effective extraction of key focus areas, enabling the LLM to focus on use and generation of output from those areas.
5. Interpretability and Explainability:
Being able to support and understand how models work, and being able to provide information on why decisions are made are critical in certain use cases. Traditional NLP can analyze LLM-generated text to provide both of these, identifying key language features and patterns that contribute to the model decisions.
Determine Your Path Forward
Before tackling the integration of traditional NLP with generative AI, as we have done with our iX Product Suite, you must take into account the task requirements, your available data, and your desired outcomes. The answers to these questions will help determine your path forward. Experimenting with different combinations will help find the balance needed to obtain the best possible results for your specific use case. Whether you’re looking to improve current processes or develop new ones for increased automation and superior customer experiences, Concentrix can help.
John Georgesen
Vice President, AI/Advanced Analytics
