Building a FAQ Chatbot with TensorFlow, Fuzzy Matching, and GloVe

In this article, I’ll walk you through the creation of a sophisticated FAQ chatbot that combines TensorFlow, fuzzy matching, and pretrained GloVe embeddings. This chatbot is designed to provide accurate responses to user queries by utilizing a multi-layered approach that includes exact matching, fuzzy logic, and machine learning.

1. Introduction: Why Build a FAQ Chatbot?

A FAQ chatbot can significantly enhance user experience by providing instant answers to common questions, reducing the load on customer support, and improving accessibility. By integrating advanced techniques like fuzzy matching and machine learning, we can create a chatbot that not only responds to exact queries but also understands variations in user input.

2. Loading and Preprocessing the Data

We start by loading the FAQ data from a CSV file. The data typically includes a list of questions and their corresponding answers. For our chatbot to process this data effectively, we need to clean and preprocess the text by converting it to lowercase, removing stopwords, and trimming extra spaces.

import pandas as pd
from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS as stopwords
# Load the CSV file with "|" delimiter using the 'python' engine
faq_data = pd.read_csv('data/faq_data_for_saas_company.csv', delimiter='|', engine='python')
# Data Preprocessing: Lowercasing, removing stopwords, and extra spaces
def preprocess_text(text):
    words = text.lower().strip().split()
    words = [word for word in words if word not in stopwords]
    return ' '.join(words)
faq_data['Question'] = faq_data['Question'].apply(preprocess_text)
faq_data['Response'] = faq_data['Response'].str.lower().str.strip()

3. Creating a Dictionary for Exact Matching

To quickly respond to queries that exactly match the questions in our dataset, we convert the preprocessed questions and responses into a dictionary. This allows the chatbot to instantly return a response if the user’s input matches one of the pre-existing questions.

# Convert questions to a dictionary for exact matching
faq_dict = dict(zip(faq_data['Question'], faq_data['Response']))

4. Tokenizing Text Data

Tokenization is the process of converting text into numerical data that can be fed into a machine learning model. We use the Tokenizer class from TensorFlow’s Keras API to convert the questions into sequences of integers. We also use LabelEncoder to encode the responses into numerical labels.

from tensorflow import keras
from sklearn.preprocessing import LabelEncoder
# Tokenize the text data
tokenizer = keras.preprocessing.text.Tokenizer()
tokenizer.fit_on_texts(faq_data['Question'].tolist())
train_sequences = tokenizer.texts_to_sequences(faq_data['Question'].tolist())
train_sequences = keras.preprocessing.sequence.pad_sequences(train_sequences)
# Encode the labels
label_encoder = LabelEncoder()
encoded_labels = label_encoder.fit_transform(faq_data['Response'].tolist())

5. Loading Pretrained GloVe Embeddings

GloVe (Global Vectors for Word Representation) embeddings are pretrained word vectors that capture semantic relationships between words. By loading these embeddings into our model, we can provide it with a richer understanding of language, which helps in predicting the correct response to user queries.

import numpy as np
def load_glove_embeddings(glove_path, word_index, embedding_dim=100):
    embeddings_index = {}
    with open(glove_path, 'r', encoding='utf8') as f:
        for line in f:
            values = line.split()
            word = values[0]
            coefs = np.asarray(values[1:], dtype='float32')
            embeddings_index[word] = coefs
    embedding_matrix = np.zeros((len(word_index) + 1, embedding_dim))
    for word, i in word_index.items():
        embedding_vector = embeddings_index.get(word)
        if embedding_vector is not None:
            embedding_matrix[i] = embedding_vector
    return embedding_matrix
# Update this path to where your GloVe file is located
glove_path = 'data/glove/glove.6B.100d.txt'
embedding_matrix = load_glove_embeddings(glove_path, tokenizer.word_index)

6. Building the Neural Network Model

The neural network model is the core of our chatbot. It processes the tokenized user queries and predicts the most appropriate response. Let’s dive deep into the structure of this model.

model = keras.models.Sequential()
model.add(keras.layers.Embedding(input_dim=len(tokenizer.word_index) + 1,
 output_dim=100,
                                 weights=[embedding_matrix],
                                 input_length=train_sequences.shape[1],
                                 trainable=False))
model.add(keras.layers.Bidirectional(keras.layers.LSTM(128, return_sequences=True)))
model.add(keras.layers.GlobalMaxPooling1D())
model.add(keras.layers.Dense(64, activation='relu'))
model.add(keras.layers.Dropout(0.5))
model.add(keras.layers.Dense(len(np.unique(encoded_labels)), activation='softmax'))
# Compile the model with a lower learning rate
optimizer = keras.optimizers.Adam(learning_rate=0.0001)
model.compile(optimizer=optimizer, loss='sparse_categorical_crossentropy', metrics=['accuracy'])

Model Breakdown

1. Sequential Model : The Sequential model in Keras allows us to stack layers sequentially, which is perfect for our straightforward architecture.

2. Embedding Layer :
** Purpose :** Converts input tokens (words) into dense vectors, using the pretrained GloVe embeddings.

** input_dim** : Size of the vocabulary plus one (for padding).

** output_dim**: Dimension of the embedding vectors (set to 100).

** weights** : Initialized with GloVe embeddings, providing rich, pretrained representations.

** trainable=False** : Keeps the GloVe weights fixed during training, preserving their pretrained quality.

3. Bidirectional LSTM :
** Purpose** : Captures dependencies in the sequence in both forward and backward directions, which is crucial for understanding context in language.

128 units: Defines the number of LSTM cells, determining the capacity of the model to capture information.

** return_sequences=True**: Ensures the LSTM returns the entire sequence of outputs, not just the final state.

4. Global Max Pooling:
** Purpose** : Reduces the output from the LSTM layer by keeping only the maximum value from each feature map, which distills the most important information.

5. Dense Layer:
** Purpose** : Fully connected layer with 64 neurons, introducing non-linearity with a ReLU activation function to learn complex patterns.

6. Dropout Layer:
** Purpose** : Regularization technique that randomly drops 50% of the neurons during training, helping to prevent overfitting.

7. Output Dense Layer :
** Purpose** : Maps the output to a number of classes equal to the number of unique responses. Uses softmax to convert these outputs into probabilities.

8. Compiling the Model :
** Optimizer :** The Adam optimizer with a low learning rate is used to minimize the loss function effectively.

** Loss Function :** sparse\_categorical\_crossentropy is ideal for multi-class classification problems with integer-encoded labels.

** Metrics :** Accuracy is tracked to monitor the performance during training.

7. Training the Model

To ensure that the model generalizes well to new data, we use early stopping during training. This technique monitors the validation loss and stops training when the loss stops improving, thus preventing overfitting.

# Early stopping to avoid overfitting
early_stopping = keras.callbacks.EarlyStopping(monitor='val_loss', patience=10, restore_best_weights=True)
# Train the model with validation split
model.fit(train_sequences, encoded_labels, epochs=100, batch_size=32, validation_split=0.3, callbacks=[early_stopping])

8. Implementing Fuzzy Matching

Fuzzy matching allows the chatbot to handle user queries that don’t exactly match any question in the dataset. We use the fuzzywuzzy library to calculate similarity ratios between the user’s query and the pre-existing questions. If a close match is found, the chatbot returns the corresponding answer.

from fuzzywuzzy import fuzz
def fuzzy_match(query, faq_dict):
    best_match = None
    highest_ratio = 0
    for question, response in faq_dict.items():
        ratio = fuzz.ratio(query, question)
        if ratio > highest_ratio:
            highest_ratio = ratio
            best_match = response
    return best_match if highest_ratio > 70 else None  # Adjust threshold as needed

9. Generating Responses

The chatbot first checks if the user’s input matches any question in the dictionary. If no exact match is found, it uses fuzzy matching. If fuzzy matching also fails, the model predicts the response based on the trained LSTM. Additionally, we use TF-IDF vectorization to ensure that the predicted response is similar enough to the input query, otherwise, a fallback response is provided.

from sklearn.metrics.pairwise import cosine_similarity
from sklearn.feature_extraction.text import TfidfVectorizer
# Prepare TF-IDF vectorizer for similarity checking
vectorizer = TfidfVectorizer().fit(faq_data['Question'].tolist())
def generate_response(text):
    # Preprocess input
    text = preprocess_text(text)
    # Exact or fuzzy matching
    if text in faq_dict:
        return faq_dict[text]
    else:
        fuzzy_response = fuzzy_match(text, faq_dict)
        if fuzzy_response:
            return fuzzy_response
    # Use the model if exact or fuzzy match is not found
    sequence = tokenizer.texts_to_sequences([text])
    sequence = keras.preprocessing.sequence.pad_sequences(sequence, maxlen=train_sequences.shape[1])
    # Predict and decode the label
    prediction = model.predict(sequence)
    predicted_label = np.argmax(prediction)
    response = label_encoder.inverse_transform([predicted_label])[0]
    # Check similarity
    input_vector = vectorizer.transform([text])
    similarities = cosine_similarity(input_vector, vectorizer.transform(faq_data['Question'].tolist()))
    max_similarity = similarities.max()
    # Return a fallback response if similarity is below a threshold
    if max_similarity < 0.7:
        return "Sorry, I cannot answer that question, it is not in my data."
    return response

10. Creating a Chat Loop

Finally, we set up a chat loop where the user can enter queries, and the chatbot will generate responses using the logic and model we’ve built. This loop continuously prompts the user for input and returns the chatbot’s response.

# Chat loop
while True:
    user_input = input("Enter a message: ")
    response = generate_response(user_input)
    print("ChatBot: ", response)

Conclusion

By combining exact matching, fuzzy logic, and a neural network model with GloVe embeddings, we’ve built a powerful FAQ chatbot capable of handling a wide variety of user queries. The detailed breakdown of the neural network model helps to understand the role of each component, making it clear how the chatbot processes and responds to user input. An input can still be routed to the wrong answer. Evaluate with held-out questions before relying on the similarity threshold. The model can be further enhanced by fine-tuning, adding more data, or integrating additional NLP techniques to improve its performance in real-world applications.

Feel free to experiment and adapt the code to your specific use case. Happy coding!

“In this article, I am using the .venv/app-csv.py file.

github repo : https://github.com/fatih255/building-a-simple-chatbot-with-python

Do not hesitate to point out the wrong parts of the work that I am curious about and have done for trial purposes. :)