Semantic Caching for LLMs#
RedisVL provides a SemanticCache interface to utilize Redis’ built-in caching capabilities AND vector search in order to store responses from previously-answered questions. This reduces the number of requests and tokens sent to the Large Language Models (LLM) service, decreasing costs and enhancing application throughput (by reducing the time taken to generate responses).
This notebook will go over how to use Redis as a Semantic Cache for your applications
First, we will import OpenAI to use their API for responding to user prompts. We will also create a simple ask_openai helper method to assist.
import os
import getpass
import time
import numpy as np
from openai import OpenAI
os.environ["TOKENIZERS_PARALLELISM"] = "False"
api_key = os.getenv("OPENAI_API_KEY") or getpass.getpass("Enter your OpenAI API key: ")
client = OpenAI(api_key=api_key)
def ask_openai(question: str) -> str:
response = client.completions.create(
model="gpt-3.5-turbo-instruct",
prompt=question,
max_tokens=200
)
return response.choices[0].text.strip()
# Test
print(ask_openai("What is the capital of France?"))
The capital of France is Paris.
Initializing SemanticCache#
SemanticCache will automatically create an index within Redis upon initialization for the semantic cache content.
from redisvl.extensions.llmcache import SemanticCache
llmcache = SemanticCache(
name="llmcache", # underlying search index name
redis_url="redis://localhost:6379", # redis connection url string
distance_threshold=0.1 # semantic cache distance threshold
)
22:11:38 redisvl.index.index INFO Index already exists, not overwriting.
# look at the index specification created for the semantic cache lookup
!rvl index info -i llmcache
Index Information:
╭──────────────┬────────────────┬──────────────┬─────────────────┬────────────╮
│ Index Name │ Storage Type │ Prefixes │ Index Options │ Indexing │
├──────────────┼────────────────┼──────────────┼─────────────────┼────────────┤
│ llmcache │ HASH │ ['llmcache'] │ [] │ 0 │
╰──────────────┴────────────────┴──────────────┴─────────────────┴────────────╯
Index Fields:
╭───────────────┬───────────────┬─────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬────────────────┬─────────────────┬────────────────╮
│ Name │ Attribute │ Type │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │ Field Option │ Option Value │
├───────────────┼───────────────┼─────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼────────────────┼─────────────────┼────────────────┤
│ prompt │ prompt │ TEXT │ WEIGHT │ 1 │ │ │ │ │ │ │
│ response │ response │ TEXT │ WEIGHT │ 1 │ │ │ │ │ │ │
│ inserted_at │ inserted_at │ NUMERIC │ │ │ │ │ │ │ │ │
│ updated_at │ updated_at │ NUMERIC │ │ │ │ │ │ │ │ │
│ prompt_vector │ prompt_vector │ VECTOR │ algorithm │ FLAT │ data_type │ FLOAT32 │ dim │ 768 │ distance_metric │ COSINE │
╰───────────────┴───────────────┴─────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴────────────────┴─────────────────┴────────────────╯
Basic Cache Usage#
question = "What is the capital of France?"
# Check the semantic cache -- should be empty
if response := llmcache.check(prompt=question):
print(response)
else:
print("Empty cache")
Empty cache
Our initial cache check should be empty since we have not yet stored anything in the cache. Below, store the question,
proper response, and any arbitrary metadata (as a python dictionary object) in the cache.
# Cache the question, answer, and arbitrary metadata
llmcache.store(
prompt=question,
response="Paris",
metadata={"city": "Paris", "country": "france"}
)
'llmcache:115049a298532be2f181edb03f766770c0db84c22aff39003fec340deaec7545'
Now we will check the cache again with the same question and with a semantically similar question:
# Check the cache again
if response := llmcache.check(prompt=question, return_fields=["prompt", "response", "metadata"]):
print(response)
else:
print("Empty cache")
[{'prompt': 'What is the capital of France?', 'response': 'Paris', 'metadata': {'city': 'Paris', 'country': 'france'}, 'key': 'llmcache:115049a298532be2f181edb03f766770c0db84c22aff39003fec340deaec7545'}]
# Check for a semantically similar result
question = "What actually is the capital of France?"
llmcache.check(prompt=question)[0]['response']
'Paris'
Customize the Distance Threshhold#
For most use cases, the right semantic similarity threshhold is not a fixed quantity. Depending on the choice of embedding model, the properties of the input query, and even business use case – the threshhold might need to change.
Fortunately, you can seamlessly adjust the threshhold at any point like below:
# Widen the semantic distance threshold
llmcache.set_threshold(0.3)
# Really try to trick it by asking around the point
# But is able to slip just under our new threshold
question = "What is the capital city of the country in Europe that also has a city named Nice?"
llmcache.check(prompt=question)[0]['response']
'Paris'
# Invalidate the cache completely by clearing it out
llmcache.clear()
# should be empty now
llmcache.check(prompt=question)
[]
Utilize TTL#
Redis uses TTL policies (optional) to expire individual keys at points in time in the future. This allows you to focus on your data flow and business logic without bothering with complex cleanup tasks.
A TTL policy set on the SemanticCache allows you to temporarily hold onto cache entries. Below, we will set the TTL policy to 5 seconds.
llmcache.set_ttl(5) # 5 seconds
llmcache.store("This is a TTL test", "This is a TTL test response")
time.sleep(6)
# confirm that the cache has cleared by now on it's own
result = llmcache.check("This is a TTL test")
print(result)
[]
# Reset the TTL to null (long lived data)
llmcache.set_ttl()
Simple Performance Testing#
Next, we will measure the speedup obtained by using SemanticCache. We will use the time module to measure the time taken to generate responses with and without SemanticCache.
def answer_question(question: str) -> str:
"""Helper function to answer a simple question using OpenAI with a wrapper
check for the answer in the semantic cache first.
Args:
question (str): User input question.
Returns:
str: Response.
"""
results = llmcache.check(prompt=question)
if results:
return results[0]["response"]
else:
answer = ask_openai(question)
return answer
start = time.time()
# asking a question -- openai response time
question = "What was the name of the first US President?"
answer = answer_question(question)
end = time.time()
print(f"Without caching, a call to openAI to answer this simple question took {end-start} seconds.")
# add the entry to our LLM cache
llmcache.store(prompt=question, response="George Washington")
Without caching, a call to openAI to answer this simple question took 0.9034533500671387 seconds.
'llmcache:67e0f6e28fe2a61c0022fd42bf734bb8ffe49d3e375fd69d692574295a20fc1a'
# Calculate the avg latency for caching over LLM usage
times = []
for _ in range(10):
cached_start = time.time()
cached_answer = answer_question(question)
cached_end = time.time()
times.append(cached_end-cached_start)
avg_time_with_cache = np.mean(times)
print(f"Avg time taken with LLM cache enabled: {avg_time_with_cache}")
print(f"Percentage of time saved: {round(((end - start) - avg_time_with_cache) / (end - start) * 100, 2)}%")
Avg time taken with LLM cache enabled: 0.09753389358520508
Percentage of time saved: 89.2%
# check the stats of the index
!rvl stats -i llmcache
Statistics:
╭─────────────────────────────┬─────────────╮
│ Stat Key │ Value │
├─────────────────────────────┼─────────────┤
│ num_docs │ 1 │
│ num_terms │ 19 │
│ max_doc_id │ 6 │
│ num_records │ 53 │
│ percent_indexed │ 1 │
│ hash_indexing_failures │ 0 │
│ number_of_uses │ 45 │
│ bytes_per_record_avg │ 45.0566 │
│ doc_table_size_mb │ 0.000134468 │
│ inverted_sz_mb │ 0.00227737 │
│ key_table_size_mb │ 2.76566e-05 │
│ offset_bits_per_record_avg │ 8 │
│ offset_vectors_sz_mb │ 3.91006e-05 │
│ offsets_per_term_avg │ 0.773585 │
│ records_per_doc_avg │ 53 │
│ sortable_values_size_mb │ 0 │
│ total_indexing_time │ 19.454 │
│ total_inverted_index_blocks │ 21 │
│ vector_index_sz_mb │ 3.0161 │
╰─────────────────────────────┴─────────────╯
# Clear the cache AND delete the underlying index
llmcache.delete()