Optimizing Prompts to Reduce Token Usage and Costs

You are a helpful, friendly, and knowledgeable customer service assistant
for TechCorp, a leading technology company that specializes in innovative
software solutions. Your role is to assist customers with their questions
and concerns in a professional and courteous manner. You should always
strive to provide accurate, helpful, and comprehensive responses to all
customer inquiries. Please remember to be patient, understanding, and
empathetic when dealing with customer issues. If you don't know the answer
to a question, please let the customer know that you will find out and
get back to them. Always maintain a positive and helpful attitude.

The customer has the following question about their account:

{customer_question}

Please provide a helpful and informative response to address the
customer's concerns. Make sure your response is clear, concise, and
addresses all aspects of their question. If additional information
is needed, please ask clarifying questions.

You are TechCorp's support agent. Be helpful, accurate, and concise.

Customer question: {customer_question}

Respond directly to their question. Ask for clarification only if essential.

Be concise in your response.
Keep your answer brief.
Don't write lengthy explanations.
Provide short, to-the-point responses.

Be concise (2-3 sentences max).

You are an expert software developer with extensive experience in Python,
JavaScript, TypeScript, and various other programming languages. You have
deep knowledge of software design patterns, best practices, clean code
principles, and modern development methodologies including Agile and DevOps.
You should provide helpful, accurate, and well-thought-out responses to
programming questions. When writing code, always include comments explaining
what the code does. Follow industry best practices and conventions. If you're
unsure about something, acknowledge your uncertainty. Be friendly and
professional in your interactions.

Expert developer. Python/JS/TS. Write clean, commented code.
Acknowledge uncertainty. Be concise.

Role: Senior security analyst
Expertise: Penetration testing, vulnerability assessment
Tone: Technical, precise

After analyzing the customer review, I've determined that the overall
sentiment is positive. The customer expresses satisfaction with the
product quality and delivery speed. The confidence level for this
sentiment analysis is approximately 92 percent. The main positive
aspects mentioned are quality and shipping, while no negative aspects
were identified.

{"sentiment": "positive", "confidence": 0.92, "positive": ["quality", "shipping"], "negative": []}

// Verbose: 45 tokens
{"customer_name": "John", "order_status": "shipped", "estimated_delivery": "2026-01-20"}

// Abbreviated: 29 tokens
{"n": "John", "s": "shipped", "d": "2026-01-20"}

Summarize in exactly 3 bullet points.
Answer in one sentence.
List top 5 only.
Max 100 words.

Classify sentiment. Examples:
"Great product!" -> positive
"Terrible experience" -> negative
"It's okay" -> neutral
"Love it!" -> positive
"Waste of money" -> negative

Classify: "The delivery was fast but packaging was damaged"

Classify sentiment as positive, negative, or neutral.
Text: "The delivery was fast but packaging was damaged"

Classify sentiment (positive/negative/neutral).
Example: "Great but expensive" -> neutral
Text: "{input}"

Here's a 10,000-word document: {full_document}
Answer: What are the key financial metrics?

Document summary: {500_word_summary}
Full metrics section: {relevant_section}
Answer: What are the key financial metrics?

# Inefficient: retrieve 10 large chunks
chunks = retriever.get_top_k(query, k=10, chunk_size=1000)

# Efficient: retrieve fewer, smaller, more relevant chunks
chunks = retriever.get_top_k(query, k=3, chunk_size=300)

def optimize_context(document: str, max_tokens: int = 2000) -> str:
    # Keep beginning (usually has key info)
    # Keep end (often has conclusions)
    # Summarize middle
    beginning = document[:1000]
    end = document[-500:]
    middle_summary = summarize(document[1000:-500])
    return f"{beginning}\n[Summary: {middle_summary}]\n{end}"

sentiment1 = analyze_sentiment("Review 1...")
sentiment2 = analyze_sentiment("Review 2...")
sentiment3 = analyze_sentiment("Review 3...")

results = analyze_sentiment_batch([
    "Review 1...",
    "Review 2...",
    "Review 3..."
])
# Returns: [{"text": "Review 1...", "sentiment": "positive"}, ...]

Analyze sentiment for each review. Return JSON array.

Reviews:
1. "{review1}"
2. "{review2}"
3. "{review3}"

Format: [{"id": 1, "sentiment": "positive/negative/neutral"}, ...]

from functools import lru_cache
import hashlib

def get_cache_key(prompt: str) -> str:
    # Normalize and hash
    normalized = prompt.lower().strip()
    return hashlib.md5(normalized.encode()).hexdigest()

@lru_cache(maxsize=10000)
def cached_completion(cache_key: str, prompt: str) -> str:
    return llm.complete(prompt)

COMMON_RESPONSES = {
    "reset_password": "To reset your password, go to Settings > Security...",
    "refund_policy": "Our refund policy allows returns within 30 days...",
    "shipping_time": "Standard shipping takes 3-5 business days..."
}

def handle_query(query: str) -> str:
    intent = classify_intent(query)  # Cheap classification
    if intent in COMMON_RESPONSES:
        return COMMON_RESPONSES[intent]  # No LLM call
    return llm.generate(query)  # Full LLM call only when needed

def log_token_usage(prompt: str, response: str, model: str):
    input_tokens = count_tokens(prompt, model)
    output_tokens = count_tokens(response, model)

    metrics.histogram("llm.input_tokens", input_tokens)
    metrics.histogram("llm.output_tokens", output_tokens)

    print(f"Input: {input_tokens}, Output: {output_tokens}")

SELECT
    prompt_template,
    COUNT(*) as request_count,
    AVG(input_tokens) as avg_input,
    AVG(output_tokens) as avg_output,
    SUM(cost) as total_cost
FROM llm_requests
GROUP BY prompt_template
ORDER BY total_cost DESC
LIMIT 10;

def get_prompt(variant: str, context: dict) -> str:
    if variant == "control":
        return verbose_prompt.format(**context)
    else:
        return optimized_prompt.format(**context)

# Track metrics by variant
variant = random.choice(["control", "optimized"])
prompt = get_prompt(variant, context)
response = llm.complete(prompt)

metrics.track("llm_cost", cost, tags={"variant": variant})
metrics.track("response_quality", quality_score, tags={"variant": variant})

def evaluate_response(response: str, expected: str) -> float:
    # Measure semantic similarity, factual accuracy, etc.
    return quality_score

# Alert if quality drops
if quality_score < threshold:
    alert("Response quality degraded after prompt optimization")