Appendix M: Common Mistakes & Anti-Patterns

Introduction

Every experienced AI engineer has a collection of battle scars from mistakes learned the hard way. The field moves quickly, best practices evolve, and the gap between “works in a demo” and “works in production” is filled with subtle pitfalls that documentation rarely captures.

This appendix consolidates common mistakes documented throughout this textbook. Learning from others’ failures is more efficient than repeating them yourself. Each entry describes what people do wrong, why it causes problems, and the correct approach. Where applicable, code examples illustrate the difference between wrong and right implementations.

Use this as a reference when reviewing designs, debugging production issues, or preparing for interviews. Many of these mistakes are common interview topics precisely because they distinguish engineers who have shipped real systems from those who have only built prototypes.

RAG Systems

Mistake 1: No Chunk Overlap (Boundary Blindness)

What people do wrong: Split documents into non-overlapping chunks using simple character or word counts.

# WRONG: No overlap
def chunk_document(text: str, size: int = 500) -> list[str]:
    words = text.split()
    return [' '.join(words[i:i+size]) for i in range(0, len(words), size)]

Why it’s a problem: Important information often spans chunk boundaries. When a user asks a question that requires context from both sides of a boundary, retrieval fails because neither chunk contains the complete answer.

The correct approach: Use overlapping chunks so that context from one chunk bleeds into the next.

# RIGHT: With overlap
def chunk_document(text: str, size: int = 500, overlap: int = 50) -> list[str]:
    words = text.split()
    chunks = []
    for i in range(0, len(words), size - overlap):
        chunk = ' '.join(words[i:i+size])
        if chunk:
            chunks.append(chunk)
    return chunks

Reference: Chapter 4 (Common Mistake #5), Chapter 7

Mistake 2: Index-Embedding Mismatch

What people do wrong: Use different embedding models for indexing documents versus embedding queries at retrieval time.

Why it’s a problem: Embedding models map text into specific vector spaces. If you index with model A and query with model B, the vector spaces may not align, producing nonsensical similarity scores and broken retrieval.

The correct approach: Version your embeddings with your index. Store the embedding model identifier as metadata. When updating embedding models, re-index the entire corpus.

# RIGHT: Version tracking
class VectorIndex:
    def __init__(self, embedding_model: str):
        self.embedding_model = embedding_model
        self.index = []

    def query(self, text: str, query_model: str):
        if query_model != self.embedding_model:
            raise ValueError(
                f"Query model {query_model} doesn't match index model {self.embedding_model}. "
                "Re-index or use matching model."
            )
        # ... proceed with search

Reference: Chapter 7 (Common Production Pitfalls)

Mistake 3: Stale Document Indexes

What people do wrong: Index documents once and assume the index stays valid indefinitely, even as source documents are updated or deleted.

Why it’s a problem: Users receive outdated or incorrect information. Worse, the system presents stale data with high confidence, eroding trust.

The correct approach: Implement incremental re-indexing. Track document modification timestamps. Set up pipelines that detect changes and update affected chunks.

Reference: Chapter 7 (Common Production Pitfalls)

Mistake 4: Testing Retrieval and Generation Together Only

What people do wrong: Only evaluate the final RAG output without testing retrieval quality in isolation.

Why it’s a problem: When answers are wrong, you cannot diagnose whether the problem is retrieval (wrong chunks retrieved) or generation (model ignoring or misusing context). Debugging becomes guesswork.

The correct approach: Test each component separately. Evaluate retrieval recall and precision independently. Only after retrieval quality is validated, test the generation component.

# RIGHT: Separate evaluation
def evaluate_rag_system(queries, ground_truth):
    results = {
        'retrieval': [],
        'generation': []
    }

    for query, expected in zip(queries, ground_truth):
        # Test retrieval in isolation
        retrieved_chunks = retriever.search(query, k=5)
        retrieval_recall = compute_recall(retrieved_chunks, expected['relevant_chunks'])
        results['retrieval'].append(retrieval_recall)

        # Test generation with known-good context
        answer = generator.generate(query, expected['relevant_chunks'])
        generation_quality = evaluate_answer(answer, expected['answer'])
        results['generation'].append(generation_quality)

    return results

Reference: Chapter 4 (Common Mistake #6)

Mistake 5: Dumping Too Many Documents into Context

What people do wrong: Retrieve 20+ documents hoping the model finds what it needs.

Why it’s a problem: LLMs exhibit the “lost in the middle” problem (Liu et al., 2024). They attend strongly to content at the beginning and end of long contexts but often ignore the middle. More documents don’t mean better answers if the most relevant ones get buried.

The correct approach: Aggressive reranking to ensure top documents are truly best. Prioritize quality over quantity. Consider summarizing less-relevant documents. Put the most relevant content first or last.

Reference: Chapter 7 (Q3 Self-Assessment)

Mistake 6: Query Injection Vulnerabilities

What people do wrong: Concatenate user queries directly into prompts without sanitization or clear delimiters.

Why it’s a problem: User queries become part of the prompt context. Malicious users can inject instructions that override system behavior, extract system prompts, or manipulate outputs.

The correct approach: Use clear delimiters to separate user content from instructions. Sanitize inputs. Never trust that user input is benign.

# WRONG: Direct concatenation
prompt = f"Answer this question: {user_query}\nUsing this context: {context}"

# RIGHT: Clear delimiters and separation
prompt = f"""You are a helpful assistant. Answer the user's question based only on the provided context.

<context>
{context}
</context>

<user_question>
{user_query}
</user_question>

Provide your answer based solely on the context above."""

Reference: Chapter 7, Chapter 12

Agent Development

Mistake 7: Over-Engineering Agent Architecture

What people do wrong: Build complex multi-agent systems, planning frameworks, or supervisor hierarchies when a simple ReAct loop would suffice.

Why it’s a problem: Complexity increases debugging difficulty, latency, and cost without proportional benefit. Most tasks don’t need sophisticated architectures.

The correct approach: Start simple. A ReAct loop with good prompting handles the majority of use cases. Add complexity only when you hit specific failure modes that require it.

Reference: Chapter 8

Mistake 8: Relying on Prompts for Safety Enforcement

What people do wrong: Add instructions like “never delete important files” or “don’t send spam” to the system prompt and assume the agent will follow them reliably.

Why it’s a problem: LLMs are not reliable executors of safety policies. Models may misunderstand instructions, be manipulated by adversarial inputs, or simply make mistakes. A prompt saying “don’t do X” won’t reliably prevent X.

The correct approach: Enforce safety in code, not prompts. Implement capability scoping, allowlists, sandboxing, and rate limiting as programmatic controls.

# WRONG: Safety via prompt only
system_prompt = "You can send emails but NEVER send more than 10 per hour."

# RIGHT: Programmatic enforcement
class EmailTool:
    def __init__(self, max_per_hour: int = 10):
        self.rate_limiter = RateLimiter(max_per_hour, window_seconds=3600)

    def send(self, to: str, subject: str, body: str):
        if not self.rate_limiter.allow():
            raise RateLimitExceeded("Email rate limit exceeded")
        if not self._is_valid_recipient(to):
            raise InvalidRecipient(f"Recipient {to} not in allowlist")
        # ... send email

Reference: Chapter 8 (Safety Engineering)

Mistake 9: No Loop Detection or Termination Limits

What people do wrong: Let agents run indefinitely without maximum iteration counts or repetitive action detection.

Why it’s a problem: Agents can get stuck in loops, repeatedly trying failed approaches, consuming tokens and compute while producing no useful output. Without limits, costs spiral and users wait forever.

The correct approach: Implement maximum iteration limits, detect repetitive action patterns, and add timeout mechanisms.

# RIGHT: Safety limits
class Agent:
    def __init__(self, max_iterations: int = 20, max_same_action: int = 3):
        self.max_iterations = max_iterations
        self.max_same_action = max_same_action

    def run(self, task: str):
        recent_actions = []
        for i in range(self.max_iterations):
            action = self.plan_next_action()

            # Detect repetition
            recent_actions.append(action)
            if len(recent_actions) >= self.max_same_action:
                if all(a == recent_actions[-1] for a in recent_actions[-self.max_same_action:]):
                    return self.handle_stuck_state()

            result = self.execute(action)
            if self.is_complete(result):
                return result

        return self.handle_max_iterations_reached()

Reference: Chapter 8

Mistake 10: Insufficient Tool Capability Scoping

What people do wrong: Give agents broad permissions “for flexibility.” Full database access, arbitrary file system operations, unrestricted API calls.

Why it’s a problem: Every capability granted to an LLM is a capability an attacker might exploit. Broad permissions create enormous attack surface.

The correct approach: Apply the principle of least privilege rigorously. Scope to specific operations, specific data, specific users.

# WRONG: Broad permissions
def database_tool(query: str):
    return db.execute(query)  # Can do anything!

# RIGHT: Scoped permissions
def get_user_orders(user_id: str, limit: int = 10):
    """Read-only access to a specific user's orders."""
    return db.execute(
        "SELECT * FROM orders WHERE user_id = ? ORDER BY created_at DESC LIMIT ?",
        (user_id, min(limit, 100))  # Also cap the limit
    )

Reference: Chapter 8, Chapter 12

Prompt Engineering

Mistake 11: Vague Instructions Without Examples

What people do wrong: Use prompts like “Classify this message into a category” without defining categories or showing examples.

Why it’s a problem: The model has no way to know what categories exist, what the boundaries between them are, or what format you expect. Results are inconsistent and often wrong.

The correct approach: Define categories explicitly with descriptions. Provide examples. Specify output format. Add rules for edge cases.

# WRONG
prompt = f"Classify this message: {message}"

# RIGHT
prompt = f"""Classify the customer message into exactly ONE category.

CATEGORIES:

- ORDER_STATUS: Tracking, delivery timing, shipment questions
- RETURNS: Return requests, refund status, exchange inquiries
- PRODUCT: Product questions, sizing, compatibility
- BILLING: Payment failures, invoice requests, pricing disputes
- ESCALATION: Angry customers, legal threats, media mentions
- OTHER: Anything that doesn't fit above

RULES:
1. If message mentions both order status AND return, classify as RETURNS
2. If customer expresses significant frustration, classify as ESCALATION regardless of topic

<message>
{message}
</message>

Respond with only the category name in uppercase. No explanation."""

Reference: Chapter 6 (Real-World Case Study)

Mistake 12: Not Using Streaming in User-Facing Applications

What people do wrong: Wait for the complete LLM response before showing anything to the user.

Why it’s a problem: Users stare at a spinner for 2-10 seconds. The experience feels slow and unresponsive, even if total generation time is the same.

The correct approach: Stream tokens as they’re generated. Users see content appearing in ~200ms, which feels responsive even if the full response takes longer.

# WRONG: Wait for full response
response = await client.generate(prompt)
return response.text

# RIGHT: Stream tokens
async def generate_streaming(prompt: str):
    async for chunk in client.generate_stream(prompt):
        yield chunk.text

Reference: Chapter 4 (Common Mistake #3)

Mistake 13: Context Stuffing Without Structure

What people do wrong: Dump large amounts of context into prompts without clear organization or delimiters.

Why it’s a problem: The model struggles to distinguish between instructions, context, and user input. Attention is spread across undifferentiated text. Extraction and following instructions become unreliable.

The correct approach: Use clear delimiters (XML tags, markdown headers) to create semantic regions. Structure context hierarchically.

# WRONG: Unstructured dump
prompt = f"Here is some context: {context} Now answer this: {question}"

# RIGHT: Clear structure
prompt = f"""You are a technical support assistant.

<documentation>
{documentation}
</documentation>

<user_history>
{previous_messages}
</user_history>

<current_question>
{question}
</current_question>

Based on the documentation above, answer the user's question.
If the answer isn't in the documentation, say so."""

Reference: Chapter 6 (Why Delimiters Matter)

Deployment & Infrastructure

Mistake 14: Optimizing Without Profiling

What people do wrong: Assume where the bottleneck is and optimize that. “Attention is slow, so I’ll implement custom attention kernels.”

Why it’s a problem: The assumed bottleneck may not be the actual bottleneck. Attention might be 10% of runtime while data loading is 60%. You spend engineering effort optimizing the wrong thing.

The correct approach: Always profile first. Use tools like PyTorch Profiler, NVIDIA Nsight, or custom instrumentation to identify actual bottlenecks before optimizing.

Reference: Chapter 21 (Pitfall 1)

Mistake 15: Forgetting Warmup in Benchmarks

What people do wrong: Report the latency of the first inference call as representative.

Why it’s a problem: First calls include JIT compilation, CUDA context creation, memory allocation, and model loading. A 2-second first call followed by 50ms subsequent calls produces misleading metrics.

The correct approach: Run warmup iterations before measuring. Report steady-state latency.

# RIGHT: Proper benchmarking
def benchmark_model(model, inputs, warmup=10, iterations=100):
    # Warmup phase - don't measure these
    for _ in range(warmup):
        model(inputs)

    # Measurement phase
    torch.cuda.synchronize()
    times = []
    for _ in range(iterations):
        start = time.perf_counter()
        model(inputs)
        torch.cuda.synchronize()  # Wait for GPU
        times.append(time.perf_counter() - start)

    return {
        'mean': np.mean(times),
        'std': np.std(times),
        'p95': np.percentile(times, 95)
    }

Reference: Chapter 21 (Pitfall 2)

Mistake 16: Missing GPU Synchronization in Timing

What people do wrong: Time GPU operations without synchronization, reporting 1ms when actual latency is 50ms.

Why it’s a problem: CUDA operations are asynchronous. The Python call returns immediately, but the GPU work continues. You measure kernel launch time, not execution time.

The correct approach: Call torch.cuda.synchronize() before timing endpoints.

Reference: Chapter 21 (Pitfall 3)

Mistake 17: Testing at Wrong Scale

What people do wrong: Validate optimizations on small inputs (100 tokens) then deploy to production with large inputs (10,000 tokens).

Why it’s a problem: Algorithmic complexity differences only manifest at scale. Optimizations that work at small scale may fail or even hurt performance at production scale.

The correct approach: Test with production-representative input sizes. Profile at multiple scales to understand scaling behavior.

Reference: Chapter 21 (Pitfall 4)

Mistake 18: Ignoring Memory Fragmentation

What people do wrong: Assume free GPU memory means allocatable GPU memory.

Why it’s a problem: Memory fragmentation can prevent allocation even with substantial free memory. OOM errors with “40GB free, 60GB reserved” indicate fragmentation issues.

The correct approach: Monitor fragmentation. Use torch.cuda.empty_cache() judiciously. Consider memory-efficient allocation strategies.

def check_memory_health():
    allocated = torch.cuda.memory_allocated()
    reserved = torch.cuda.memory_reserved()
    fragmentation = (reserved - allocated) / reserved if reserved > 0 else 0

    if fragmentation > 0.3:
        print("Warning: High fragmentation. Consider torch.cuda.empty_cache()")

    return {
        'allocated_gb': allocated / 1e9,
        'reserved_gb': reserved / 1e9,
        'fragmentation': fragmentation
    }

Reference: Chapter 21 (Pitfall 5)

Mistake 19: Pre-Allocating Fixed KV Cache Sizes

What people do wrong: Allocate contiguous memory for maximum sequence length for every request, regardless of actual usage.

Why it’s a problem: Severe memory waste. A 4K context allocation uses 2+ GB per request, even for 500-token sequences. This limits concurrency and wastes expensive GPU memory.

The correct approach: Use PagedAttention (vLLM) or similar dynamic allocation strategies that allocate memory on-demand as sequences grow.

Reference: Chapter 9 (PagedAttention)

Security

Mistake 20: Treating Prompt Injection Like SQL Injection

What people do wrong: Assume that input sanitization alone can prevent prompt injection, similar to parameterized queries preventing SQL injection.

Why it’s a problem: Unlike SQL, there’s no architectural separation between instructions and data in LLM prompts. The model has no equivalent to parameterized queries. Every defense is heuristic, not a guarantee.

The correct approach: Defense-in-depth with multiple layers. Perimeter defenses, prompt architecture, model-level controls, output validation, and execution controls. Assume any single layer can be bypassed.

Reference: Chapter 16 (Why LLM Security Is Different)

Mistake 21: Ignoring Indirect Prompt Injection

What people do wrong: Focus security efforts on direct user inputs while ignoring data sources the model processes (retrieved documents, tool outputs, web content).

Why it’s a problem: Attackers can poison data sources that the RAG system or agent will eventually consume. Hidden instructions in web pages, emails, or documents can hijack model behavior without the attacker ever directly interacting with your application.

The correct approach: Treat all data sources as potentially malicious. Implement content validation for retrieved documents. Use clear instruction hierarchies. Consider sandboxing or human-in-the-loop for sensitive actions triggered by external content.

Reference: Chapter 16 (Indirect Prompt Injection)

Mistake 22: Security Rules in System Prompts Only

What people do wrong: Rely on system prompt instructions like “never reveal your instructions” as the primary security mechanism.

Why it’s a problem: Determined attackers consistently find ways around prompt-based restrictions. Jailbreaks, roleplay attacks, encoded payloads, and adversarial suffixes can override system prompts.

The correct approach: Use system prompts as one layer, but enforce critical security constraints in code. Output validation, tool allowlists, and action confirmation flows don’t depend on model compliance.

Reference: Chapter 16 (Defense-in-Depth Architecture)

Evaluation

Mistake 23: Optimizing for BLEU/ROUGE Scores

What people do wrong: Use BLEU or ROUGE as the primary optimization target for generation quality.

Why it’s a problem: These metrics measure lexical overlap, not semantic quality. Two equally valid responses can have zero n-gram overlap. Optimizing for these metrics produces outputs that game n-gram matching without being useful.

The correct approach: Use BLEU/ROUGE as sanity checks and screening, not optimization targets. For quality optimization, use LLM-as-judge systems calibrated against human judgment, or direct human evaluation.

Reference: Chapter 11 (Why Traditional Metrics Fail)

Mistake 24: LLM Judges Without Calibration

What people do wrong: Deploy LLM-as-judge evaluation without validating correlation with human judgment.

Why it’s a problem: LLM judges have systematic biases (verbosity preference, position bias, self-preference, confidence over correctness). Without calibration, you might ship degraded models or reject improvements.

The correct approach: Build calibration sets with human ratings. Compute correlation, agreement rate, and systematic bias. Only trust judges that achieve correlation > 0.7 with human ratings.

def calibrate_judge(judge, calibration_examples):
    human_ratings = [ex['human_rating'] for ex in calibration_examples]
    judge_ratings = [judge.evaluate(ex) for ex in calibration_examples]

    correlation = np.corrcoef(human_ratings, judge_ratings)[0, 1]
    bias = np.mean(judge_ratings) - np.mean(human_ratings)

    if correlation < 0.7 or abs(bias) > 0.5:
        raise ValueError("Judge calibration failed. Adjust prompt or model.")

Reference: Chapter 11 (Calibration: Making Judges Reliable)

Mistake 25: Pairwise Comparison Without Position Debiasing

What people do wrong: Run pairwise comparisons in a single order (A vs B) without checking for position bias.

Why it’s a problem: LLM judges often prefer whichever response appears first (or last). The same two responses compared in opposite orders can yield opposite judgments.

The correct approach: Run comparisons in both orders. If results disagree, mark as a tie or low confidence. Only trust consistent preferences.

Reference: Chapter 11 (Pairwise Comparison)

Mistake 26: Peeking During A/B Tests

What people do wrong: Check results daily and stop the experiment when p < 0.05.

Why it’s a problem: This inflates false positive rates from the nominal 5% to 20-30%. You’ll conclude there are differences when there aren’t.

The correct approach: Pre-commit to sample sizes and stopping rules. Use sequential testing methods if early stopping is required. Never peek without proper statistical adjustments.

Reference: Chapter 11 (Avoiding Common Statistical Pitfalls)

Mistake 27: Tuning on Test Data

What people do wrong: Check test performance, make changes, check again, and iterate.

Why it’s a problem: You’re effectively training on the test set. Final accuracy will be optimistically biased. This is data leakage.

The correct approach: Use a separate validation set for iterative development. Touch the test set only once for final evaluation. If you must iterate, use cross-validation.

Reference: Chapter 3 (A Common Mistake)

Cost Management

Mistake 28: Ignoring Costs During Development

What people do wrong: Iterate quickly during development without tracking API costs, then discover hundreds of dollars spent.

Why it’s a problem: Unlike traditional software, AI has meaningful per-request marginal costs. Development and testing can rack up significant bills before you realize it.

The correct approach: Track costs from day one. Set up budgets and alerts. Log token usage alongside functionality.

# RIGHT: Track costs alongside functionality
class CostTracker:
    def __init__(self):
        self.total_input_tokens = 0
        self.total_output_tokens = 0

    def track(self, input_tokens: int, output_tokens: int):
        self.total_input_tokens += input_tokens
        self.total_output_tokens += output_tokens

    def estimated_cost(self, input_price: float, output_price: float) -> float:
        return (self.total_input_tokens / 1_000_000 * input_price +
                self.total_output_tokens / 1_000_000 * output_price)

Reference: Chapter 4 (Common Mistake #8), Chapter 26

Mistake 29: Not Caching Repeated Queries

What people do wrong: Process every query through the full LLM pipeline, even when many queries are identical or highly similar.

Why it’s a problem: Real-world query distributions often have heavy tails. The top 10% of queries might represent 60% of traffic. Processing these repeatedly wastes compute and money.

The correct approach: Implement semantic caching. Cache exact matches. Consider approximate matching for highly similar queries.

Reference: Chapter 26 (Token Economics)

Mistake 30: Using Premium Models for All Workloads

What people do wrong: Use GPT-5.5 or Claude Opus 4.8 for everything, including simple tasks that smaller models handle well.

Why it’s a problem: Premium models cost 10-100x more than capable smaller models. Simple classification, extraction, or templated generation often doesn’t need frontier model capabilities.

The correct approach: Route requests to appropriate models based on complexity. Use smaller models for simple tasks, reserving premium models for tasks that require their capabilities.

Reference: Chapter 26

Mistake 31: Verbose Prompts When Concise Suffices

What people do wrong: Use 8,000-token prompts when 2,000 tokens would produce equivalent results.

Why it’s a problem: Input tokens cost money. Verbose prompts also increase latency and reduce the effective context window available for actual content.

The correct approach: Iterate on prompt engineering to achieve the same quality with fewer tokens. Measure quality vs. prompt length tradeoffs explicitly.

Reference: Chapter 26 (Introduction)

Mistake 32: Ignoring Hidden TCO Costs

What people do wrong: Compare options based only on visible infrastructure costs (compute, API calls).

Why it’s a problem: Engineering time often represents 40-60% of total cost of ownership. A system that costs $50K/month in compute but requires a full-time engineer to maintain has 30% hidden costs. Evaluation, monitoring, compliance, and disaster recovery also add up.

The correct approach: Calculate full TCO including direct costs (compute, storage, APIs), indirect costs (engineering time, opportunity cost), and hidden costs (compliance, monitoring, DR).

Reference: Chapter 26 (Total Cost of Ownership)

Development Practices

Mistake 33: Committing API Keys to Git

What people do wrong: Store API keys in code, .env files, or configuration that gets committed.

Why it’s a problem: API keys in git history are exposed to anyone with repository access. Bots scan GitHub for leaked keys. Credentials in production configs create security vulnerabilities.

The correct approach: Use environment variables. Never create .env files with real credentials. Add sensitive files to .gitignore. Use secrets management services in production.

Reference: Chapter 4 (Common Mistake #1)

Mistake 34: Retrying All Errors

What people do wrong: Implement retry logic that retries on any error.

Why it’s a problem: Some errors (invalid API key, malformed request, content policy violations) will never succeed no matter how many retries. Retrying these wastes time and potentially money.

The correct approach: Only retry transient errors (rate limits, timeouts, temporary server errors). Fail fast on permanent errors.

# RIGHT: Selective retry
RETRYABLE_ERRORS = (RateLimitError, TimeoutError, ServiceUnavailableError)

async def call_with_retry(func, max_retries=3):
    for attempt in range(max_retries):
        try:
            return await func()
        except RETRYABLE_ERRORS as e:
            if attempt == max_retries - 1:
                raise
            await asyncio.sleep(2 ** attempt)  # Exponential backoff
        except (AuthenticationError, InvalidRequestError):
            raise  # Don't retry these

Reference: Chapter 4 (Common Mistake #4)

Mistake 35: Using Global pip Instead of Virtual Environment

What people do wrong: Install packages with global pip, mixing dependencies across projects.

Why it’s a problem: Dependency conflicts between projects. Reproducibility issues. “Works on my machine” problems in deployment.

The correct approach: Always use virtual environments. Verify which pip shows the venv path before installing.

Reference: Chapter 4 (Common Mistake #2)

Mistake 36: Ignoring Edge Cases During Development

What people do wrong: Build for the happy path, ignoring error conditions, malformed inputs, and boundary cases.

Why it’s a problem: The happy path works great in demos. Production users find every edge case. Malformed documents, encoding issues, unexpected API responses, and unusual queries cause failures that are expensive to debug later.

The correct approach: Build edge case handling from the start. Test with malformed inputs. Handle encoding issues. Validate assumptions about external data.

Reference: Chapter 4 (Common Mistake #7)

Cross-Team and Process Mistakes

Mistake 37: Specializing in Tools Instead of Concepts

What people do wrong: Build expertise as “LangChain expert” or “specific-framework specialist.”

Why it’s a problem: Tools change; concepts persist. Framework expertise becomes dated. Concept expertise (agentic systems, evaluation methodology, retrieval architectures) transfers across tools.

The correct approach: Learn concepts deeply. Use tools to implement concepts, but don’t let tool expertise substitute for conceptual understanding.

Reference: Chapter 17 (Common Pitfalls in Specialization)

Mistake 38: Writing ADRs That Nobody Reads

What people do wrong: Create multi-page Architecture Decision Records with exhaustive detail.

Why it’s a problem: If ADRs exceed two pages, they don’t get read. The decision record becomes a design document that nobody consults.

The correct approach: Keep ADRs concise. Focus on the decision, key alternatives considered, and reasoning. If you need more detail, write a separate design doc and reference it.

Reference: Chapter 20 (ADR Anti-Patterns)

Mistake 39: Treating All Decisions as One-Way Doors

What people do wrong: Apply the same deliberation process to reversible decisions as to irreversible ones.

Why it’s a problem: Most decisions are two-way doors that can be reversed. Over-deliberating these slows execution without proportional benefit.

The correct approach: Match decision speed to stakes. Two-way doors should be fast. One-way doors merit deliberation.

Reference: Chapter 20 (Match Decision Speed to Stakes)

Mistake 40: Premature Optimization

What people do wrong: Spend a week writing custom CUDA kernels before validating the product.

Why it’s a problem: The product might pivot, making the optimization worthless. Optimization before validation is wasted effort.

The correct approach: Get it working first, make it right second, make it fast third. Only optimize after validating that the system solves a real problem and identifying actual bottlenecks.

Reference: Chapter 21 (Pitfall 6)

Summary Checklist

Use this checklist during design reviews:

RAG Systems - [ ] Chunks have appropriate overlap - [ ] Embedding model is versioned with index - [ ] Incremental re-indexing is implemented - [ ] Retrieval is tested separately from generation - [ ] User input is delimited from instructions

Agents - [ ] Architecture complexity is justified - [ ] Safety is enforced in code, not just prompts - [ ] Loop detection and iteration limits exist - [ ] Capabilities are minimally scoped

Prompts - [ ] Instructions are specific with examples - [ ] Output format is specified - [ ] Context is clearly structured - [ ] Streaming is used for user-facing responses

Deployment - [ ] Actual bottleneck is profiled before optimizing - [ ] Benchmarks include warmup - [ ] GPU timing includes synchronization - [ ] Testing uses production-scale inputs

Security - [ ] Multiple defense layers exist - [ ] Indirect injection vectors are considered - [ ] Critical constraints are enforced in code - [ ] All data sources are treated as potentially malicious

Evaluation - [ ] Metrics correlate with actual quality goals - [ ] LLM judges are calibrated against humans - [ ] A/B tests have pre-committed stopping rules - [ ] Test data is held out from development

Cost - [ ] Token usage is tracked from day one - [ ] Caching is implemented for repeated queries - [ ] Model routing matches task complexity - [ ] Full TCO is calculated, not just compute

Chapter 4: Your First LLM Application (development mistakes)
Chapter 6: Prompt Engineering (prompt design patterns)
Chapter 7: RAG Systems Deep Dive (retrieval pitfalls)
Chapter 8: Agentic Systems (agent safety)
Chapter 9: LLM Deployment & Infrastructure (serving optimization)
Chapter 11: MLOps & Evaluation (evaluation methodology)
Chapter 12: Security & Adversarial Robustness (security patterns)
Chapter 21: Performance Engineering (optimization pitfalls)
Chapter 26: Cost Engineering (economic mistakes)

# Appendix M: Common Mistakes & Anti-Patterns {.unnumbered} ## Introduction Every experienced AI engineer has a collection of battle scars from mistakes learned the hard way. The field moves quickly, best practices evolve, and the gap between "works in a demo" and "works in production" is filled with subtle pitfalls that documentation rarely captures. This appendix consolidates common mistakes documented throughout this textbook. Learning from others' failures is more efficient than repeating them yourself. Each entry describes what people do wrong, why it causes problems, and the correct approach. Where applicable, code examples illustrate the difference between wrong and right implementations. Use this as a reference when reviewing designs, debugging production issues, or preparing for interviews. Many of these mistakes are common interview topics precisely because they distinguish engineers who have shipped real systems from those who have only built prototypes. --- ## RAG Systems ### Mistake 1: No Chunk Overlap (Boundary Blindness) **What people do wrong**: Split documents into non-overlapping chunks using simple character or word counts. ```python # WRONG: No overlap def chunk_document(text: str, size: int = 500) -> list[str]: words = text.split() return [' '.join(words[i:i+size]) for i in range(0, len(words), size)] ``` **Why it's a problem**: Important information often spans chunk boundaries. When a user asks a question that requires context from both sides of a boundary, retrieval fails because neither chunk contains the complete answer. **The correct approach**: Use overlapping chunks so that context from one chunk bleeds into the next. ```python # RIGHT: With overlap def chunk_document(text: str, size: int = 500, overlap: int = 50) -> list[str]: words = text.split() chunks = [] for i in range(0, len(words), size - overlap): chunk = ' '.join(words[i:i+size]) if chunk: chunks.append(chunk) return chunks ``` *Reference: Chapter 4 (Common Mistake #5), Chapter 7* --- ### Mistake 2: Index-Embedding Mismatch **What people do wrong**: Use different embedding models for indexing documents versus embedding queries at retrieval time. **Why it's a problem**: Embedding models map text into specific vector spaces. If you index with model A and query with model B, the vector spaces may not align, producing nonsensical similarity scores and broken retrieval. **The correct approach**: Version your embeddings with your index. Store the embedding model identifier as metadata. When updating embedding models, re-index the entire corpus. ```python # RIGHT: Version tracking class VectorIndex: def __init__(self, embedding_model: str): self.embedding_model = embedding_model self.index = [] def query(self, text: str, query_model: str): if query_model != self.embedding_model: raise ValueError( f"Query model {query_model} doesn't match index model {self.embedding_model}. " "Re-index or use matching model." ) # ... proceed with search ``` *Reference: Chapter 7 (Common Production Pitfalls)* --- ### Mistake 3: Stale Document Indexes **What people do wrong**: Index documents once and assume the index stays valid indefinitely, even as source documents are updated or deleted. **Why it's a problem**: Users receive outdated or incorrect information. Worse, the system presents stale data with high confidence, eroding trust. **The correct approach**: Implement incremental re-indexing. Track document modification timestamps. Set up pipelines that detect changes and update affected chunks. *Reference: Chapter 7 (Common Production Pitfalls)* --- ### Mistake 4: Testing Retrieval and Generation Together Only **What people do wrong**: Only evaluate the final RAG output without testing retrieval quality in isolation. **Why it's a problem**: When answers are wrong, you cannot diagnose whether the problem is retrieval (wrong chunks retrieved) or generation (model ignoring or misusing context). Debugging becomes guesswork. **The correct approach**: Test each component separately. Evaluate retrieval recall and precision independently. Only after retrieval quality is validated, test the generation component. ```python # RIGHT: Separate evaluation def evaluate_rag_system(queries, ground_truth): results = { 'retrieval': [], 'generation': [] } for query, expected in zip(queries, ground_truth): # Test retrieval in isolation retrieved_chunks = retriever.search(query, k=5) retrieval_recall = compute_recall(retrieved_chunks, expected['relevant_chunks']) results['retrieval'].append(retrieval_recall) # Test generation with known-good context answer = generator.generate(query, expected['relevant_chunks']) generation_quality = evaluate_answer(answer, expected['answer']) results['generation'].append(generation_quality) return results ``` *Reference: Chapter 4 (Common Mistake #6)* --- ### Mistake 5: Dumping Too Many Documents into Context **What people do wrong**: Retrieve 20+ documents hoping the model finds what it needs. **Why it's a problem**: LLMs exhibit the "lost in the middle" problem (Liu et al., 2024). They attend strongly to content at the beginning and end of long contexts but often ignore the middle. More documents don't mean better answers if the most relevant ones get buried. **The correct approach**: Aggressive reranking to ensure top documents are truly best. Prioritize quality over quantity. Consider summarizing less-relevant documents. Put the most relevant content first or last. *Reference: Chapter 7 (Q3 Self-Assessment)* --- ### Mistake 6: Query Injection Vulnerabilities **What people do wrong**: Concatenate user queries directly into prompts without sanitization or clear delimiters. **Why it's a problem**: User queries become part of the prompt context. Malicious users can inject instructions that override system behavior, extract system prompts, or manipulate outputs. **The correct approach**: Use clear delimiters to separate user content from instructions. Sanitize inputs. Never trust that user input is benign. ```python # WRONG: Direct concatenation prompt = f"Answer this question: {user_query}\nUsing this context: {context}" # RIGHT: Clear delimiters and separation prompt = f"""You are a helpful assistant. Answer the user's question based only on the provided context. <context> {context} </context> <user_question> {user_query} </user_question> Provide your answer based solely on the context above.""" ``` *Reference: Chapter 7, Chapter 12* --- ## Agent Development ### Mistake 7: Over-Engineering Agent Architecture **What people do wrong**: Build complex multi-agent systems, planning frameworks, or supervisor hierarchies when a simple ReAct loop would suffice. **Why it's a problem**: Complexity increases debugging difficulty, latency, and cost without proportional benefit. Most tasks don't need sophisticated architectures. **The correct approach**: Start simple. A ReAct loop with good prompting handles the majority of use cases. Add complexity only when you hit specific failure modes that require it. *Reference: Chapter 8* --- ### Mistake 8: Relying on Prompts for Safety Enforcement **What people do wrong**: Add instructions like "never delete important files" or "don't send spam" to the system prompt and assume the agent will follow them reliably. **Why it's a problem**: LLMs are not reliable executors of safety policies. Models may misunderstand instructions, be manipulated by adversarial inputs, or simply make mistakes. A prompt saying "don't do X" won't reliably prevent X. **The correct approach**: Enforce safety in code, not prompts. Implement capability scoping, allowlists, sandboxing, and rate limiting as programmatic controls. ```python # WRONG: Safety via prompt only system_prompt = "You can send emails but NEVER send more than 10 per hour." # RIGHT: Programmatic enforcement class EmailTool: def __init__(self, max_per_hour: int = 10): self.rate_limiter = RateLimiter(max_per_hour, window_seconds=3600) def send(self, to: str, subject: str, body: str): if not self.rate_limiter.allow(): raise RateLimitExceeded("Email rate limit exceeded") if not self._is_valid_recipient(to): raise InvalidRecipient(f"Recipient {to} not in allowlist") # ... send email ``` *Reference: Chapter 8 (Safety Engineering)* --- ### Mistake 9: No Loop Detection or Termination Limits **What people do wrong**: Let agents run indefinitely without maximum iteration counts or repetitive action detection. **Why it's a problem**: Agents can get stuck in loops, repeatedly trying failed approaches, consuming tokens and compute while producing no useful output. Without limits, costs spiral and users wait forever. **The correct approach**: Implement maximum iteration limits, detect repetitive action patterns, and add timeout mechanisms. ```python # RIGHT: Safety limits class Agent: def __init__(self, max_iterations: int = 20, max_same_action: int = 3): self.max_iterations = max_iterations self.max_same_action = max_same_action def run(self, task: str): recent_actions = [] for i in range(self.max_iterations): action = self.plan_next_action() # Detect repetition recent_actions.append(action) if len(recent_actions) >= self.max_same_action: if all(a == recent_actions[-1] for a in recent_actions[-self.max_same_action:]): return self.handle_stuck_state() result = self.execute(action) if self.is_complete(result): return result return self.handle_max_iterations_reached() ``` *Reference: Chapter 8* --- ### Mistake 10: Insufficient Tool Capability Scoping **What people do wrong**: Give agents broad permissions "for flexibility." Full database access, arbitrary file system operations, unrestricted API calls. **Why it's a problem**: Every capability granted to an LLM is a capability an attacker might exploit. Broad permissions create enormous attack surface. **The correct approach**: Apply the principle of least privilege rigorously. Scope to specific operations, specific data, specific users. ```python # WRONG: Broad permissions def database_tool(query: str): return db.execute(query) # Can do anything! # RIGHT: Scoped permissions def get_user_orders(user_id: str, limit: int = 10): """Read-only access to a specific user's orders.""" return db.execute( "SELECT * FROM orders WHERE user_id = ? ORDER BY created_at DESC LIMIT ?", (user_id, min(limit, 100)) # Also cap the limit ) ``` *Reference: Chapter 8, Chapter 12* --- ## Prompt Engineering ### Mistake 11: Vague Instructions Without Examples **What people do wrong**: Use prompts like "Classify this message into a category" without defining categories or showing examples. **Why it's a problem**: The model has no way to know what categories exist, what the boundaries between them are, or what format you expect. Results are inconsistent and often wrong. **The correct approach**: Define categories explicitly with descriptions. Provide examples. Specify output format. Add rules for edge cases. ```python # WRONG prompt = f"Classify this message: {message}" # RIGHT prompt = f"""Classify the customer message into exactly ONE category. CATEGORIES: - ORDER_STATUS: Tracking, delivery timing, shipment questions - RETURNS: Return requests, refund status, exchange inquiries - PRODUCT: Product questions, sizing, compatibility - BILLING: Payment failures, invoice requests, pricing disputes - ESCALATION: Angry customers, legal threats, media mentions - OTHER: Anything that doesn't fit above RULES: 1. If message mentions both order status AND return, classify as RETURNS 2. If customer expresses significant frustration, classify as ESCALATION regardless of topic <message> {message} </message> Respond with only the category name in uppercase. No explanation.""" ``` *Reference: Chapter 6 (Real-World Case Study)* --- ### Mistake 12: Not Using Streaming in User-Facing Applications **What people do wrong**: Wait for the complete LLM response before showing anything to the user. **Why it's a problem**: Users stare at a spinner for 2-10 seconds. The experience feels slow and unresponsive, even if total generation time is the same. **The correct approach**: Stream tokens as they're generated. Users see content appearing in ~200ms, which feels responsive even if the full response takes longer. ```python # WRONG: Wait for full response response = await client.generate(prompt) return response.text # RIGHT: Stream tokens async def generate_streaming(prompt: str): async for chunk in client.generate_stream(prompt): yield chunk.text ``` *Reference: Chapter 4 (Common Mistake #3)* --- ### Mistake 13: Context Stuffing Without Structure **What people do wrong**: Dump large amounts of context into prompts without clear organization or delimiters. **Why it's a problem**: The model struggles to distinguish between instructions, context, and user input. Attention is spread across undifferentiated text. Extraction and following instructions become unreliable. **The correct approach**: Use clear delimiters (XML tags, markdown headers) to create semantic regions. Structure context hierarchically. ```python # WRONG: Unstructured dump prompt = f"Here is some context: {context} Now answer this: {question}" # RIGHT: Clear structure prompt = f"""You are a technical support assistant. <documentation> {documentation} </documentation> <user_history> {previous_messages} </user_history> <current_question> {question} </current_question> Based on the documentation above, answer the user's question. If the answer isn't in the documentation, say so.""" ``` *Reference: Chapter 6 (Why Delimiters Matter)* --- ## Deployment & Infrastructure ### Mistake 14: Optimizing Without Profiling **What people do wrong**: Assume where the bottleneck is and optimize that. "Attention is slow, so I'll implement custom attention kernels." **Why it's a problem**: The assumed bottleneck may not be the actual bottleneck. Attention might be 10% of runtime while data loading is 60%. You spend engineering effort optimizing the wrong thing. **The correct approach**: Always profile first. Use tools like PyTorch Profiler, NVIDIA Nsight, or custom instrumentation to identify actual bottlenecks before optimizing. *Reference: Chapter 21 (Pitfall 1)* --- ### Mistake 15: Forgetting Warmup in Benchmarks **What people do wrong**: Report the latency of the first inference call as representative. **Why it's a problem**: First calls include JIT compilation, CUDA context creation, memory allocation, and model loading. A 2-second first call followed by 50ms subsequent calls produces misleading metrics. **The correct approach**: Run warmup iterations before measuring. Report steady-state latency. ```python # RIGHT: Proper benchmarking def benchmark_model(model, inputs, warmup=10, iterations=100): # Warmup phase - don't measure these for _ in range(warmup): model(inputs) # Measurement phase torch.cuda.synchronize() times = [] for _ in range(iterations): start = time.perf_counter() model(inputs) torch.cuda.synchronize() # Wait for GPU times.append(time.perf_counter() - start) return { 'mean': np.mean(times), 'std': np.std(times), 'p95': np.percentile(times, 95) } ``` *Reference: Chapter 21 (Pitfall 2)* --- ### Mistake 16: Missing GPU Synchronization in Timing **What people do wrong**: Time GPU operations without synchronization, reporting 1ms when actual latency is 50ms. **Why it's a problem**: CUDA operations are asynchronous. The Python call returns immediately, but the GPU work continues. You measure kernel launch time, not execution time. **The correct approach**: Call `torch.cuda.synchronize()` before timing endpoints. *Reference: Chapter 21 (Pitfall 3)* --- ### Mistake 17: Testing at Wrong Scale **What people do wrong**: Validate optimizations on small inputs (100 tokens) then deploy to production with large inputs (10,000 tokens). **Why it's a problem**: Algorithmic complexity differences only manifest at scale. Optimizations that work at small scale may fail or even hurt performance at production scale. **The correct approach**: Test with production-representative input sizes. Profile at multiple scales to understand scaling behavior. *Reference: Chapter 21 (Pitfall 4)* --- ### Mistake 18: Ignoring Memory Fragmentation **What people do wrong**: Assume free GPU memory means allocatable GPU memory. **Why it's a problem**: Memory fragmentation can prevent allocation even with substantial free memory. OOM errors with "40GB free, 60GB reserved" indicate fragmentation issues. **The correct approach**: Monitor fragmentation. Use `torch.cuda.empty_cache()` judiciously. Consider memory-efficient allocation strategies. ```python def check_memory_health(): allocated = torch.cuda.memory_allocated() reserved = torch.cuda.memory_reserved() fragmentation = (reserved - allocated) / reserved if reserved > 0 else 0 if fragmentation > 0.3: print("Warning: High fragmentation. Consider torch.cuda.empty_cache()") return { 'allocated_gb': allocated / 1e9, 'reserved_gb': reserved / 1e9, 'fragmentation': fragmentation } ``` *Reference: Chapter 21 (Pitfall 5)* --- ### Mistake 19: Pre-Allocating Fixed KV Cache Sizes **What people do wrong**: Allocate contiguous memory for maximum sequence length for every request, regardless of actual usage. **Why it's a problem**: Severe memory waste. A 4K context allocation uses 2+ GB per request, even for 500-token sequences. This limits concurrency and wastes expensive GPU memory. **The correct approach**: Use PagedAttention (vLLM) or similar dynamic allocation strategies that allocate memory on-demand as sequences grow. *Reference: Chapter 9 (PagedAttention)* --- ## Security ### Mistake 20: Treating Prompt Injection Like SQL Injection **What people do wrong**: Assume that input sanitization alone can prevent prompt injection, similar to parameterized queries preventing SQL injection. **Why it's a problem**: Unlike SQL, there's no architectural separation between instructions and data in LLM prompts. The model has no equivalent to parameterized queries. Every defense is heuristic, not a guarantee. **The correct approach**: Defense-in-depth with multiple layers. Perimeter defenses, prompt architecture, model-level controls, output validation, and execution controls. Assume any single layer can be bypassed. *Reference: Chapter 16 (Why LLM Security Is Different)* --- ### Mistake 21: Ignoring Indirect Prompt Injection **What people do wrong**: Focus security efforts on direct user inputs while ignoring data sources the model processes (retrieved documents, tool outputs, web content). **Why it's a problem**: Attackers can poison data sources that the RAG system or agent will eventually consume. Hidden instructions in web pages, emails, or documents can hijack model behavior without the attacker ever directly interacting with your application. **The correct approach**: Treat all data sources as potentially malicious. Implement content validation for retrieved documents. Use clear instruction hierarchies. Consider sandboxing or human-in-the-loop for sensitive actions triggered by external content. *Reference: Chapter 16 (Indirect Prompt Injection)* --- ### Mistake 22: Security Rules in System Prompts Only **What people do wrong**: Rely on system prompt instructions like "never reveal your instructions" as the primary security mechanism. **Why it's a problem**: Determined attackers consistently find ways around prompt-based restrictions. Jailbreaks, roleplay attacks, encoded payloads, and adversarial suffixes can override system prompts. **The correct approach**: Use system prompts as one layer, but enforce critical security constraints in code. Output validation, tool allowlists, and action confirmation flows don't depend on model compliance. *Reference: Chapter 16 (Defense-in-Depth Architecture)* --- ## Evaluation ### Mistake 23: Optimizing for BLEU/ROUGE Scores **What people do wrong**: Use BLEU or ROUGE as the primary optimization target for generation quality. **Why it's a problem**: These metrics measure lexical overlap, not semantic quality. Two equally valid responses can have zero n-gram overlap. Optimizing for these metrics produces outputs that game n-gram matching without being useful. **The correct approach**: Use BLEU/ROUGE as sanity checks and screening, not optimization targets. For quality optimization, use LLM-as-judge systems calibrated against human judgment, or direct human evaluation. *Reference: Chapter 11 (Why Traditional Metrics Fail)* --- ### Mistake 24: LLM Judges Without Calibration **What people do wrong**: Deploy LLM-as-judge evaluation without validating correlation with human judgment. **Why it's a problem**: LLM judges have systematic biases (verbosity preference, position bias, self-preference, confidence over correctness). Without calibration, you might ship degraded models or reject improvements. **The correct approach**: Build calibration sets with human ratings. Compute correlation, agreement rate, and systematic bias. Only trust judges that achieve correlation > 0.7 with human ratings. ```python def calibrate_judge(judge, calibration_examples): human_ratings = [ex['human_rating'] for ex in calibration_examples] judge_ratings = [judge.evaluate(ex) for ex in calibration_examples] correlation = np.corrcoef(human_ratings, judge_ratings)[0, 1] bias = np.mean(judge_ratings) - np.mean(human_ratings) if correlation < 0.7 or abs(bias) > 0.5: raise ValueError("Judge calibration failed. Adjust prompt or model.") ``` *Reference: Chapter 11 (Calibration: Making Judges Reliable)* --- ### Mistake 25: Pairwise Comparison Without Position Debiasing **What people do wrong**: Run pairwise comparisons in a single order (A vs B) without checking for position bias. **Why it's a problem**: LLM judges often prefer whichever response appears first (or last). The same two responses compared in opposite orders can yield opposite judgments. **The correct approach**: Run comparisons in both orders. If results disagree, mark as a tie or low confidence. Only trust consistent preferences. *Reference: Chapter 11 (Pairwise Comparison)* --- ### Mistake 26: Peeking During A/B Tests **What people do wrong**: Check results daily and stop the experiment when p < 0.05. **Why it's a problem**: This inflates false positive rates from the nominal 5% to 20-30%. You'll conclude there are differences when there aren't. **The correct approach**: Pre-commit to sample sizes and stopping rules. Use sequential testing methods if early stopping is required. Never peek without proper statistical adjustments. *Reference: Chapter 11 (Avoiding Common Statistical Pitfalls)* --- ### Mistake 27: Tuning on Test Data **What people do wrong**: Check test performance, make changes, check again, and iterate. **Why it's a problem**: You're effectively training on the test set. Final accuracy will be optimistically biased. This is data leakage. **The correct approach**: Use a separate validation set for iterative development. Touch the test set only once for final evaluation. If you must iterate, use cross-validation. *Reference: Chapter 3 (A Common Mistake)* --- ## Cost Management ### Mistake 28: Ignoring Costs During Development **What people do wrong**: Iterate quickly during development without tracking API costs, then discover hundreds of dollars spent. **Why it's a problem**: Unlike traditional software, AI has meaningful per-request marginal costs. Development and testing can rack up significant bills before you realize it. **The correct approach**: Track costs from day one. Set up budgets and alerts. Log token usage alongside functionality. ```python # RIGHT: Track costs alongside functionality class CostTracker: def __init__(self): self.total_input_tokens = 0 self.total_output_tokens = 0 def track(self, input_tokens: int, output_tokens: int): self.total_input_tokens += input_tokens self.total_output_tokens += output_tokens def estimated_cost(self, input_price: float, output_price: float) -> float: return (self.total_input_tokens / 1_000_000 * input_price + self.total_output_tokens / 1_000_000 * output_price) ``` *Reference: Chapter 4 (Common Mistake #8), Chapter 26* --- ### Mistake 29: Not Caching Repeated Queries **What people do wrong**: Process every query through the full LLM pipeline, even when many queries are identical or highly similar. **Why it's a problem**: Real-world query distributions often have heavy tails. The top 10% of queries might represent 60% of traffic. Processing these repeatedly wastes compute and money. **The correct approach**: Implement semantic caching. Cache exact matches. Consider approximate matching for highly similar queries. *Reference: Chapter 26 (Token Economics)* --- ### Mistake 30: Using Premium Models for All Workloads **What people do wrong**: Use GPT-5.5 or Claude Opus 4.8 for everything, including simple tasks that smaller models handle well. **Why it's a problem**: Premium models cost 10-100x more than capable smaller models. Simple classification, extraction, or templated generation often doesn't need frontier model capabilities. **The correct approach**: Route requests to appropriate models based on complexity. Use smaller models for simple tasks, reserving premium models for tasks that require their capabilities. *Reference: Chapter 26* --- ### Mistake 31: Verbose Prompts When Concise Suffices **What people do wrong**: Use 8,000-token prompts when 2,000 tokens would produce equivalent results. **Why it's a problem**: Input tokens cost money. Verbose prompts also increase latency and reduce the effective context window available for actual content. **The correct approach**: Iterate on prompt engineering to achieve the same quality with fewer tokens. Measure quality vs. prompt length tradeoffs explicitly. *Reference: Chapter 26 (Introduction)* --- ### Mistake 32: Ignoring Hidden TCO Costs **What people do wrong**: Compare options based only on visible infrastructure costs (compute, API calls). **Why it's a problem**: Engineering time often represents 40-60% of total cost of ownership. A system that costs $50K/month in compute but requires a full-time engineer to maintain has 30% hidden costs. Evaluation, monitoring, compliance, and disaster recovery also add up. **The correct approach**: Calculate full TCO including direct costs (compute, storage, APIs), indirect costs (engineering time, opportunity cost), and hidden costs (compliance, monitoring, DR). *Reference: Chapter 26 (Total Cost of Ownership)* --- ## Development Practices ### Mistake 33: Committing API Keys to Git **What people do wrong**: Store API keys in code, `.env` files, or configuration that gets committed. **Why it's a problem**: API keys in git history are exposed to anyone with repository access. Bots scan GitHub for leaked keys. Credentials in production configs create security vulnerabilities. **The correct approach**: Use environment variables. Never create `.env` files with real credentials. Add sensitive files to `.gitignore`. Use secrets management services in production. *Reference: Chapter 4 (Common Mistake #1)* --- ### Mistake 34: Retrying All Errors **What people do wrong**: Implement retry logic that retries on any error. **Why it's a problem**: Some errors (invalid API key, malformed request, content policy violations) will never succeed no matter how many retries. Retrying these wastes time and potentially money. **The correct approach**: Only retry transient errors (rate limits, timeouts, temporary server errors). Fail fast on permanent errors. ```python # RIGHT: Selective retry RETRYABLE_ERRORS = (RateLimitError, TimeoutError, ServiceUnavailableError) async def call_with_retry(func, max_retries=3): for attempt in range(max_retries): try: return await func() except RETRYABLE_ERRORS as e: if attempt == max_retries - 1: raise await asyncio.sleep(2 ** attempt) # Exponential backoff except (AuthenticationError, InvalidRequestError): raise # Don't retry these ``` *Reference: Chapter 4 (Common Mistake #4)* --- ### Mistake 35: Using Global pip Instead of Virtual Environment **What people do wrong**: Install packages with global pip, mixing dependencies across projects. **Why it's a problem**: Dependency conflicts between projects. Reproducibility issues. "Works on my machine" problems in deployment. **The correct approach**: Always use virtual environments. Verify `which pip` shows the venv path before installing. *Reference: Chapter 4 (Common Mistake #2)* --- ### Mistake 36: Ignoring Edge Cases During Development **What people do wrong**: Build for the happy path, ignoring error conditions, malformed inputs, and boundary cases. **Why it's a problem**: The happy path works great in demos. Production users find every edge case. Malformed documents, encoding issues, unexpected API responses, and unusual queries cause failures that are expensive to debug later. **The correct approach**: Build edge case handling from the start. Test with malformed inputs. Handle encoding issues. Validate assumptions about external data. *Reference: Chapter 4 (Common Mistake #7)* --- ## Cross-Team and Process Mistakes ### Mistake 37: Specializing in Tools Instead of Concepts **What people do wrong**: Build expertise as "LangChain expert" or "specific-framework specialist." **Why it's a problem**: Tools change; concepts persist. Framework expertise becomes dated. Concept expertise (agentic systems, evaluation methodology, retrieval architectures) transfers across tools. **The correct approach**: Learn concepts deeply. Use tools to implement concepts, but don't let tool expertise substitute for conceptual understanding. *Reference: Chapter 17 (Common Pitfalls in Specialization)* --- ### Mistake 38: Writing ADRs That Nobody Reads **What people do wrong**: Create multi-page Architecture Decision Records with exhaustive detail. **Why it's a problem**: If ADRs exceed two pages, they don't get read. The decision record becomes a design document that nobody consults. **The correct approach**: Keep ADRs concise. Focus on the decision, key alternatives considered, and reasoning. If you need more detail, write a separate design doc and reference it. *Reference: Chapter 20 (ADR Anti-Patterns)* --- ### Mistake 39: Treating All Decisions as One-Way Doors **What people do wrong**: Apply the same deliberation process to reversible decisions as to irreversible ones. **Why it's a problem**: Most decisions are two-way doors that can be reversed. Over-deliberating these slows execution without proportional benefit. **The correct approach**: Match decision speed to stakes. Two-way doors should be fast. One-way doors merit deliberation. *Reference: Chapter 20 (Match Decision Speed to Stakes)* --- ### Mistake 40: Premature Optimization **What people do wrong**: Spend a week writing custom CUDA kernels before validating the product. **Why it's a problem**: The product might pivot, making the optimization worthless. Optimization before validation is wasted effort. **The correct approach**: Get it working first, make it right second, make it fast third. Only optimize after validating that the system solves a real problem and identifying actual bottlenecks. *Reference: Chapter 21 (Pitfall 6)* --- ## Summary Checklist Use this checklist during design reviews: **RAG Systems** - [ ] Chunks have appropriate overlap - [ ] Embedding model is versioned with index - [ ] Incremental re-indexing is implemented - [ ] Retrieval is tested separately from generation - [ ] User input is delimited from instructions **Agents** - [ ] Architecture complexity is justified - [ ] Safety is enforced in code, not just prompts - [ ] Loop detection and iteration limits exist - [ ] Capabilities are minimally scoped **Prompts** - [ ] Instructions are specific with examples - [ ] Output format is specified - [ ] Context is clearly structured - [ ] Streaming is used for user-facing responses **Deployment** - [ ] Actual bottleneck is profiled before optimizing - [ ] Benchmarks include warmup - [ ] GPU timing includes synchronization - [ ] Testing uses production-scale inputs **Security** - [ ] Multiple defense layers exist - [ ] Indirect injection vectors are considered - [ ] Critical constraints are enforced in code - [ ] All data sources are treated as potentially malicious **Evaluation** - [ ] Metrics correlate with actual quality goals - [ ] LLM judges are calibrated against humans - [ ] A/B tests have pre-committed stopping rules - [ ] Test data is held out from development **Cost** - [ ] Token usage is tracked from day one - [ ] Caching is implemented for repeated queries - [ ] Model routing matches task complexity - [ ] Full TCO is calculated, not just compute --- ## Further Reading - Chapter 4: Your First LLM Application (development mistakes) - Chapter 6: Prompt Engineering (prompt design patterns) - Chapter 7: RAG Systems Deep Dive (retrieval pitfalls) - Chapter 8: Agentic Systems (agent safety) - Chapter 9: LLM Deployment & Infrastructure (serving optimization) - Chapter 11: MLOps & Evaluation (evaluation methodology) - Chapter 12: Security & Adversarial Robustness (security patterns) - Chapter 21: Performance Engineering (optimization pitfalls) - Chapter 26: Cost Engineering (economic mistakes)