Prompt Engineering & Evaluation

Prompt Engineering & Evaluation is the art and science of communicating effectively with AI models. It's where human creativity meets AI capability to produce reliable, high-quality results.

The Anatomy of a Great Prompt

Basic Prompt Structure:

text

System Message + Context + Instruction + Examples + Format + Constraints

1. System Prompts: Setting the Stage

System prompts define the AI's role, behavior, and constraints.

Basic System Prompts:

python

# Generic assistant
system_basic = "You are a helpful AI assistant."

# Role-specific
system_expert = "You are an experienced software engineer with 10 years of Python expertise."

# Constrained behavior
system_constrained = """
You are a customer service agent for TechCorp.
- Be polite and professional
- Never make promises you can't keep
- Escalate complex issues to human agents
- Keep responses under 100 words
"""

Advanced System Prompt Patterns:

python

# Self-evaluation system prompt
system_self_eval = """
You are a meticulous analyst. Follow this process:
1. First, provide your initial response
2. Then, critically evaluate your own response
3. Identify potential weaknesses or improvements
4. Provide a revised, improved version
5. Explain what changed and why
"""

2. Instruction Crafting: The Art of Being Specific

Bad vs Good Instructions:

python

# Vague and ambiguous
bad_prompt = "Tell me about AI"

# Specific and contextual
good_prompt = """
Explain artificial intelligence to a high school student:
- Focus on machine learning basics
- Use simple analogies
- Include 2-3 real-world examples
- Keep it under 300 words
"""

# Open-ended
bad_prompt = "Write some code"

# Constrained and specific
good_prompt = """
Write a Python function that:
- Takes a list of numbers as input
- Returns the mean and standard deviation
- Includes type hints and docstring
- Handles empty list edge case
- Has time complexity O(n)
"""

Advanced Instruction Patterns:

python

# Perspective-taking instructions
perspective_prompt = """
Analyze this business problem from three perspectives:

Problem: {problem}

1. **CEO Perspective**: Strategic impact and business value
2. **CTO Perspective**: Technical feasibility and requirements  
3. **CFO Perspective**: Cost analysis and ROI

For each perspective, provide 3 key considerations.
"""

# Comparative analysis instructions
comparison_prompt = """
Compare and contrast these two technologies:

Technologies: {tech_a} vs {tech_b}

Structure your analysis:
- **Similarities** (3-4 points)
- **Differences** (3-4 points each)
- **Best use cases** for each
- **Learning curve** comparison
- **Community and support**

Provide a final recommendation based on {criteria}.
"""

# Step-by-step decomposition
step_by_step_prompt = """
Solve this complex problem by breaking it down:

Problem: {complex_problem}

Follow these steps:
1. **Problem Analysis**: What's the core challenge?
2. **Constraint Identification**: What limitations exist?
3. **Solution Brainstorming**: List 3 possible approaches
4. **Approach Evaluation**: Pros/cons of each approach
5. **Optimal Solution**: Choose and justify the best approach
6. **Implementation Plan**: Step-by-step execution guide
"""

3. Few-Shot Learning: Teaching by Example

Providing examples dramatically improves performance and consistency.

Basic Few-Shot Pattern:

python

few_shot_prompt = """
Classify the sentiment of these customer reviews:

Review: "The product is amazing! Works perfectly."
Sentiment: Positive

Review: "Terrible quality. Broke after one week."  
Sentiment: Negative

Review: "It's okay, does the job but nothing special."
Sentiment: Neutral

Review: "Absolutely love this! Best purchase ever."
Sentiment: Positive

Now classify this new review:
Review: "{new_review}"
Sentiment:
"""

Advanced Few-Shot Patterns:

python

# Complex reasoning examples
reasoning_examples = """
Q: If a store has 5 apples and sells 2, then buys 3 more, how many apples does it have?
A: Let's think step by step:
   - Start with 5 apples
   - Sell 2: 5 - 2 = 3 apples  
   - Buy 3: 3 + 3 = 6 apples
   - Final answer: 6 apples

Q: A train leaves Station A at 9 AM going 60 mph. Another leaves Station B, 180 miles away, at 10 AM going 70 mph. When do they meet?
A: Let's think step by step:
   - From 9-10 AM, first train travels: 60 miles
   - Remaining distance: 180 - 60 = 120 miles
   - Relative speed: 60 + 70 = 130 mph
   - Time to meet: 120 / 130 ≈ 0.923 hours ≈ 55 minutes
   - Meeting time: 10:00 AM + 55 minutes = 10:55 AM
   - Final answer: 10:55 AM

Q: {new_question}
A: Let's think step by step:
"""

# Code generation with patterns
code_examples = """
Task: Create a function to find the maximum number in a list
Code:
def find_max(numbers):
    if not numbers:
        return None
    max_num = numbers[0]
    for num in numbers:
        if num > max_num:
            max_num = num
    return max_num

Task: Create a function to calculate factorial
Code:  
def factorial(n):
    if n == 0:
        return 1
    result = 1
    for i in range(1, n + 1):
        result *= i
    return result

Task: {new_task}
Code:
"""

4. Self-Evaluation & Reflection

Teaching the AI to critique and improve its own work.

Basic Self-Evaluation:

python

self_eval_prompt = """
First, provide an answer to the question.
Then, evaluate your answer using this rubric:

1. **Accuracy**: Is the information correct and factual?
2. **Completeness**: Did you address all parts of the question?  
3. **Clarity**: Is the explanation easy to understand?
4. **Conciseness**: Is it unnecessarily wordy?

Finally, provide an improved version if needed.

Question: {question}
"""

Advanced Reflection Patterns:

python

# Multi-pass refinement
refinement_prompt = """
Follow this 3-step process:

STEP 1 - INITIAL ANSWER:
Provide your first response to: {question}

STEP 2 - CRITICAL EVALUATION:
Evaluate your initial answer:
- What assumptions did you make?
- Are there any factual inaccuracies?
- Is the structure logical?
- What could be clearer or more complete?

STEP 3 - REFINED ANSWER:
Incorporate your evaluation insights to create an improved version.
"""

# Perspective-based evaluation
multi_perspective_eval = """
Answer the question, then evaluate from different perspectives:

ANSWER:
{initial_answer}

EVALUATION:
- **Expert Perspective**: Technical accuracy and depth
- **Beginner Perspective**: Clarity and accessibility  
- **Critical Perspective**: Potential weaknesses or counterarguments
- **Practical Perspective**: Real-world applicability

IMPROVED ANSWER:
Based on these evaluations, provide a comprehensive final answer.
"""

# Confidence calibration
confidence_prompt = """
Provide your answer, then rate your confidence:

QUESTION: {question}

ANSWER: {answer}

CONFIDENCE ASSESSMENT:
- High Confidence (90-100%): Very certain, based on solid knowledge
- Medium Confidence (60-89%): Reasonably sure, but some uncertainty
- Low Confidence (0-59%): Significant uncertainty or guessing

Justify your confidence level and note any areas of uncertainty.
"""

5. Prompt Optimization Techniques

A/B Testing Prompts:

python

def test_prompt_variations(question, prompt_variations, model):
    results = {}
    for name, prompt in prompt_variations.items():
        full_prompt = prompt.format(question=question)
        response = model.generate(full_prompt)
        results[name] = {
            'response': response,
            'length': len(response),
            'quality_score': evaluate_response_quality(response)  # Custom metric
        }
    return results

# Test different prompt styles
prompt_variations = {
    'direct': "Answer this question: {question}",
    'step_by_step': "Think step by step: {question}",
    'expert_role': "As a domain expert, answer: {question}",
    'teaching': "Explain this to a student: {question}"
}

results = test_prompt_variations("How do neural networks learn?", prompt_variations, model)

Automated Prompt Optimization:

python

class PromptOptimizer:
    def __init__(self, base_prompt, evaluation_metric):
        self.base_prompt = base_prompt
        self.evaluation_metric = evaluation_metric
        self.best_prompt = base_prompt
        self.best_score = 0
        
    def generate_variations(self, prompt):
        """Generate semantic variations of a prompt"""
        variations = [
            prompt + " Think step by step.",
            "You are an expert. " + prompt,
            "Provide a comprehensive answer to: " + prompt,
            "Break this down and explain: " + prompt,
            "Answer in detail: " + prompt
        ]
        return variations
    
    def evaluate_prompt(self, prompt, test_questions):
        """Evaluate prompt performance on test set"""
        total_score = 0
        for question in test_questions:
            response = self.generate_response(prompt.format(question=question))
            score = self.evaluation_metric(question, response)
            total_score += score
        return total_score / len(test_questions)
    
    def optimize(self, test_questions, iterations=5):
        """Find the best prompt variation"""
        for i in range(iterations):
            variations = self.generate_variations(self.best_prompt)
            for variation in variations:
                score = self.evaluate_prompt(variation, test_questions)
                if score > self.best_score:
                    self.best_score = score
                    self.best_prompt = variation
                    print(f"New best prompt (score: {score:.3f}): {variation}")
        
        return self.best_prompt, self.best_score

6. Evaluation Frameworks

Automated Evaluation Metrics:

python

def evaluate_response_quality(question, response, reference_answers=None):
    """Comprehensive response evaluation"""
    
    metrics = {}
    
    # Length appropriateness (not too short, not too long)
    word_count = len(response.split())
    metrics['length_score'] = max(0, 1 - abs(word_count - 150) / 150)
    
    # Structure evaluation
    metrics['structure_score'] = evaluate_structure(response)
    
    # Content evaluation (if reference answers available)
    if reference_answers:
        metrics['content_score'] = evaluate_content_similarity(response, reference_answers)
    
    # Readability
    metrics['readability_score'] = calculate_readability(response)
    
    return sum(metrics.values()) / len(metrics)

def evaluate_structure(response):
    """Evaluate response structure and organization"""
    structure_indicators = [
        len(response.split('\n\n')) > 1,  # Multiple paragraphs
        any(marker in response for marker in ['•', '- ', '1.', '2.']),  # List indicators
        '**' in response or '##' in response,  # Formatting
        len([s for s in response.split('. ') if len(s) > 10]) > 3  # Multiple sentences
    ]
    return sum(structure_indicators) / len(structure_indicators)

def calculate_readability(text):
    """Simple readability metric"""
    sentences = text.split('. ')
    words = text.split()
    
    if len(sentences) == 0 or len(words) == 0:
        return 0
    
    avg_sentence_length = len(words) / len(sentences)
    avg_word_length = sum(len(word) for word in words) / len(words)
    
    # Ideal: sentences 15-25 words, words 4-6 characters
    sentence_score = max(0, 1 - abs(avg_sentence_length - 20) / 20)
    word_score = max(0, 1 - abs(avg_word_length - 5) / 5)
    
    return (sentence_score + word_score) / 2

Human Evaluation Framework:

python

class HumanEvaluation:
    def __init__(self):
        self.criteria = {
            'accuracy': "Is the information factually correct?",
            'completeness': "Does it address all aspects of the question?",
            'clarity': "Is the response easy to understand?",
            'conciseness': "Is it appropriately brief without missing key points?",
            'helpfulness': "Would this response actually help someone?"
        }
    
    def create_evaluation_form(self, question, response):
        """Generate evaluation form for human raters"""
        form = f"""
        QUESTION: {question}
        
        RESPONSE: {response}
        
        Please rate on scale 1-5:
        """
        
        for criterion, description in self.criteria.items():
            form += f"""
            {criterion.title()} ({description}):
            [ ] 1 - Very Poor
            [ ] 2 - Poor  
            [ ] 3 - Adequate
            [ ] 4 - Good
            [ ] 5 - Excellent
            """
        
        form += """
        ADDITIONAL COMMENTS:
        What could be improved?
        """
        
        return form

7. Practical Prompt Templates

Template Library:

python

PROMPT_TEMPLATES = {
    'analysis': """
    Analyze: {topic}
    
    Required sections:
    1. Key Components
    2. Pros and Cons  
    3. Use Cases
    4. Alternatives
    5. Recommendations
    
    Format: Markdown with clear headings
    Length: 500-800 words
    """,
    
    'comparison': """
    Compare: {item_a} vs {item_b}
    
    Comparison criteria:
    - Performance
    - Cost
    - Ease of Use
    - Community Support
    - Learning Curve
    
    Provide a final recommendation for: {use_case}
    """,
    
    'tutorial': """
    Create a tutorial about: {topic}
    
    Audience: {audience}
    Prerequisites: {prerequisites}
    
    Structure:
    - Overview and objectives
    - Step-by-step instructions with code examples
    - Common pitfalls and how to avoid them
    - Next steps and additional resources
    
    Tone: Friendly and encouraging
    """,
    
    'debugging': """
    Debug this issue: {problem_description}
    
    Code: {code_snippet}
    
    Follow this process:
    1. Identify potential causes
    2. Suggest diagnostic steps
    3. Provide fix recommendations
    4. Explain the root cause
    
    Focus on: {specific_concern}
    """
}

def get_template(template_name, **kwargs):
    """Get and format a prompt template"""
    template = PROMPT_TEMPLATES.get(template_name)
    if not template:
        raise ValueError(f"Template {template_name} not found")
    return template.format(**kwargs)

Key Takeaways

1. Be Specific: Vague prompts get vague answers

2. Provide Context: Role, audience, and constraints matter

3. Use Examples: Few-shot learning dramatically improves results

4. Structure Matters: Clear organization helps the AI follow your thinking

5. Self-Evaluation: Teach the AI to critique and improve its own work

6. Test and Iterate: A/B test different prompt strategies

7. Quality over Quantity: One well-crafted prompt beats many mediocre ones

Prompt engineering is an iterative process. The best prompts evolve through testing, evaluation, and refinement based on real-world performance!