The Independent Variable
DSPy Part 2: Fact-Checking 101 – Teaching AI to Stop Making Stuff Up

Photo by Dennys Hess on Unsplash

DSPy Part 2: Fact-Checking 101 – Teaching AI to Stop Making Stuff Up

Or: How to Turn a Bullshtting Intern into a Trustworthy Assistant*

Mehmet Öner Yalçın's photo
Mehmet Öner Yalçın
·

4 min read

The Problem: AI Loves to Wing It

Let’s revisit Chatty, our overeager intern. You ask them to fact-check the claim “The moon is made of cheese.”

Manual Prompting Approach:
You beg, “Chatty, please verify this claim. Check NASA’s website. Be accurate. Don’t make things up.”
Chatty responds: “The moon is NOT made of cheese. However, NASA once sent a mouse named Algernon to test lunar cheese viability.”

Sigh. Now you’re stuck editing prompts like a paranoid poet: “Chatty, use only verified sources. NO CREATIVE WRITING.”

Why this sucks:

  • You’re not coding—you’re negotiating with a gremlin.

  • Every tweak is guesswork.

  • Swap Chatty for Claude? Start over.

DSPy’s Solution: Teach Principles, Not Poetry

Instead of pleading with Chatty, let’s program their training. Think of DSPy as a coach that drills the intern with practice problems until they learn the principles of fact-checking.

Step 1: Define the Task

In DSPy, you declare what you want, not how to ask for it. Let’s build a FactCheck module:

class FactCheck(dspy.Module):  # <-- Your training manual for Chatty  
    def __init__(self):  
        super().__init__()  
        # Teach Chatty: "Given a claim, output (is_correct, explanation)"  
        self.generate_answer = dspy.Predict("claim -> is_correct, explanation")  

    def forward(self, claim):  
        return self.generate_answer(claim=claim)

What’s happening here:

  • dspy.Predict defines the input/output structure. Think of it as a mad-libs template:
    “Hey Chatty, here’s a claim*. I need an **is_correct label and an explanation**.”*

  • No prompts! DSPy will learn how to fill in the blanks.

Step 2: Train with Examples

Coaches use drills. DSPy uses teleprompters (optimizers) to teach your module. Let’s feed it practice claims:

trainset = [  
    dspy.Example(claim="The moon is made of cheese", is_correct=False, explanation="NASA confirms the moon is rocky."),  
    dspy.Example(claim="Water boils at 90°C at sea level", is_correct=False, explanation="Boiling point is 100°C."),  
]  

# Coach Chatty with these examples  
teleprompter = dspy.teleprompt.BootstrapFewShot()  
compiled_factcheck = teleprompter.compile(FactCheck(), trainset=trainset)

What’s happening here:

  • The BootstrapFewShot teleprompter auto-generates prompts and examples tailored to your task.

  • It’s like the coach whispering: “See? For false claims, cite a source. For true ones, explain briefly.”

Step 3: Deploy the Optimized Fact-Checker

Now, let’s test our trained compiled_factcheck:

response = compiled_factcheck(claim="The Earth is flat")  
print(response)

Output:

is_correct: False  
explanation: "Satellite imagery and centuries of scientific research confirm the Earth is an oblate spheroid."

No Shakespearean sonnets! DSPy’s “coaching” forced Chatty to internalize the principles of fact-checking.

Why This Works

  1. Declarative > Imperative: You define what (inputs/outputs), not how (prompts).

  2. Compile-Time Learning: The teleprompter finds the best “teaching strategy” upfront.

  3. Portability: The same FactCheck code can train Claude, GPT-4, or Llama 3.

The Magic Behind the Scenes

When you ran teleprompter.compile(), DSPy did this:

  1. Generated Candidate Prompts:
    “You are a fact-checking assistant. Analyze the claim and cite sources.”
    “Given a claim, determine its truthfulness. Be concise.”

  2. Tested Them: Ran both prompts on the trainset, tracked accuracy.

  3. Selected the Best: Chose the prompt that performed best.

You didn’t write a single prompt. DSPy treated it like a math problem—optimize for accuracy—not a creative writing exercise.

Before/After: Manual vs. DSPy

Manual Approach:

# Begging the model  
prompt = """  
You are a fact-checker. Verify this claim: {claim}.  
Rules:  
1. Be accurate.  
2. Cite sources.  
3. NO CREATIVITY.  
"""  
# (Pray)

DSPy Approach:

# Define logic, not pleas  
class FactCheck(dspy.Module):  
    def __init__(self):  
        self.generate_answer = dspy.Predict("claim -> is_correct, explanation")

Result:

  • Manual: Fragile, model-specific, exhausting.

  • DSPy: Robust, portable, automated.

What’s Next?

Right now, our fact-checker is a know-it-all—it doesn’t look up information. In Part 3, we’ll give it a library card: teaching it to retrieve facts from databases/Wikipedia before answering.

Teaser:

self.retrieve = dspy.Retrieve(k=3)  # "Go fetch 3 relevant facts, Chatty!"

TL;DR: DSPy replaces prompt begging with programmatic teaching. Define tasks, feed examples, and let algorithms find the best way to communicate with your model.

Stay tuned for Part 3, where we’ll turn Chatty into a librarian with a PhD in “Googling Stuff.” Code, analogies, and dad jokes included.

Homework: Try the code above! Swap the claim to “Mars is inhabited by robots” and see what happens. (Spoiler: It’s true. Say hi to Perseverance!)

dspy