Digitise handwritten forms with AI

The problem

You've got thousands of paper forms from events, surveys, or service intake. Typing them into a spreadsheet manually would take weeks. You need the data digitised so you can analyse it, but manual data entry is expensive and error-prone.

The solution

Take photos of the forms and use a vision AI model (GPT-4V or Claude) to extract the information into structured data. The AI reads the handwriting, identifies which field is which, and outputs CSV or JSON. You review confidence scores to spot anything that needs checking. What would take weeks becomes hours.

What you get

A CSV or JSON file with all your form data: names, addresses, responses, checkboxes ticked. Each entry includes a confidence score so you can review uncertain readings. You can spot-check low-confidence entries rather than reviewing everything.

Before you start

• Forms photographed or scanned (phone photos are fine if clear)
• A blank version of the form so you can show the AI what fields to extract
• An OpenAI API key or Claude account
• For batch processing: basic Python skills or willingness to adapt example code
• DATA PROTECTION CRITICAL: If forms contain beneficiary data (health, circumstances, personal details), you MUST conduct a Data Protection Impact Assessment (DPIA) before proceeding. Ensure your API subscription tier includes data privacy guarantees - check that data is not used for training and that you have a Data Processing Agreement in place. Consider redacting names/addresses with a black marker before scanning if you only need anonymised data (e.g., survey responses).

When to use this

• You've got more than 50 handwritten forms and manual entry isn't practical
• The forms are mostly legible (doesn't need to be perfect)
• You need the data quickly for a deadline
• Forms follow a standard template so the AI knows what to extract
• LOW-RISK data: Anonymous event feedback, survey responses, registration forms for non-sensitive activities - these are ideal candidates
• HIGH-RISK data: Beneficiary intake forms, health assessments, safeguarding disclosures - require DPIA, may need redaction, and should use enterprise-tier APIs with DPAs in place

When not to use this

• You've only got a handful of forms - might be quicker to type them
• The handwriting is genuinely illegible (heavy scribbles, water damage)
• Forms vary wildly in layout and the AI can't tell what's what
• The forms contain highly sensitive data you can't upload to external services

Steps

1. 1

   Photograph or scan your forms

   Take clear photos with your phone or use a scanner. Make sure the whole form is in frame, lighting is even, and text is readable. You don't need professional equipment - a modern phone camera works fine. Save as JPG or PNG.

2. 2

   Prepare a template image

   Take a photo of a blank form or create a diagram showing what each field is. This helps you explain to the AI what information to extract and what to call each field in the output.

3. 3

   Test with one form manually

   Upload one form image to Claude or ChatGPT Plus (which has vision). Give it the template and ask: "Extract the data from this form into JSON format. Here are the fields: [list them]. Include a confidence score (0-100) for each field." Check the result.

4. 4

   Refine your prompt

   If the AI misreads fields or gets confused, adjust your instructions. Be specific about field names, formats (dates, phone numbers), and what to do with blank fields. Test on 5-10 forms to get the prompt working reliably.

5. 5

   Process in batches

   For 10-20 forms, you can paste images into Claude or ChatGPT manually. For hundreds, use the API with the example code to process them automatically. The code loops through your images and saves results to CSV.

6. 6

   Review low-confidence entries

   Sort your results by confidence score. Anything below 80% probably needs checking against the original form. This is much faster than reviewing everything. Fix obvious errors (like reading '5' as 'S').

7. 7

   Validate against known totals(optional)

   If you know you should have 500 forms, check you got 500 rows. If forms have checkboxes, check the totals make sense. Look for impossible dates or missing required fields. This catches systematic errors.

Example code

from openai import OpenAI
import base64
import json
import pandas as pd
from pathlib import Path

client = OpenAI()

# Your form fields - adapt these to match your form
fields = {
    "name": "Full name",
    "email": "Email address",
    "phone": "Phone number",
    "postcode": "Postcode",
    "consent": "Consent checkbox (yes/no)",
    "feedback": "Free text feedback"
}

def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')

def extract_form_data(image_path):
    base64_image = encode_image(image_path)

    prompt = f"""Extract data from this form image.

Return JSON with these fields:
{json.dumps(fields, indent=2)}

For each field, include the extracted value and a confidence score (0-100).
If a field is blank or unclear, set value to null and explain in a 'notes' field.

Format:
{{
  "name": {{"value": "John Smith", "confidence": 95}},
  "email": {{"value": "john@example.com", "confidence": 100}},
  ...
}}"""

    response = client.chat.completions.create(
        model="gpt-4o",  # Vision-enabled model
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{base64_image}"
                        }
                    }
                ]
            }
        ],
        max_tokens=500
    )

    # Extract JSON from response (handles markdown-wrapped responses)
    content = response.choices[0].message.content.strip()
    if content.startswith('{'):
        return json.loads(content)
    else:
        # Extract JSON if wrapped in markdown code fences
        import re
        json_match = re.search(r'\{[\s\S]*\}', content)
        return json.loads(json_match.group()) if json_match else {}

# Process all images in a folder
results = []
image_folder = Path("form_images")

for image_path in image_folder.glob("*.jpg"):
    print(f"Processing {image_path.name}...")
    data = extract_form_data(image_path)

    # Flatten for CSV
    row = {"filename": image_path.name}
    for field, details in data.items():
        row[field] = details.get("value")
        row[f"{field}_confidence"] = details.get("confidence")

    results.append(row)

# Save to CSV
df = pd.DataFrame(results)
df.to_csv("extracted_forms.csv", index=False)

print(f"\nProcessed {len(results)} forms")
print(f"Average confidence: {df[[c for c in df.columns if 'confidence' in c]].mean().mean():.1f}%")

# Show low confidence entries for review
low_confidence = df[df[[c for c in df.columns if 'confidence' in c]].min(axis=1) < 80]
if len(low_confidence) > 0:
    print(f"\n{len(low_confidence)} forms need review (confidence < 80%):")
    print(low_confidence[["filename"]].to_string(index=False))

Tools

Resources