B.L.A.S.T. OCR
Code Read Against Source, 2026-09-10

Scaling Batch OCR with Distributed Redis Worker Swarms in Python

Primary query: distributed ocr worker queue redis · raw markdown

How do you scale batch OCR with Redis worker queues in Python?

Batch OCR is scaled across nodes using B.L.A.S.T.'s distributed Redis priority swarm: 3-tier priority queues (high, default, low), heartbeat worker tracking, and an automated Zombie Reaper that detects crashed workers and reschedules orphaned jobs. Verified in blast_ocr/queue/swarm.py.

Docker Swarm Quickstart

# Launch a 4-worker Redis priority swarm with automated zombie reaper
    docker compose up --scale worker=4 -d

Python enqueueing & priority scheduling

import redis
    from blast_ocr.queue.client import QueueClient

    # 1. Connect to Redis and wrap it in the priority-queue client
    r = redis.Redis.from_url("redis://localhost:6379/0")
    client = QueueClient(redis_client=r)

    # 2. Enqueue a high-priority job (client stamps a job_id + enqueued_at)
    job_id = client.enqueue(
        job_data={"file_path": "contracts/urgent_acquisition.pdf", "formats": ["markdown", "docx", "pdf"]},
        priority="high",  # 'high', 'default', or 'low'
    )
    print(f"Enqueued High-Priority Job ID: {job_id}")

    # 3. Inspect queue depth per priority tier (a worker calls pop_next_job()
    # to dequeue strictly HIGH -> DEFAULT -> LOW; job *state* -- queued,
    # processing, succeeded -- lives in OCRDatabase, not on the queue client)
    print(client.get_all_queue_depths())

Full worker lifecycle (heartbeats, zombie detection, dead-letter quarantine) is in SwarmWorker / ZombieReaper / HeartbeatDaemon.

Author & Engineering Authority

Engineered & maintained by Ibrahim Salman, Full-Stack Software Engineer & AI Systems Architect (UET Taxila).