Code Read Against Source, 2026-09-10
Scaling Batch OCR with Distributed Redis Worker Swarms in Python
How do you scale batch OCR with Redis worker queues in Python?
Batch OCR is scaled across nodes using B.L.A.S.T.'s distributed Redis priority swarm: 3-tier priority queues (high, default, low), heartbeat worker tracking, and an automated Zombie Reaper that detects crashed workers and reschedules orphaned jobs. Verified in blast_ocr/queue/swarm.py.
Docker Swarm Quickstart
# Launch a 4-worker Redis priority swarm with automated zombie reaper
docker compose up --scale worker=4 -d
Python enqueueing & priority scheduling
import redis
from blast_ocr.queue.client import QueueClient
# 1. Connect to Redis and wrap it in the priority-queue client
r = redis.Redis.from_url("redis://localhost:6379/0")
client = QueueClient(redis_client=r)
# 2. Enqueue a high-priority job (client stamps a job_id + enqueued_at)
job_id = client.enqueue(
job_data={"file_path": "contracts/urgent_acquisition.pdf", "formats": ["markdown", "docx", "pdf"]},
priority="high", # 'high', 'default', or 'low'
)
print(f"Enqueued High-Priority Job ID: {job_id}")
# 3. Inspect queue depth per priority tier (a worker calls pop_next_job()
# to dequeue strictly HIGH -> DEFAULT -> LOW; job *state* -- queued,
# processing, succeeded -- lives in OCRDatabase, not on the queue client)
print(client.get_all_queue_depths())
Full worker lifecycle (heartbeats, zombie detection, dead-letter quarantine) is in SwarmWorker / ZombieReaper / HeartbeatDaemon.