The job queue with nobody in the middle

Every Python project I have worked on hits the same moment. Something in a request handler is too slow to do inline — resize the image, send the email, regenerate the report — and the obvious answer is "put it on a queue."

So you go looking. And the answer the ecosystem gives you is:

	What you install	What you operate
Celery	celery, kombu, billiard, vine, amqp, …	a Redis or RabbitMQ server
RQ	rq, redis	a Redis server
Dramatiq	dramatiq, pika or redis	a broker

That is roughly 40 MB of transitive dependencies and a server process you now have to run, monitor, back up, upgrade, and explain to whoever is on call. For a fleet of machines, that is a completely reasonable trade. For a cron box, a CLI tool, a desktop app, a CI runner, or a Raspberry Pi, it is enormous overhead bolted onto a problem that fits in a file.

I spent a hackathon weekend finding out what is actually left if you delete the broker. The answer is nobroker: a durable, crash-safe job queue for Python that survives kill -9, is safe across processes, and has zero dependencies — not "one small one," not "just for dev." dependencies = [], verified mechanically in CI.

The name is the thesis. Every other job queue makes you run a broker. This one doesn't.

from nobroker import Queue, Worker

q = Queue("./jobs")
q.enqueue({"send_email_to": "ada@example.com"})

def handle(job):
    send_email(**job.payload)      # raising = retry with backoff

Worker(q, handle).run()            # Ctrl-C finishes in-flight work, then exits

That is the whole setup. No server, no connection string, no pip install beyond the package itself.

---

First: what is a broker actually doing?

This is the question the whole project turns on, and it is worth being precise, because "just use Redis" is usually said without unpacking it.

A message broker gives you exactly three things:

1. Durable storage. Jobs survive the process that created them.
2. A serialisation point. Two workers never get handed the same job at the same instant.
3. A network endpoint. Producers and consumers on different machines can reach the same queue.

Now notice what happens when all your producers and consumers are on one machine. You do not need (3) at all — and the operating system has been shipping industrial-strength implementations of (1) and (2) for about fifty years:

• Durability is fsync. That is the primitive Postgres and SQLite and Redis are all ultimately calling. There is nothing underneath it.
• Serialisation is a kernel file lock — fcntl.flock on POSIX, msvcrt.locking on Windows. The kernel arbitrates, it is fair, and — this is the good part — it releases the lock automatically when the holding process dies, which is more than most brokers manage.

So a single-machine job queue does not need a broker. It needs a file and a lock, used correctly. Everything else people associate with a queue — leases, visibility timeouts, retries with backoff, dead-letter queues, priorities — was never in the broker anyway. In a Celery deployment all of that is application logic that happens to ship inside a package. Redis does not know what a "visibility timeout" is.

Which reframes the project. I am not reimplementing Redis. I am writing the layer that would have sat on top of Redis regardless, and swapping a network protocol for two syscalls.

---

The design in one page

There is exactly one durable thing on disk: an append-only write-ahead log.

jobs/
  emails.000001.log     the log — the only durable state
  emails.current        one line: which generation is authoritative
  emails.lock           the cross-process lock

Everything in memory — the priority heaps, the lease table, the dead-letter queue — is a cache of that log. This is the single most important decision in the project, and it is worth stating why: there is no second source of truth, so there is nothing to keep consistent. Recovery isn't a repair algorithm. It is "read the file from the start."

Record framing

Records are appended, never modified:

file header:  <8s magic><H version><I generation><H reserved>    16 bytes
record:       <I length><B type><I crc32><json payload>          9 + n bytes

Two independent guards, catching two different failures:

• The length prefix catches an incomplete record. The process died mid-write; there are fewer bytes here than the header promised.
• The CRC-32 catches a complete-length record with a hole in the middle. This is the one that would otherwise be applied silently, and silent is by far the worse failure.

The test suite flips every single bit of a record and asserts each flip is detected.

The payload is JSON, deliberately, and it costs me about 7% of a non-durable enqueue versus a binary encoder. I took that trade knowingly: a durability format you cannot read with your eyes is a durability format you cannot debug. When something goes wrong at 2am, nobroker inspect prints the log record by record and you can just look at it. That was worth more than 7%.

(pickle was never a candidate. It executes arbitrary code on load, which is an absurd property for a file whose entire job is to be read back after a crash.)

Every operation is four beats

1. Take the file lock.
2. Read forward from our last offset — apply whatever peers appended.
3. Reclaim leases whose visibility timeout expired.
4. Append the new records, fsync, THEN apply them in memory.

Beat 2 is what makes multi-process work with no coordinator. A process that has been idle for an hour doesn't need to be told anything; it takes the lock, reads forward, and finds out what happened. There is no gossip, no heartbeat protocol, no membership. The log is the communication channel.

Beat 4's ordering is not negotiable. Disk before memory, always. A crash between the append and the in-memory apply replays to exactly the same state on restart. The other order loses an operation the caller was already told had succeeded — which is the one thing a durable queue must never do.

---

The idea that makes the whole thing testable

If I could keep only one sentence from this project, it would be this:

Replay is a pure function of the log.

Nothing in the replay path reads the clock, generates a UUID, or samples random jitter. Every non-deterministic value — the lease deadline, the jittered retry timestamp, the job id — is decided once, by the writer, and recorded as an absolute value in the record.

That sounds like a small stylistic preference. It is not. It is the difference between a queue you can test and one you can only hope about.

Because replay is deterministic, a log has exactly one correct recovered state. Which means I can write this test:

Take a real log. Truncate it at byte 1. Recover. Assert the state is consistent. Truncate at byte 2. Recover. Assert. Byte 3. Byte 4. …every byte to the end.

A crash can only interrupt a write at a byte boundary. So if recovery is correct at all ~1,300 byte boundaries of a real log, it is correct for any crash that log could possibly have suffered. That is not a sample of the failure space; it is the whole space, enumerated.

unittest's subTest turns that into ~1,300 independently reported cases inside one test method, each naming its byte offset when it fails. It accounts for about 33 seconds of the suite's 40-second runtime and I would not trade it for anything.

If jitter had been re-sampled at replay time — the obvious, natural way to write it — there would be no single right answer to assert against, and that test could not exist. One property, taken seriously early, bought the entire testing strategy.

---

Leases, and the lie of exactly-once

Workers don't remove jobs from the queue. They lease them:

job = q.lease_one()          # invisible to everyone else for 30 seconds
if job:
    try:
        do_the_work(job.payload)
        q.ack(job)                       # done
    except Exception as exc:
        q.nack(job, error=str(exc))      # retry later, or DLQ after max_attempts

If the worker dies holding a lease, nothing needs to notice. The lease deadline passes, the next operation reclaims it, another worker picks it up. Failure handling requires no failure detector — it's just a timestamp comparison.

This buys durability. It also means a job can be delivered more than once, and I want to be very blunt about that, because the marketing around queues is not.

Here is the exact window, and no amount of engineering closes it:

1. A worker leases a job and runs the handler.
2. The handler completes its side effect — the email is actually sent.
3. The worker is killed before it can call ack().
4. The lease expires; another worker runs the job.

The email goes out twice. Step 2 and step 3 are in different systems. To make that atomic you would need the handler's side effect and the queue's acknowledgement to commit in a single transaction — which means the queue has to live inside your database, at which point it is not a general-purpose queue any more.

So: nobroker is at-least-once. Your handlers must be idempotent. That is stated in the README, in the docs site, in the API reference, and on the badge. It is not a limitation I plan to fix in v2, because it is the strongest honest guarantee a queue of this shape can make.

What I can do is make the failure loud instead of silent:

Fencing tokens. Every lease carries a random token. If your lease quietly expired and the job was redelivered to someone else, your ack() is rejected with NotLeasedError rather than cheerfully completing a delivery that is no longer yours. The idea is from Martin Kleppmann's writing on distributed locks, and it is about fifteen lines. It converts a silent correctness hole into an exception you can see in your logs.

Lease heartbeats. The Worker extends the lease of a handler that is still running, so a slow-but-healthy worker is not a source of duplicate work. Only real failures cause redelivery.

Idempotent enqueue. q.enqueue(payload, job_id="order-42") de-duplicates on the key. This is the one place exactly-once is honestly available, because de-duplication on a key is something a log can actually do. A retried HTTP request that enqueues twice schedules one job.

---

Two heaps, not one

Small design note that I think is the nicest bit of the scheduler.

Jobs have an integer priority, and jobs can be delayed (delay=3600). The tempting implementation is one heap sorted by (priority, available_at).

That is broken, and interestingly so. Enqueue an urgent job scheduled for tomorrow, and it sits permanently at the head of the queue. Every lease looks at it, decides it isn't due yet, and returns nothing — while a thousand ready jobs sit behind it, starved by a job nobody can run.

So there are two heaps:

• _ready — eligible jobs, sorted by priority.
• _scheduled — not-yet-due jobs, sorted by availability time.

promote(now) moves due jobs from the second to the first, and it is the only place the clock is read on the read path.

The other half of this is that heapq has no remove operation, which is normally where people pip install sortedcontainers. It turns out the heapq docs themselves document the workaround: leave the stale entry in place, tag each entry with a version number, and skip it when it surfaces. Eight lines. Reading the standard library docs all the way to the end was worth more than a dependency.

---

Four bugs

The build was smoother than it had any right to be — the first end-to-end smoke test passed on the first run. Then the tests that actually simulate hostility started finding things. These four are the reason I trust the result at all.

Bug 1 — the in-memory index kept jobs the log had forgotten

Symptom. After compact(), stats() still reported total=3, done=1, but the compacted log on disk held only the 2 live jobs.

Cause. compact() rebuilt the heaps but left the job table untouched.

Fix. Index.retain(live) replaces the job table and rebuilds the heaps.

The fix is one line of description and completely uninteresting. What is interesting is the failure shape: the disagreement would have vanished on the next restart. Correct after a reboot, wrong until then. Those are the bugs that survive for years, because every attempt to reproduce them starts with restarting the process.

I only caught it because the test asserted on stats() immediately after compaction, rather than reopening the queue first. Reopening would have hidden it. There is a lesson in that about what "verify the result" means — check the live object, not a freshly-loaded one.

Bug 2 — os.replace can't overwrite an open file on Windows

Symptom. PermissionError: [WinError 5] from compact() whenever a second process had the queue open — i.e. in exactly the multi-process case compaction exists to serve.

First attempt. Windows refuses to delete or rename a file anyone has open unless every holder opened it with FILE_SHARE_DELETE, which os.open does not set. Fine — ctypes is standard library, so I opened the log through kernel32.CreateFileW directly and handed the handle to the C runtime with msvcrt.open_osfhandle.

It did not fix it. FILE_SHARE_DELETE permits deletion and handle-based renames, but MoveFileEx — which os.replace uses underneath — refuses regardless. I verified that with a ten-line standalone reproduction before concluding, and I'm glad I did, because "add the flag and move on" would have left me with a fix that didn't fix anything and a plausible story about why it should have.

The actual fix was a design change. Stop overwriting the log. Version it:

emails.000001.log     the live generation
emails.current        one line of text naming it

Compaction writes a new numbered generation, fsyncs it, and then replaces the tiny pointer file. Nobody ever holds the pointer open — it is opened, read, and closed inside a single call — so replacing it is always permitted.

And that flip is now the commit point, with a genuinely pleasant property: before it, the old generation is authoritative and the new file is ignorable garbage; after it, the reverse. There is no in-between state. Compaction is all-or-nothing, on both platforms, with no platform-specific branch.

Peers detect compaction by comparing an integer instead of stat'ing inodes. The design the platform limitation forced on me is better than the one I set out to write, which is a thing that happens more often than I expect it to.

(I kept the ctypes work. FILE_SHARE_DELETE is still what lets the sweep delete a retired generation while a peer has it open — ordinary POSIX behaviour that Windows otherwise forbids.)

Bug 3 — four processes overwriting each other's records (the serious one)

This is the bug that justifies the whole test suite.

Symptom. The very first run of test_concurrency.py: four producer processes enqueueing 40 jobs each produced 35 jobs, not 160. Four consumer processes leased 615 jobs from a queue of 200.

Cause. LogFile._end — the offset to append at — was a per-process cached value, and scan() never refreshed it. Process A appends at offset 1000. Process B, whose cache still says 600, appends on top of A's records. Total corruption. The duplicated leases weren't a leasing bug at all; they were a downstream symptom of a mangled log being replayed.

Fix. scan() now takes the file end from the kernel (lseek(SEEK_END)) rather than from its own memory. Since scan() runs at the top of every operation, under the lock, the offset is always current at the moment of the append.

Why this one matters more than the others: it was invisible to every single-process test. The entire 51-test queue suite passed throughout, before and after. It only appears with genuinely separate OS processes — which is precisely the claim the project rests on. If it had shipped, nobroker's headline feature would have been silently broken, and the demo would have worked perfectly.

There is a general lesson here that I keep relearning: a cache of kernel state is a bug waiting for a second process. The kernel is the only thing that knows how big the file is. Anything else is a guess that happened to be right so far.

I pinned it with a test at the layer where the bug actually lived (test_a_stale_end_offset_is_refreshed_by_scanning), not through the queue — because a test that reproduces it through four processes proves it's fixed, but a test at the log layer explains what was wrong.

Bug 4 — os.open defaults to text mode on Windows

Symptom. One test failed roughly 1 run in 6. The reopened queue was missing a varying number of trailing jobs, and recovery reported BAD_CHECKSUM — in a file that compaction had just written and fsynced.

Intermittent, varying, and in freshly-written data. Genuinely unpleasant.

Cause. On Windows, a descriptor from os.open without O_BINARY is in text mode, and the C runtime silently rewrites every 0x0A byte in your buffer as 0x0D 0x0A. The main log escaped this because it goes through msvcrt.open_osfhandle(..., O_BINARY). Compaction used a plain os.open.

And here is why it was intermittent. JSON escapes newlines inside payloads, so the payload never contains a raw 0x0A. The corruption only fired when a record's binary length or CRC field happened to contain a 0x0A byte — which depends on the exact size and checksum of that particular record. So it appeared randomly, and it surfaced at reopen, far from the code that caused it.

Fix. O_BINARY = getattr(os, "O_BINARY", 0), OR-ed into every os.open that writes bytes. Zero on POSIX, so it costs nothing there.

The regression test is my favourite thing in the suite. Rather than hope for a bad record, I made one deterministically: the payload {"a":"bc"} encodes to exactly 10 bytes, which makes the little-endian length prefix b"\x0a\x00\x00\x00" — a literal newline inside the frame, every single time. 150 consecutive compaction round-trips now pass; before the fix, about one in six failed.

If you take one thing from this section: when a bug is intermittent, the fix is not "run it more times." It is to find the input that makes it deterministic.

---

Performance, and the number I lost

I profiled with cProfile before optimising anything, which I mention only because I nearly didn't.

Two findings on a non-durable enqueue loop:

• Opening and closing the lock file was 44% of the time. Not locking — open and close. Fixed by holding the lock descriptor for the queue's lifetime and only taking and releasing the lock itself. Both flock and msvcrt.locking associate the lock with the open file description, so this is exactly as exclusive as reopening every time, and the kernel still drops the lock when the process dies, because that is the descriptor closing.
• Re-reading the pointer file was another 11%. Replaced with a stat and a cached value: compaction replaces the pointer atomically, so a replacement always arrives as a different file identity, and an unchanged stat is proof the answer hasn't changed.

Result: 2,550 → 6,014 non-durable enqueues/sec (2.4×), and 980 → 1,348 durable ones. No semantic change; all 122 tests green before and after.

Here is the published table:

Operation	ops/sec	µs/op	Notes
enqueue (fsync per job)	1,348	741.7	the durability guarantee, paid one job at a time
enqueue_many (one fsync)	57,676	17.3	batching amortises the fsync
enqueue (fsync=False)	6,014	166.3	not durable — shown for contrast
lease+ack round trip	1,433	697.9
cold-start replay	47,732	21.0	full replay, including CRC of every record
compact	537,184	1.9	2.6 MB → 16 bytes
nack + reschedule	6,712	149.0

(Windows 11, NVMe SSD, Python 3.11.9. NVMe versus a spinning disk changes the fsync rows by orders of magnitude — run it on your own hardware, that is the only number that matters.)

"Is it faster than Redis?"

No. Redis will do 50,000–100,000 ops/sec over loopback. nobroker does about 1,300 durable enqueues per second.

I published that number prominently anyway, because the comparison is not what it looks like. Redis, by default, is not calling fsync on every write. It is running appendfsync everysec, which can lose up to a second of writes it has already acknowledged. nobroker is paying for a guarantee Redis is not making.

The honest comparisons:

• Against Redis with appendfsync always — the config that makes the same promise — you are in the same order of magnitude, and nobroker saves you a network hop and a server.
• Against Redis default — Redis wins on throughput, with a weaker guarantee. Queue(fsync=False) is the comparable nobroker setting, and it is labelled "not durable" everywhere it appears, including in that table.
• On batches, enqueue_many does 57k/sec durably, because an fsync costs the same for one record as for a thousand.

If you need 100k jobs/sec, run a broker. If you need 1,000 jobs/sec that are still there after the power cut, this is simpler and there is nothing to operate.

I would rather publish a number I lose on with the reason attached than quietly benchmark against something that isn't making the same promise.

---

What the standard library actually gave me

I kept a running list of every point where I would normally have typed pip install. It ended up at twenty entries. A few highlights:

Would have installed	Used instead
redis + a Redis server	an append-only file, os.fsync, fcntl.flock
msgpack	struct.Struct("<IBI") + json
crc32c	zlib.crc32
filelock	fcntl.flock / msvcrt.locking, behind a 25-line shim
sortedcontainers	two heapq heaps + the lazy-deletion pattern from heapq's own docs
tenacity	min(base * factor ** (n-1), max_delay) plus random.uniform
apscheduler	a float timestamp and a comparison
pydantic	@dataclass(slots=True) with explicit to_dict/from_dict
click	argparse with set_defaults(func=…)
pywin32	ctypes.windll.kernel32 + msvcrt.open_osfhandle
pytest	unittest — 122 tests, no plugins
freezegun	injecting the backoff policy instead of patching the clock
deptry	ast + sys.stdlib_module_names

Three of those changed how I think.

filelock exists because there is no portable stdlib call — not because the stdlib lacks the capability. fcntl and msvcrt are both perfectly good; the gap is a single function name that works on both. Twenty-five lines closed it. And notice what my shim does not contain: no pid file, no stale-lock timeout, no "is that process still alive?" heuristic. Those are all things you write when your lock is not kernel-backed, and they are all subtly wrong. A kernel lock just… goes away when you die.

freezegun is a workaround for a design where the clock is implicit. I expected to need it for the retry tests. I didn't, because making the backoff policy an injected object meant tests that care about retry state could set the delay to zero and assert on state directly. Only two tests genuinely test timing, and they sleep for 150 ms. If you find yourself reaching for a clock patcher, it is worth asking whether the clock should have been a parameter.

sys.stdlib_module_names was a genuine surprise. I did not know it existed. It is a frozen set, maintained by the people who decide what the standard library is, and it turns "are we actually zero-dependency?" from a promise into an assertion. tools/check_deps.py parses every source file with ast — reading it without importing it, which matters, because importing a module to inspect it runs its top-level code — and checks each imported top-level module against that set. CI fails if anything third-party appears.

Using deptry to prove a zero-dependency claim would have been self-refuting, so that felt like the right hundred lines to write. CI actually checks the claim at three levels: source imports, the manifest, and Requires-Dist inside the built wheel.

And where the stdlib was worse

A log of only wins is not a log. Four places it genuinely cost me:

• No portable file lock, per above.
• os.open defaults to text mode on Windows — bug 4, and it cost me a real debugging session.
• os.replace can't overwrite an open file on Windows, and unlike deletion, FILE_SHARE_DELETE doesn't rescue you — bug 2, which forced a redesign.
• fsync on a directory doesn't exist on Windows, so the durability of a rename is weaker there. Documented rather than papered over.
• json is slow relative to a binary encoder, and can't represent bytes. Accepted knowingly, in exchange for a log you can read.

---

What it does not do

Stated plainly, because a naive implementation that is honest about its corners beats a fast one that hides them:

• Single machine only. The lock is a kernel file lock. It does not work over NFS or SMB and will never coordinate two hosts. There is no distributed mode and there will not be one. If you need a fleet, run Celery — that is what it's for.
• At-least-once, never exactly-once. Handlers must be idempotent.
• Polling, not push. A broker-less queue has nobody to send a notification. An idle worker costs one stat and one short read per poll — cheap, but not free and not zero-latency.
• The whole index lives in memory. Roughly 500 bytes per job, so a million jobs is ~500 MB. Fine for the workloads this targets; it is not a database.
• stats() is O(n). It counts by iterating.
• Recovery reads the entire log at startup — ~48k records/sec, so a million-record log takes about 20 seconds to open. Compact regularly.
• No result backend, no chaining, no workflows, no cron DSL, no async handlers. All deliberately out of scope, and scope discipline is most of why the thing finished.

"Why not just use sqlite3? It's in the standard library too."

The best objection, and I got it from three different people.

SQLite would give me storage and transactions. It would not give me any of the things this project actually is: lease semantics, visibility timeouts, backoff with jitter, fencing tokens, a dead-letter queue, priority ordering that interacts correctly with delayed jobs, or crash-recovery semantics I can reason about end to end.

A job queue on SQLite is all of this same code, written on top of a query layer I don't need, plus a schema, plus the BEGIN IMMEDIATE / busy_timeout dance required to stop polling workers livelocking. More code, not less.

The honest counterpoint: SQLite's storage engine is vastly better tested than mine — by roughly a trillion device-years. If you need a queue you'd bet a company on today, that is a real argument. If you want to understand what the primitives underneath actually cost, this is the more interesting build, and it was the one the weekend was for.

---

The takeaways I'd actually keep

Find the property that makes your system testable, and take it seriously early. "Replay is a pure function of the log" is one sentence, and it is the reason a test can enumerate every byte offset of a crash instead of sampling a few. If jitter were re-sampled at replay time — the natural way to write it — that test could not exist.

Test the thing you claim. Three of the four bugs were invisible to single-process happy-path tests. The concurrency test spawns four real OS processes; the crash test uses os.fork() + os._exit(9) so that atexit handlers, finally blocks and buffer flushes are all skipped, exactly as they would be under kill -9. Whatever is on disk at that instant is all recovery gets, which is the condition being tested. A gentler test proves nothing about the claim on the box.

Intermittent means you haven't found the input yet. Bug 4 went from "fails 1 run in 6" to "fails every time" the moment I picked a payload whose length prefix contained the byte I needed. Then it took ten minutes.

Platform limitations sometimes hand you a better design. I did not want generation files and a pointer flip. Windows made me, and the result is all-or-nothing compaction with no platform branch and peers that detect changes by comparing an integer.

Publish the number you lose on. "Slower than Redis, here's the number, here's why the comparison is unfair in both directions" is more useful — and more credible — than a benchmark quietly chosen to win.

Read the docs to the end. heapq's own documentation contains the removal workaround people install sortedcontainers for. sys.stdlib_module_names was sitting there the whole time. A surprising fraction of "I need a package for this" is "I stopped reading at the third heading."

---

Try it

pip install nobroker          # installs one package and nothing else

• Docs and live demo: ayanbag.github.io/nobroker
• Interactive playground: ayanbag.github.io/nobroker/playground.html — enqueue jobs, kill the process mid-write, flip a bit in the log, and watch recovery find the tear. It reimplements the real record frame in JavaScript, with real CRC-32 and real byte offsets, and it says in its own footer exactly which parts are real and which are simulated.
• Source: github.com/ayanbag/nobroker
• STDLIB.md — all twenty "I'd normally have installed X" entries, including the places the standard library was the worse option.

~3,000 lines of library, ~1,700 lines of tests, 122 tests, 0 dependencies.

Built for the Zero Dependency Hackathon, Track D — Data & Storage. The premise of that event is that you find out how much the standard library already gives you. It gave me a job queue.

---

Related Links

github.com

github/nobroker

The source code, with the README, the test suite, and the CI that verifies zero dependencies

ayanbag.github.io

ayanbag.github.io/nobroker

The project site, with docs, a live demo, and an interactive playground that reimplements the record framing