Short write‑ups of problems we hit running a real commerce operation and what we changed in the code because of them. No product announcements; just the reasoning.
We asked eBay for new orders forty thousand times a day to find about twenty. A customer still waited eighty minutes for one. Why checking more often was the wrong fix, what we did instead (check the channels that sell, and check the moment a customer comes looking), and the column we added so we can tell whether it worked. Results pending.
One morning every page in the back office threw errors for about two seconds, and the trail led somewhere strange — a queue that had been renamed in June and never given a worker, quietly growing to 6.9 million jobs and 18 GB inside Redis until the box was living in swap and one routine Lua script stalled past the busy threshold. How we traced a frontend error bucket to a key nobody was reading, why deleting it had to happen before fixing it, the eager-loading bug hiding behind the first fix, and the one-hour check that would have caught the whole thing in week one.
Everything that is not a page view runs on Horizon — twenty‑six supervisors across two servers, from "urgent" to "historic backfill". It is the best thing about the system and was the source of its worst mornings: stray workers surviving a deploy and running a query 780,000 times an hour; a supervisor killed but its children kept alive so every queue ran twice; a retry‑after shorter than the job; ten thousand jobs in a queue from a loop that should have been one. What each one taught us, in the order they hurt — and how the beast was finally tamed.