Short write‑ups of problems we hit running a real commerce operation and what we changed in the code because of them. No product announcements; just the reasoning.
For as long as the storefront has rendered on the server, a page missing from the cache cost about a second, and we kept blaming the size of the page. It was a translation helper. Every translated word left a small reactive effect behind in the rendering process, nothing ever cleaned it up, and every page drawn in a different language re-ran all of them. How a cost that grew with uptime gave it away, a heap diff full of translation keys, the twenty-line fix, and what changed once a cache miss cost 36 ms instead of 1.2 s.
We built reply suggestions for the customer service chat inbox, set a goal of 60% close to what staff really sent, and spent ten days between 10 and 20%. Better writers, a critic, 23,000 past replies as examples and a coverage gate did not move the live number. We stopped. For a drafter that gets one shot, the technology at a cost that makes sense is not there yet, so the next attempt puts each staff member's own AI assistant on the conversation instead.
We read only the Gmail inbox, on purpose, and real customers were getting lost in Gmail spam. We now read the spam folder too. Rules clear most of it, a System One model sorts the rest for about four cents a month, anybody who has bought from us gets straight through, and nothing out of spam gets to open a file on its own.
We asked eBay for new orders forty thousand times a day to find about twenty. A customer still waited eighty minutes for one. Why checking more often was the wrong fix, what we did instead (check the channels that sell, and check the moment a customer comes looking), and the column we added so we can tell whether it worked. Results pending.
A label sheet is designed in a browser and printed by mPDF — two renderers that share no code and must agree to the millimetre. Where they disagreed, what happened when a barcode library quietly accepted an invalid EAN13 and encoded a different number than the digits printed under it, why the PDF thumbnails are rendered on the client, and the one Blade character that cost an afternoon.
The factory counts in units, the shop counts in boxes, and no amount of wishing makes 16 divide by 10. Rather than force one side to surrender, we named the exchange rate, converted at exactly two edges, and taught the system to say "one sealed box and six loose" out loud. Then we let small orders wait at the roadside until something bigger came past going the same way.
Maya ran warehouse pallet and stock work; Han handled clocking in and out. Both React Native apps were finished, tested and production-ready, and neither was ever launched. What beat them was not a new technology but a question nobody asked early enough, because the answer had been sitting on every supermarket checkout since the eighties. How we proved in production that nothing had ever called them, the two threads out of 58,000 lines that still reached back into the live app, and what a year of good work that never met a user is actually worth.
A dashboard that makes ordering easy invites dumping. What stops the purchaser ticking everything red and calling it someone else's problem - a hard cap built from measured deliveries, a shared warehouse split fairly, and a rule about never letting anyone edit the number that judges them.
Data can only predict the past. Our stock forecaster leans into that - measure the past honestly (including the days the shelf was empty), then say how wrong you might be.
Almost every piece of paper the business touches is a PDF we generated — invoices, delivery notes, picking sheets, pallet returns, timesheets, barcode labels in five widths, carrier labels from three APIs. How one rule decides between mpdf and headless Chrome, how a single invoice template serves shops in different countries, languages and tax regimes without forking, and the gotchas: a bakery called "& Fils" that broke shipping, labels that must scale from 55 mm to 125 mm, and why we merge PDFs with a renderer that never renders them.
One morning every page in the back office threw errors for about two seconds, and the trail led somewhere strange — a queue that had been renamed in June and never given a worker, quietly growing to 6.9 million jobs and 18 GB inside Redis until the box was living in swap and one routine Lua script stalled past the busy threshold. How we traced a frontend error bucket to a key nobody was reading, why deleting it had to happen before fixing it, the eager-loading bug hiding behind the first fix, and the one-hour check that would have caught the whole thing in week one.
The new system's name wasn't invented for a rebrand — and it wasn't even the first name. In October 2020, years before aiku served its first request, a file called trait.Aiku.php appeared inside Aurora, pushing entity changes to a system that barely existed. The app that eventually caught those changes was born as "Pika" — an experiment to extract warehouse picking into a micro-service, started on a warehouse visit in Eastern Europe — and only became aiku by a rename ticket in March 2023. This is the archaeology of a name.
Every codebase has a birthday; Aurora — the system aiku replaced — has a timestamp, a timezone and a table by the window. The header at the top of its entry point reads "Created: 27 August 2015 12:49:03 GMT+8, Singapore", and the commit from that day says "thought experiments to a full SPA". This is the origin story — the 2008 system that came before it, the habit of writing the city into every file header, and how "aiku" was a trait inside Aurora years before it was a product.
14,880 commits from 2008 to today, read not as a changelog but as a diary. The ticket prefixes that name the eras (KAKTUS, CSER, PIKA), the hour histogram with its 3am tail, a New Year's Day spent chasing a search bug with the typos getting worse by the hour, "brexit nightmare", "leap year hack", and the honest admission a commit message almost never makes: "Stock Levels don't add up..we think".
The next thing we are building — an MCP server for customers, so a trade buyer or a dropshipper can connect their own AI assistant to their account and ask, browse and eventually order without an integration project. What it will do first, what it will not do until someone accountable says so, and the groundwork already in place.
Every file in Aurora starts with a header recording the exact time — and the city — where it was created. Nobody asked for it, and for seven years it recorded nothing but a company name. Then, from October 2015, the headers start naming places, and read end to end they become a travel diary — Spain, Sheffield, Macau, Penang, a factory city in China, Bali, Bangkok, Mexico City, Tokyo, Kuala Lumpur — including three files created mid-flight. The typos are the proof it was all typed live.
Aurora has no framework, because in 2008 there was nothing worth adopting — so it grew its own. A 5,864-line router, an active-record ORM with a built-in audit trail, a process-forking job runner years before queues were a composer require, translations in sixteen languages with a homemade string extractor, and a Content-Security-Policy assembled by string concatenation. A tour of what one developer hand-rolls in fourteen years, what each piece is called today, and why the answer to "was it bad?" is more interesting than yes or no.
We did not embed the business into a vector store and hope the right chunk came back. We gave the model forty‑five tools — the same actions the staff app runs, under the same permissions — and a guide on how to use them. The result is a report that used to take three people a day, or a ticket to engineering, arriving in thirty seconds with real figures. The reactions, in order: disbelief, checking whether the numbers were random, and then a very quiet "oh".
Nobody on the team is an SEO specialist, and dozens of storefronts on one codebase rank anyway. That is because almost everything that matters is plumbing that a storefront engine either does for every shop or does for none — a computed canonical URL nobody types by hand, a 301 that follows a discontinued product to its family, a sitemap index regenerated nightly with a real lastmod, structured data that obeys the same price rules as the page. What we built, what we skipped on purpose, and the three things that bit us anyway.
We built an internal messenger into the ERP instead of sending the warehouse to a third-party app. Why it had to be separate from customer chat, why the first users were pickers on tablets, transparent translation between languages, chats that start from an order, and the rule that staff chat is work — so HR may read it.