Most of what I know about building backend systems came from running them, not from designing them. The lessons below are the ones that showed up across enough unrelated projects — the Quic-based networking library at NEXSOFT, the multi-tenant payment platform at ESOL, OrderX, the Web3 wallet platform, Multi Aura, Hihand Shortener — that I now treat them as defaults rather than heuristics.
Owning a shared library is owning a contract
The single most leveraged decision in my time at NEXSOFT was treating the internal Go networking library as a contract, not as a wrapper. The library exposed bidirectional streaming, progress reporting, and cancellation; every consumer saw the same shape; every consumer could be hardened in one place instead of N places.
The discipline that makes a shared library valuable is the discipline that makes it expensive to maintain: breaking changes go through a deprecation window, the interface evolves on a slower clock than the implementations, and the on-call burden belongs to the library team, not the consumers. A library that is "owned" by everyone is a library no one is willing to change.
The lesson carries to anything shared: a constant, an HTTP client factory, an idempotency helper. Pick one owner and let the owner be slow.
Change one hot path at a time
A backend system under load has hot paths that everyone wants to improve. The mistake is to change several at once. Two improvements that each move a latency metric 20% can, when shipped together, move the metric 5% in either direction and leave the team unable to tell which change did what.
The version that holds up is one-hot-path-at-a-time, with a clear before-and-after measurement. Pick a path, change it, measure it, and only then move on. This is slower and produces better systems.
Observability before scaling
Every backend system I have watched scale looked worse after scaling than before. The reason is usually that the team scales the request handlers before the team has the dashboards that explain the request handlers. The new instances are healthy, the latency metric is the same, and the team finds out three weeks later that a background queue is silently slow.
The version that holds up: ship the dashboards before the scale. Logs that include the request id, metrics that include the consumer group lag, traces that cross the async seam. The cost of the dashboards is small compared to the cost of guessing.
Contract-driven integration wins in the long run
The integrations I trust the most are the ones defined by a contract first — Protobuf for gRPC services, a typed event envelope for Kafka topics, an Open API spec for partner surfaces — and implemented second. The integrations I trust the least are the ones defined by a JSON object that was convenient at the time.
A contract-driven integration pays for itself twice: once when the second consumer is built, and once when the producer ships a change without breaking the consumer. Protobuf and similar tools are not the point; the discipline of writing the contract down before writing the code is the point.
Idempotency is the storage layer, not the handler
I have written this lesson in some form on every project that touched a queue. The longer version is that idempotency is also how you keep the on-call rotation sane. A consumer that crashes mid-side-effect and quietly retries on the next instance is a consumer that nobody pages for; the same crash against a non-idempotent consumer is a page every time.
If a side-effect is not idempotent, it should not be on the queue. Find a way to make it idempotent first; the queue handling is the easy part.
Polyglot persistence is a per-workload decision
Choosing MongoDB for content, Neo4j for relationships, PostgreSQL for the canonical ledger, and Redis for hot lookups on Multi Aura was not a "use the right tool" reflex. Each store earned its place by being the right shape for one workload. The lesson is to separate the "what shape is this" question from the "what store do we already have" question, and to do that separation per workload, not per system.
The harder corollary is that polyglot persistence is an operational choice, not just a modeling one. A team that runs PostgreSQL well will run MongoDB badly until it learns to run MongoDB; the cost is real and the cost is yours.
Decision-shaped documentation beats feature-shaped documentation
The case study prose I am proudest of is the prose that reads as a series of decisions. "Chose X over Y because Z" not "supports X, Y, Z". The first version teaches the reader how to think about the trade-off; the second version teaches them what code is on disk.
The same lesson applies to code review comments, to design docs, and to post-mortems. Decisions are what age well; feature lists are what date.
The minimum shape
The lessons above are not principles. They are habits that I have picked up from running systems that came back to bite me when I skipped them. Most of them look slower than the alternative, and most of them produce systems that survive the next person who has to operate them. That trade-off is the trade-off I keep choosing.