The operations log (oplog).

Suppose a merchant asks why a payment failed. We load the payment and get status: failed. That answer is correct. It is also almost useless.

The status does not tell us who initiated the payment, whether a worker claimed it, what the provider returned, or whether two attempts raced. We can probably find those facts. The request has a log. The worker has a trace. The provider callback has another log. The database has the final row.

But now the person investigating the payment has to know how our system is assembled before they can explain what the system did. That seems backwards.

What do we mean by an oplog?

Oplog is short for operations log. It is an append-only list of the meaningful operations performed on one resource. In our case, that resource might be an order, payment, payout, refund or financial account.

The word operation is doing some work here. An event is often named after the successful fact we want to announce: payment.succeeded, for example. An operation starts earlier. It can succeed, fail, be rejected, be skipped, do nothing, or remain pending. All of those outcomes matter when the question is “how did this payment get here?”

An entry therefore records the resource, action and outcome. It also records the actor, the source of the work, timing, causality, related resources and a small amount of safe evidence. The entry should make sense without first reading the service that wrote it.

What did we borrow?

None of the underlying ideas is new. The name most obviously recalls MongoDB's oplog: an ordered record of database changes used to replicate state. But the shared name should not hide the difference. MongoDB records successful data modifications for replication and recovery. Our oplog records business operations for investigation, including attempts that were rejected, skipped, failed or changed nothing. Those outcomes are precisely what may explain the resource.

PostgreSQL's write-ahead log contributes a more important lesson than a data model: evidence required for recovery cannot be an optional write after the mutation. We borrow that insistence on shared fate. We do not record physical database changes, and we do not use the oplog to recover the database.

Event sourcing and Temporal's event history show what becomes possible when an ordered history can reconstruct state. That is not the promise here. Inttegro's domain row remains the source of current state; replay is for inspection. AWS CloudTrail is useful prior art for recording the actor, source and action, but it is an account-wide activity record rather than an ordered history of one business resource.

So the claim is not that we invented append-only logs. The useful boundary is narrower: business-semantic operations, ordered per resource, with failure outcomes and redacted evidence, without making current state a projection of the log.

Why not just keep the logs?

The obvious answer is to keep the logs longer and make them easier to search. That only works if an execution log and a resource history are the same thing.

Logs and traces describe an execution. A request arrived, a query took 40 milliseconds, a provider returned an error, a worker retried. They are organised around the request or job that happened to be running at the time.

A commerce resource usually outlives any one execution. A payment may be created in an HTTP request, claimed by a worker, changed by a provider callback and retried by another worker hours later. Keeping the logs longer does not change their unit of organisation. It only gives us more hay.

These are different questions. We should not force one store to answer all of them.

Reading one real order

To see whether this is useful in practice, I opened a recent order in our internal oplog explorer. It belongs to a real customer, so I will not include its identifier, merchant, customer, amount, payment method, provider or timestamps. None of those details is necessary to show the investigation.

The order row told us that its payment had failed. The connected oplogs told us more. The order was created, associated with a customer, given a payment and finalised. A payment method was attached and confirmation was requested. In the payment history, recording that confirmation failed twice before succeeding. A duplicate confirmation request was then recognised as a no-op because confirmation had already been recorded.

Execution was claimed and a payment prompt was issued. The customer did not approve it, so the attempt was cancelled. A later retry issued another prompt; that one was not approved either. This rules out several theories quickly. The payment was not abandoned before execution, the provider handoff happened twice, and a repeated confirmation did not mutate it twice.

The payment failed because the customer did not approve either payment prompt.

The top-level payment entry still carries the generic error payment_attempt_failed. If we stopped there, we would know only that execution failed. Following the connected attempt history gives us the human cause: both prompts went unapproved. That is why the graph matters. The terminal operation says how the system closed the payment; the related attempt explains why.

This is also a design review disguised as an investigation. If an investigator has to follow several edges to learn that the prompts went unapproved, the current presentation is hiding too much. We should surface decisive related evidence beside the terminal operation while preserving which resource supplied it. The history should become more explanatory as we learn which questions it makes unnecessarily difficult.

Ordering, but only where it matters

Each resource gets its own sequence. If a payout has entries 17 and 18, we know 18 followed 17 even if two workers tried to update the payout at the same time.

Do we need one global sequence for every operation in Inttegro? No. It would be expensive, and it would answer a question we rarely ask. During an investigation, we usually need to know whether this payout was selected before it was sent to the provider, or whether this payment failed before the order was cancelled. Resource-local order gets us most of the way there. Causality and related-resource references connect the remaining timelines.

The storage model follows from that choice. One small head item allocates the next sequence. Immutable entry items contain the history. Optional checkpoints make long replays cheaper, but a checkpoint is derived data. If it is wrong, we ignore it and return to the entries.

The write has to be part of the mutation

The obvious implementation is to update the payment and then write its oplog entry. Two writes. Simple enough.

What happens when the first write succeeds and the second fails?

The payment has changed, but its history says it has not. Worse, nobody looking at the timeline knows there is a hole. A history that is occasionally fictional may be less useful than having no history at all.

For required oplog writes, we put the resource mutation, conditional sequence advance, immutable entry and optional deduplication guard in the same DynamoDB transaction. They commit together or not at all. We also require a known actor and source, unless the caller gives a specific reason why either is unavailable.

Not every integration does this yet. Some older order, payment and payout paths save the resource first and append the entry on a best-effort basis. Failing the API request after the resource has already changed would be misleading, so those paths report the oplog failure and continue. They need to move. The distinction matters because the code supports atomic history; that does not mean every caller is already using it.

Why is this practical now?

Keeping a history used to sound like the responsible thing that would also make the system slower and more expensive. Every payment would create more data and require more work before we could respond. Why not keep the final row and move on?

The economics have changed. Oplog entries are small, and storage is cheap. A million one-kilobyte entries amount to roughly one gigabyte before indexes and overhead. Keeping the short explanation of a payment is usually cheaper than asking an engineer to reconstruct it later from logs, traces and provider callbacks.

The extra write does not need to become a long procession of database calls either. DynamoDB is built for single-digit-millisecond reads and writes, and its transactions let us save the resource and its history in one all-or-nothing request. We still measure the real latency—especially the slowest requests—but we do not have to choose between a fast write and a trustworthy history.

Go gives us room to do the surrounding work in the same request. Its lightweight goroutines allow independent work to continue while another operation waits for input or output. Loading context, checking the operation and preparing safe evidence do not all have to stand in a single queue. We can do more before responding without turning every step into more waiting.

Cheap does not mean limitless. The entries still need to be small, and anything unrelated to the truth of the mutation should stay out of the request path. But the oplog itself belongs there. Storage, DynamoDB and Go have made it practical to preserve the explanation at the same moment we preserve the fact.

What should we record?

Once you have a durable investigation record, the tempting answer is “everything”. Store the request body, provider payload, customer details, and the complete object before and after every change. We would certainly have more data.

We would also have built an expensive copy of sensitive data that is difficult to read and even harder to justify.

Each domain instead defines a compact snapshot containing the fields needed to understand its transitions. The oplog applies a common redaction pass to snapshots, evidence, field changes, source metadata and checkpoints. An idempotency key records that one existed, not its value. A deduplication key is hashed before it becomes part of a table key.

Redaction is the second boundary, though. The first is judgement. If a raw provider payload does not belong in the oplog, a sanitizer should not be the reason we put it there.

Replay does not make this event sourcing

The word replay causes another assumption: if entries are append-only and ordered, perhaps we intend to rebuild the production object from them.

We do not. The domain table is the source for the payment's current state. The oplog is the source for the path it took. Those are separate responsibilities.

The replay API returns ordered frames, including the time between operations and the duration of each operation where we know it. An investigation tool can step through the frames, preserve the original timing, or compress long waits. Checkpoints can provide a useful starting state. None of this requires application reads to depend on replay.

Could we derive more state from the oplog later? Probably. But that is not the guarantee we are making today, and making it accidentally would leave us with the obligations of event sourcing without having designed for them.

What does the oplog make possible?

The immediate opportunity is a better support tool. A person should be able to open an order, follow its related payment and refund histories, and answer “what happened?” without knowing which service owns each transition. The explorer we use internally is a start. The useful product is the explanation, not the table of entries.

A reliable history can also catch trouble before somebody asks. We can look for operations that remain pending too long, repeated attempts that end the same way, impossible transitions, missing causal links and resources whose current row disagrees with their last recorded operation. Those checks are much harder when the evidence is scattered across execution logs.

Redacted histories can become regression fixtures. Instead of inventing a clean example, we can take the shape of a real failure, remove identifying evidence and ask whether a change still produces the same transition sequence. Operation timing can show where a lifecycle waits. Causality graphs can show which upstream action created a downstream failure.

The same structure is useful to software agents, with an important limit. An agent can traverse the recorded chain and explain what the entries prove. It should not fill gaps with a plausible story. Machine-readable history is valuable because it gives the agent evidence, not because it gives the agent permission to guess.

Where this leaves us

The test for the oplog is not whether it contains a great deal of information. It is whether someone can explain a surprising resource without already knowing which service, worker and provider touched it.

We are not there for every resource and every mutation. The model does, however, give us a useful place to put the answer: next to the resource, in order, with the boundary between current state and history left intact.

That is enough to make the next failure less archaeological than the last one. For now, that is the point.