Measurement

You cannot optimise what you cannot attribute

Every adaptive decisioning system depends on knowing what worked. Get attribution wrong and the optimiser learns confidently from a biased signal.

Dr Arshan Iqbal · 7 minute read

Every adaptive decisioning system rests on the same assumption: that you can tell what worked. Remove that and the whole apparatus collapses. The optimiser has nothing to learn from, the business case has nothing to prove, and the strategy reverts to whatever the loudest person in the room believes.

Attribution is treated as a reporting concern. It is actually an architectural one, and it needs to be designed before the first campaign runs rather than reconstructed afterwards.

The problem with the obvious approach

The default is last-click. Someone opened an email, they later transacted, credit the email. It is simple and it is available in every marketing tool.

It is also wrong in both directions. It over-credits the last touch in a sequence that may have involved several. It ignores conversions that happened through a channel you cannot see, which in many businesses is most of them. And it silently treats an unattributable conversion as a non-conversion, which biases the entire learning signal toward whatever happens to be measurable rather than whatever actually works.

For a bandit-driven system this is not a cosmetic problem. The reward signal is the input to the posterior update. Systematically biased attribution produces systematically biased beliefs, and the optimiser will converge confidently on the wrong answer.

A system that learns from a biased reward signal will converge, confidently, on the wrong answer.

Confidence tiers rather than a binary answer

On a revenue decisioning platform I built for accommodation operators, I took a different approach. Rather than forcing every conversion into attributed or not attributed, the engine assigns each one to a five-tier confidence model.

At the top sit conversions with a direct causal trail: a tracked click followed by a transaction within a defined window, on the same identity. At the bottom sit conversions that are plausibly related but unprovable: a transaction from someone who received a campaign, outside the attribution window, with no engagement recorded.

The tiers between them capture the realistic middle: engagement recorded but no click, click recorded but a different booking channel, timing consistent but identity match imperfect.

Three things follow from this design.

Revenue reporting becomes honest. Instead of a single attributed figure that everyone privately distrusts, the operator sees a confidence-weighted number with the composition visible. High-confidence revenue is stated as such. Speculative revenue is not laundered into the headline.

The learning signal is weighted, not binary. A tier-one conversion updates the optimiser's beliefs more strongly than a tier-four one. The system learns fastest from the evidence that is actually reliable, rather than treating all outcomes as equally informative.

Disputes have somewhere to go. When a client questions a number, the answer is not "the system says so". It is a tier, a rule, and a timestamped record.

Bookings as the single source of truth

A related architectural decision, and one I would repeat on any similar build: the transaction table is the source of truth for customer identity, not the customer table.

This sounds like a technicality. It is not. In most operational systems the customer record is edited, merged, deduplicated and corrected over time, which means it does not reliably represent what was true at the moment a decision was made. The transaction record does not move. Anchoring identity to it means the attribution logic can always reconstruct the state of the world at decision time, months later, without depending on a record that has since been rewritten.

The same principle drove another decision: the payload used to make each outbound decision is retained permanently as an audit trail, alongside a flag confirming the exact content that was sent. Storage is cheap. Being unable to answer "what did the system see, and what did it send" is expensive.

What it produced

On the pilot deployment, running against a database of 684 past guests, the platform produced £12,087 in attributed revenue across 20 direct bookings, with a 41.6 per cent email open rate.

The number I care about in that sentence is not the revenue. It is the word attributed. That figure survived a client conversation, because every booking in it could be traced to a tier, a rule and a record. A larger number that nobody trusted would have been worth less.

Where this applies beyond marketing

The same reasoning governs any decisioning system where outcomes are delayed or partially observable.

In collections, did the payment arrive because of the contact, or would it have arrived anyway? In credit, did the account perform because the policy was correct, or because the economic cycle was favourable? In fraud, was the case genuinely prevented, or was it never going to happen?

None of those questions have clean answers. But a system that acknowledges the uncertainty and grades it will outperform one that pretends the uncertainty is not there, because it will learn from better evidence and it will survive scrutiny.