← All case studies
PDPMagento 2EXP-02Aug 2026

SA tiles & bathroom retailer · Tiles, bathroom & flooring retailTwo tests that only worked on a cold cache

A bathroom-set selector and a 'complete your bathroom' panel both showed the same symptom: visible on the first load, gone on the next page and on Back. The bug was in how the experiments were activated, not in the variants.

ClientSA tiles & bathroom retailer
PlatformMagento 2
PagePDP
Measured onAdd-to-cart, Set chosen, ATC from selector
Observation
  • Both experiments targeted product pages by reading the SKU out of the buy-box form inside a Convert location condition.
  • QA saw the variants on a fresh load and then never again, a classic 'works once' report that usually gets blamed on the variant code.
  • The Convert snippet loads asynchronously in the head, while the buy-box form sits roughly 2,000 lines into a 425 KB document.
Hypothesis

The experiments were not failing, they were never starting. If activation waited for the form instead of racing it, both tests would bucket every eligible visitor.

What I built
  • Traced the timing: with Convert's JavaScript cached for five minutes, the location condition executed before the form was in the DOM, evaluated false, and the experience never activated for that pageview.
  • Moved activation into a shared, status-gated Global JS function: wait for the buy-box form, then the page title, run the product-title regexes, set a window flag and push the activation event.
  • Selector test: two-option control (single item vs full set) above add-to-cart that rewrites the native form's product ID and swaps the headline price, with a stylesheet keyed on an html class to suppress conflicting promos.
  • Panel test: page targeting by title patterns that accept fixtures and reject tiles, flooring and fitting phrases.
  • Re-initialise on bfcache restore so the Back button gets a clean state.
The hard parts

A biased sample nobody could see

Only cold-cache visitors were entering the test. The dashboards showed traffic, just far less than expected, which looked like a targeting problem rather than a timing one.

The tool's retry helper was not a DOM wait

Convert's looped execute call inside a location condition did not reliably re-fire. The rule that came out of this: never use it as a wait, always gate on the element yourself.

Outcome

Measured on add-to-cart, set chosen, atc from selector. Test results belong to the client and are shared in conversation, with their permission, rather than published here.

More experiments

View all →