Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
150 changes: 93 additions & 57 deletions docs/ab-testing.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,46 +6,46 @@ render the branch they were given — no flicker, no client-side flip.

This page is for two audiences:

- **Product / experiment owners** — start with _"What you can and can't do in Phase 1"_.
- **Developers** — the rest walks through defining an experiment, reading the variant,
and how the pieces fit.
- **Product / experiment owners** — start with _"What you can and can't do in Phase 1"_.
- **Developers** — the rest walks through defining an experiment, reading the variant,
and how the pieces fit.

---

## What you can and can't do in Phase 1

**You can:**

- **Run an A/B or multi-variant test** (any number of variants, each with a traffic
weight) that spans one page or several microfrontends on that page — every app in the
request sees the **same** assignment, so a test can cover a whole funnel step.
- **Split traffic by percentage** (e.g. 50/50, or 34/33/33). Assignment is **random but
stable**: a given visitor is put in a bucket once and **keeps that variant** for the
life of the experiment (up to a year), across reloads and navigation.
- **Turn everything off instantly** with a global kill-switch (no deploy) — e.g. during
an incident. Everyone reverts to the baseline.
- **Trust it not to break the site** — if anything about the experiment layer goes wrong,
the visitor simply sees the baseline; the page still renders.
- **Add / pause / reweight experiments via configuration** (see below) rather than a code
change to the assignment logic.
- **Run an A/B or multi-variant test** (any number of variants, each with a traffic
weight) that spans one page or several microfrontends on that page — every app in the
request sees the **same** assignment, so a test can cover a whole funnel step.
- **Split traffic by percentage** (e.g. 50/50, or 34/33/33). Assignment is **random but
stable**: a given visitor is put in a bucket once and **keeps that variant** for the
life of the experiment (up to a year), across reloads and navigation.
- **Turn everything off instantly** with a global kill-switch (no deploy) — e.g. during
an incident. Everyone reverts to the baseline.
- **Trust it not to break the site** — if anything about the experiment layer goes wrong,
the visitor simply sees the baseline; the page still renders.
- **Add / pause / reweight experiments via configuration** (see below) rather than a code
change to the assignment logic.

**You can't (yet) — deferred to later phases:**

- **Reach visitors who don't reload.** A newly launched or changed experiment reaches a
visitor only on their **next full page load**. Long-lived single-page sessions won't
pick it up until they reload — there's no live push (polling/streaming) yet. Read
results knowing only reloading visitors were exposed.
- **Target an audience.** Assignment is a random split by visitor. There's no targeting
by geography, campaign/UTM, new-vs-returning, logged-in status, or user attributes.
- **Self-serve from a UI.** Experiments are defined in configuration by an engineer;
there is no management dashboard.
- **Coordinate overlapping experiments.** No mutual-exclusion / conflict rules — if two
experiments touch the same surface, that's on the authors to avoid.
- **Get analytics out of the box.** ILC assigns and delivers the variant; **measuring**
it (exposure/conversion events, dashboards) is the consuming app's/BI pipeline's job.
- **Assume consent handling.** Experiments are not gated on user consent unless a
deployment wires that up (see _Consent_). Treat consent as a prerequisite before any
production rollout in a regulated market.
- **Reach visitors who don't reload.** A newly launched or changed experiment reaches a
visitor only on their **next full page load**. Long-lived single-page sessions won't
pick it up until they reload — there's no live push (polling/streaming) yet. Read
results knowing only reloading visitors were exposed.
- **Target an audience.** Assignment is a random split by visitor. There's no targeting
by geography, campaign/UTM, new-vs-returning, logged-in status, or user attributes.
- **Self-serve from a UI.** Experiments are defined in configuration by an engineer;
there is no management dashboard.
- **Coordinate overlapping experiments.** No mutual-exclusion / conflict rules — if two
experiments touch the same surface, that's on the authors to avoid.
- **Get analytics out of the box.** ILC assigns and delivers the variant; **measuring**
it (exposure/conversion events, dashboards) is the consuming app's/BI pipeline's job.
- **Assume consent handling.** Experiments are not gated on user consent unless a
deployment wires that up (see _Consent_). Treat consent as a prerequisite before any
production rollout in a regulated market.

---

Expand All @@ -64,12 +64,12 @@ server router
ILC's `onRequest` hook is the single server-side point shared by every fragment, which
makes it the natural place to resolve a variant:

- **No flicker** — the variant is fixed before the first byte of HTML, so SSR'd fragments
render the correct branch on first paint.
- **Cross-fragment consistency** — every fragment in the request gets the same map, so one
experiment can span multiple microfrontends.
- **Fail-safe** — assignment is in-memory, never blocks the request, and on any error the
visitor falls back to the baseline.
- **No flicker** — the variant is fixed before the first byte of HTML, so SSR'd fragments
render the correct branch on first paint.
- **Cross-fragment consistency** — every fragment in the request gets the same map, so one
experiment can span multiple microfrontends.
- **Fail-safe** — assignment is in-memory, never blocks the request, and on any error the
visitor falls back to the baseline.

An app reads its variant from `appProps.experiments`. Because the value is resolved once
on the server and forwarded to the fragment for both its SSR render and its client
Expand Down Expand Up @@ -100,18 +100,18 @@ experiments: {

Per-experiment knobs:

- **`variants`** — any number of named variants (not limited to two). `weight` is the
percentage of traffic; weights should sum to 100. The **first** variant is the
baseline / fallback.
- **`status`** — `active` assigns variants; `paused` sends everyone to the baseline
without deleting the definition.
- **`consentCategory`** _(optional)_ — gate the experiment behind consent (see _Consent_).
- **`enrollment`** _(optional)_ — gate **first-time enrollment** by request path:
`enrollment: { paths: ['/sample-nodejs'] }` recruits new participants only on the
listed path prefixes (segment-aligned: `/shop` covers `/shop/cart`, not `/shopping`;
`/` is root-exact — homepage only, since "everywhere" is expressed by omitting the
field; matched against the raw request path, query excluded). Visitors who never hit
an enrollment path get **nothing** — no assignment, no `x-ab-*` cookie, no `ilc-sid`.
- **`variants`** — any number of named variants (not limited to two). `weight` is the
percentage of traffic; weights should sum to 100. The **first** variant is the
baseline / fallback.
- **`status`** — `active` assigns variants; `paused` sends everyone to the baseline
without deleting the definition.
- **`consentCategory`** _(optional)_ — gate the experiment behind consent (see _Consent_).
- **`enrollment`** _(optional)_ — gate **first-time enrollment** by request path:
`enrollment: { paths: ['/sample-nodejs'] }` recruits new participants only on the
listed path prefixes (segment-aligned: `/shop` covers `/shop/cart`, not `/shopping`;
`/` is root-exact — homepage only, since "everywhere" is expressed by omitting the
field; matched against the raw request path, query excluded). Visitors who never hit
an enrollment path get **nothing** — no assignment, no `x-ab-*` cookie, no `ilc-sid`.

This is **not a route-scoped experiment**: once a visitor is enrolled, the stored
assignment is honoured on every route — participation never toggles with navigation.
Expand Down Expand Up @@ -165,6 +165,42 @@ application library, which resolves the variant, tells you whether it's the base
defaults safely when the experiment is absent (kill-switch off, paused, or not yet
delivered). The value is a string the app branches on; ILC never renders app UI itself.

### Apps that don't receive `appProps`

`appProps.experiments` reaches route apps only. Code that ILC does not mount for a route
has no `appProps` to read:

- **Embedded apps**: bundles another app loads with `window.ILC.loadApp()` and mounts itself;
- **parcels**: mounted through `importParcelFromApp()` / `mountRootParcel()`;
- **wrapper apps** on client-side mounts;
- any other script on the page.

These read the same assignments from the page:

```js
const experiments = window.ILC.getExperiments(); // { 'homepage-hero': 'variant-b' } or {}
```

`getExperiments()` returns the **page assignments**, the map the server resolved for this
document. It is the same object ILC merges into route apps' `appProps.experiments`, so an
embedded app, a parcel and a route app on one page always see the same variant, whichever
app mounted them. The contract:

- **A snapshot, not a live value.** The map is fixed when the page loads and changes only
on the next full page load; see _No live delivery to open sessions_ below. There is no
change event.
- **Frozen, and copied per call.** Every call returns a new frozen object, so one consumer
cannot change what another reads.
- **Always an object.** With no active experiments (or the kill-switch off) it returns
`{}`. Treat a missing experiment as the baseline, as with `appProps`.
- **Read it after ILC starts.** `window.ILC` is populated when the ILC client boots. Apps
loaded through `loadApp()` always start later, so they can read it during bootstrap.
Other page scripts should read it lazily, not at parse time, and feature-detect it
(`window.ILC?.getExperiments?.() ?? {}`) to work with older ILC versions.

Measuring an experiment rendered this way is still the app's job: emit exposure only for
experiments present in the map, otherwise unassigned visitors are counted as the baseline.

---

## A/A tests
Expand Down Expand Up @@ -247,15 +283,15 @@ that while a gated experiment runs.

## Limitations & roadmap

- **No live delivery to open sessions.** New/changed experiments reach a session only on
its next full page load; a push mechanism (SSE/polling) is future work.
- **No audience targeting, no mutual-exclusion, no management UI** — see Phase 1 scope above.
- **Analytics is the app's responsibility.** ILC assigns and propagates; emitting
exposure/conversion events (and any queue/retry) belongs to the consuming app.
- **No per-variant edge caching.** Experiment responses are `no-store`; a variant-aware
cache key would be needed before edge-caching them.
- **Crawlers are assigned like any visitor.** A deployment that cares should bypass
assignment for bots.
- **No live delivery to open sessions.** New/changed experiments reach a session only on
its next full page load; a push mechanism (SSE/polling) is future work.
- **No audience targeting, no mutual-exclusion, no management UI** — see Phase 1 scope above.
- **Analytics is the app's responsibility.** ILC assigns and propagates; emitting
exposure/conversion events (and any queue/retry) belongs to the consuming app.
- **No per-variant edge caching.** Experiment responses are `no-store`; a variant-aware
cache key would be needed before edge-caching them.
- **Crawlers are assigned like any visitor.** A deployment that cares should bypass
assignment for bots.

---

Expand Down
8 changes: 8 additions & 0 deletions ilc/client/Client.js
Original file line number Diff line number Diff line change
Expand Up @@ -67,6 +67,8 @@ export class Client {

#router;

#ilcState;

#transitionHooksExecutor;

#urlProcessor;
Expand Down Expand Up @@ -129,6 +131,7 @@ export class Client {
}

const ilcState = initIlcState();
this.#ilcState = ilcState;
this.#router = new Router(
this.#configRoot,
ilcState,
Expand Down Expand Up @@ -397,6 +400,11 @@ export class Client {
mountRootParcel: singleSpa.mountRootParcel.bind(singleSpa),
importParcelFromApp: parcelApi.importParcelFromApp.bind(this),
getIntlAdapter: () => (this.#i18n ? this.#i18n.getAdapter() : null),
// Page assignments: the experiment map the server resolved for this document — the
// same object ClientRouter merges into route apps' appProps.experiments. A frozen
// snapshot: it never changes until the next full page load, and a fresh copy is
// returned on every call so a consumer cannot mutate what other apps read.
getExperiments: () => Object.freeze({ ...(this.#ilcState.experiments ?? {}) }),
getAllSharedLibNames: () => Promise.resolve(Object.keys(this.#configRoot.getConfig().sharedLibs)),
getSharedLibConfigByName: (name) => {
return Promise.resolve(this.#configRoot.getConfigForSharedLibsByName(name));
Expand Down
87 changes: 87 additions & 0 deletions ilc/client/Client.spec.js
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,9 @@ import sinon from 'sinon';
import { BundleLoader } from './BundleLoader';
import { Client } from './Client';
import Router from './ClientRouter';
import { getIlcConfigRoot } from './configuration/getIlcConfigRoot';
import ilcEvents from './constants/ilcEvents';
import initIlcState from './initIlcState';
import singleSpaEvents from './constants/singleSpaEvents';
import ErrorHandlerManager from './ErrorHandlerManager/ErrorHandlerManager';
import * as navigationEvents from './navigationEvents/setupEvents';
Expand Down Expand Up @@ -206,6 +208,87 @@ describe('Client', () => {
});
});

describe('getExperiments', () => {
const experiments = { 'homepage-hero': 'variant-b', 'example-experiment': 'variant-a' };

const appendIlcState = (state) => {
const script = document.createElement('script');
script.type = 'ilc-state';
script.innerHTML = JSON.stringify(state);
document.body.appendChild(script);
};

const rebuildClientWithIlcState = (state) => {
client.destroy();
appendIlcState(state);
client = new Client(mockConfigRoot);
};

it('should return the experiments inlined into ilc-state', () => {
rebuildClientWithIlcState({ experiments });

expect(window.ILC.getExperiments()).to.eql(experiments);
});

it('should equal the appProps.experiments ClientRouter builds from the same ilc-state', async () => {
rebuildClientWithIlcState({ experiments });

// A real router fed the same serialized ilc-state, parsed the way the client parses it.
const configRoot = getIlcConfigRoot();
const registryStub = sinon.stub(configRoot, 'registryConfiguration').value({
apps: { '@portal/hero': { spaBundle: 'https://somewhere.com/hero.js', kind: 'primary' } },
routes: [
{
routeId: 'all',
route: '*',
next: false,
template: 'commonTemplate',
slots: { hero: { appName: '@portal/hero', props: {}, kind: 'primary' } },
},
],
specialRoutes: {
404: { routeId: 404, route: '/404', next: false, template: 'errorsTemplate', slots: {} },
},
});
appendIlcState({ experiments });
// Stubbed single-spa, as in ClientRouter.spec: the real one would navigate the shared karma page.
const singleSpaStub = { navigateToUrl: () => {}, triggerAppChange: () => {}, getMountedApps: () => [] };
const router = new Router(configRoot, initIlcState(), undefined, singleSpaStub, () => {});

try {
expect(window.ILC.getExperiments()).to.eql(
router.getCurrentRouteProps('@portal/hero', 'hero').appProps.experiments,
);
} finally {
router.removeEventListeners();
registryStub.restore();
// Let pending single-spa events settle before the next spec, as ClientRouter.spec does.
await new Promise((resolve) => setTimeout(resolve, 0));
}
});

it('should return a frozen empty object when ilc-state carries no experiments', () => {
const result = window.ILC.getExperiments();

expect(result).to.eql({});
expect(Object.isFrozen(result)).to.be.true;
});

it('should return a frozen copy that a consumer cannot use to change what others read', () => {
rebuildClientWithIlcState({ experiments });

const first = window.ILC.getExperiments();
expect(Object.isFrozen(first)).to.be.true;
expect(() => {
first['homepage-hero'] = 'tampered';
}).to.throw(TypeError);

const second = window.ILC.getExperiments();
expect(second).to.not.equal(first);
expect(second).to.eql(experiments);
});
});

describe('getAllSharedLibNames', () => {
it('should return all shared library names', async () => {
const names = await window.ILC.getAllSharedLibNames();
Expand Down Expand Up @@ -429,6 +512,10 @@ describe('Client', () => {
expect(window.ILC.importParcelFromApp).to.be.a('function');
});

it('should expose getExperiments method', () => {
expect(window.ILC.getExperiments).to.be.a('function');
});

it('should expose getAppSdkAdapter method', () => {
expect(window.ILC.getAppSdkAdapter).to.be.a('function');
});
Expand Down
Loading