A contextual bandit learns which action to choose while deliberately trying alternatives. It can help with frequent decisions, a manageable action space and reliably observed rewards. It is an advanced, optional extension of this project.
Before considering one, show:
- A feasible action set, including no offer, and a bounded or suitably modelled reward tied to net contribution, not clicks alone.
- An exploration budget, a durable randomised holdout, and the exact probability of the selected assignment, recorded after eligibility and budget rules and kept if execution later fails.
- Correct delayed-reward joins, complete follow-up for mature zeros and returns, and a policy for late or missing feedback. “Not received yet” is different from zero reward.
- Offline support and uncertainty diagnostics, plus a valid method for evaluating adaptive logs.
- A kill switch and deterministic fallback with working cost, consent and fatigue controls.
For changing action sets, Vowpal Wabbit's contextual-bandit tutorial covers action-dependent features and action/cost/probability logging. Exploration methods such as epsilon-greedy or Thompson sampling must stay within feasible actions. The right one depends on the reward model, traffic and acceptable cost.
A bandit sees only the chosen action's reward. Past deterministic assignments give no overlap for the alternatives. A fixed reward horizon matters too: a bandit optimised for clicks can learn to buy clicks while losing money. With long delays, changing price or stock, or large effects on future customer state, a fixed randomised experiment with periodic policy refreshes may be easier to trust. If you move to reinforcement learning for long sequences, justify the state, reward, transition evidence and evaluation design separately.