A/B testing === 1. Give one group option A, 2nd group option B 1. Track & update distribution for each group 1. Graph mean and range (often ±95th %ile) 1. If bands pull apart, choose higher one 1. No decision until there's some separation between bands (may never) 1. Consider A/A or A/A/B test, see to if A groups appear to differ * Shows problems with group size or group consistency Probabilities ==== 1. Example: shapes: red & blue, square & triangle 1. 1 red square 1. 2 red triangle 1. 3 blue square 1. 4 blue triangle 1. Probability of an event ∈ [0-1] 1. P(red) = 3/10 = 0.3 1. P(blue) = 7/10 = 0.7 1. P(square) = 4/10 = 0.4 1. P(triangle) = 6/10 = 0.6 1. Joint probability, both occur 1. P(red square) = 1/10 = 0.1 1. P(red triangle) = 2/10 = 0.2 1. P(blue square) = 3/10 = 0.3 1. P(blue triangle) = 4/10 = 0.4 1. Probabilities of either A **or** B happening add 1. P(red) = P(red square) + P(red triangle) = 0.1 + 0.2 = 0.3 1. P(square) = P(red square) + P(blue square) = 0.1 + 0.3 = 0.4 1. Conditional probability 1. P(red | square) = probability that it's red, if we already know it's a square = 1/4 = 0.25 1. Joint probability from conditional 1. P(A B) = P(A|B) * P(B) 1. Also P(A B) = P(B|A) * P(A) 1. P(red square) = P(red | square) * P(square) = 1/4 * 4/10 = 1/10 = 0.1 1. Can use this to compute conditional: 1. P(A|B) = P(A B)/P(B) 1. P(red | square) = P(red square) / P(square) = 0.1 / 0.4 = 0.25 1. P(square | red) = P(red square) / P(red) = 0.1 / 0.3 = 0.333 1. Probability distribution / random variable 1. Function (w/ parameters) that models probability 1. Binomial distribution: fixed probability of an A/B choice 1. Gaussian distribution: typical bell curve w/ mean & variance 1. Poisson distribution: chance of things happening over time 1. Probability density: p(x) 1. Given a bunch of samples, what's the density of samples near x 1. Density ≠ probability; has to be positive, but can be > 1 (high density) 1. Gaussian bell curve is the density for the Gaussian distribution Bayes / Bayesian statistics ==== 1. Have two equations to compute joint probability from conditional * P(A B) = P(A|B) * P(B) = P(B|A) * P(A) 1. Solving for one of the conditional probabilities gives Bayes rule * P(A|B) = P(B|A) * P(A) / P(B) 1. Interpretation: A is our model, B is some observed data: 2. P(B|A) is the probability of the observed data given our model 3. P(A) is the believed probability of the model 1. Also known as the *prior* 2. We think there's a 10% chance a user will make a purchase in a week 4. P(B) is the probability of the observed data independent of the model * Normalizing factor, in many situations can ignore this 1. P(A|B) is the probability of the model given the new data (aka *posterior*) 1. Observe some data (B) and want to update belief 2. This week, 10k people purchased, and 50k didn't 1. Conjugate Prior 1. Want posterior to be same kind of distribution as prior after update 1. If B is binomial distribution (choice of A or B, or success or failure) 1. P(A) and P(A|B) can both be a Beta distribution 1. Beta is *conjugate prior* for binomial; like extension of bionomial 1. Binomial has one parameter, chance of yes 1. Beta has two parameters, $\alpha$ and $\beta$ 1. Mean of Beta is $\alpha/(\alpha+\beta)$ 1. Variance: $\alpha\beta/((\alpha+\beta)^2(\alpha+\beta+1))$ 1. Uninformed prior: $\alpha=1; \beta=1$ 1. Update: add successes to $\alpha$ and failures to $\beta$ 1. Normal distribution is its own conjugate prior, though updates are more complicated 1. Conjugate prior for Poisson is Gamma distribution. Resources === 1. [Beta-Binomial A-B testing](https://www.gdcvault.com/play/1020201/A-B-Testing-for-Game) * Game-specific with nice dashboards 2. [A-B Testing Overview](https://towardsdatascience.com/why-you-should-switch-to-bayesian-a-b-testing-364557e0af1a) 3. [Binomial & Gaussian](https://towardsdatascience.com/how-to-do-bayesian-a-b-testing-fast-41ee00d55be8) * Has both equations and sample code 4. [Bayesian modeling for Poisson](https://www.y1zhou.com/series/bayesian-stat/bayesian-stat-bayesian-inference-poisson/) 5. [Game live ops](https://www.devtodev.com/education/articles/en/351/a-b-testing-in-liveops)