Measurement =========== 1. Playtest * Observe, tweak, repeat 1. Metrics 1. Log events (death, progress, ...) 2. Graph (Examples from Forza, Paragon) 1. Forza: Talk from Game Development in Computer Science Education 2008 1. Players not meeting time mark in qualifying races 2. Time improves for 3-5 races, then starts to get worse 3. Set to allow X% of players qualify after 3 races 2. Paragon: track hero win/loss rates, buff worst & nerf best next release 3. [Build heat map](https://www.gamedeveloper.com/design/hot-failure-tuning-gameplay-with-simple-player-metrics) 1. Log events with location 2. Splat a blob sprite for each event 3. Greyscale to hue 4. Examples 1. [Heat map 1](https://eu-images.contentstack.com/v3/assets/blt740a130ae3c5d529/blte2b651864620c35b/650e93b3b9cdf62dc289a118/heatmap_580.jpg) 2. [Heat map 2](https://eu-images.contentstack.com/v3/assets/blt740a130ae3c5d529/blt075ced4e53d262d3/650e93c2abaf87661b5b76c6/Memory035_580.jpg) 3. [Heat map 3](https://eu-images.contentstack.com/v3/assets/blt740a130ae3c5d529/blte30c106e2971d59a/650e93b9b87c6272c9092a93/Memory034_580.jpg) Tuning a Game ============= 1. Gameplay 1. Power & Stamina 1. Player damage, hit probabilities 2. Enemy damage, hit probabilities 3. Player / Enemy health 4. Danger of ripple effect (see spreadsheet tuning) 2. Time limit 1. Racing 2. For expertise / challenge (e.g. Portal end-game) 3. Map design 1. Cover / safe zones (permanent or able to be destroyed) 2. Ease of traversal (e.g. jump timing) 3. Visibility of player or enemies 4. Enemy placement 4. AI Aggression 1. Sensitivity to alert 2. Chance to attack 5. Enemy count * Avoids ripple effect of stat changes 6. Resources (things you can run out of) 1. FPS: Ammo, Health; RTS: Ore, Food 2. Availability (e.g. pick-up placement before/after battle) 3. Production: more with time (FPS: healing; RTS: factories, farms) 4. Use: how long does it last (FPS: rate of fire; RTS: consumption rate) 7. Checkpoints 1. Anytime or at set points 2. How much to replay? 1. Too close to battle & no restore could lock player into low health/low ammo losing situation 2. Too close to battle & full restore could make dieing too beneficial 3. Further back allows recovery & normalization of resources to designed levels, but frustrating replay. 4. Immediately **after** tough section is often a good choice to save success 8. Giving an advantage 1. Player assist 1. Aiming 2. Map & info on map (enemy / resource locations?) 3. HUD (flash showing direction) 4. Sound 5. Input precision / Hit precision [(e.g. Canabalt)](https://www.gamedeveloper.com/design/tuning-canabalt) 1. [Character hit box](https://i0.wp.com/eightface.com/files/images/2010.10.tuning.canabalt.runner.png?w=1200): less in front, hangs over in back 2. Obstacle hit box 2 pixels high for easy jumping 3. Some gap sizes based on player velocity 2. Enemy assist 1. Boost or reduce to match player level 2. Know where player is even when not in line of sight 2. Monetization 1. What to buy 2. How much things cost 3. Rewards Tuning with a Spreadsheet ========================= 1. Interrelationship between health and weapon stats 1. Damage per turn or survivability 2. Expected player level & modifiers 3. Weapon availability 4. Monster stats 5. Tune to near 0 2. [Power progression (especially for RPG)](https://www.gamedeveloper.com/design/the-craft-of-game-systems-tuning-rpg-content) 1. Set base power 2. Magnifiers (items, skill) * Set as percent of base 3. Solve for magnifiers based on preferred result 3. [RTS production](http://www.somasim.com/blog/2015/04/how-to-tune-a-simulation-game/) 1. Graph resource chain from source to sink * Mine consumes pickaxes, produces ore; smelter consumes ore, produces iron; blacksmith consumes iron, produces ... 2. Tune to near 0 / steady state 3. [Spreadsheet of costs](https://somasim.com/wp-content/uploads/2014/12/tuning1.png) 4. [Spreadsheet tuning city](https://somasim.com/wp-content/uploads/2014/12/tuning2.png) Tuning with Equations ===================== 1. Chance to hit, expected damage per time or per turn, expected rounds to death 2. Usually don't want to show to player * Player expects to win **every** 60% match, not lose two out of every five. 3. Probabilities 1. Probability of an event P(X) is a percentage (or fraction in [0,1]) 1. Example: 1 red circle, 2 red triangles, 3 blue circles, 4 blue triangles 2. P(red) = 3/10 = 0.3 2. Joint probability, both occur: P(red triangle) = 2/10 = 0.2 1. Probabilities of A or B happening add 2. P(red) = P(red circle) + P(red triangle) = 0.1 + 0.2 = 0.3 3. Conditional probability P(A|B) = probability of A if we already know B is true 1. P(red | circle) = 1 (red circle) / 4 (circles) = 0.25 2. In general $P(A|B) = P(A B) / P(B)$, or $P(A B) = P(A|B)*P(B)$ 4. Expected value 1. Have a set of hit probabilities and amounts of damage, what's the average damage? 2. damage(A) * P(A) + damage(B) * P(B) + damage(C) * P(C) ... 3. D&D rules: 0 * 1/20 (miss) + normalHit * 18/20 + criticalHit * 1/20 5. Probability density p(X) 1. If I choose a bunch of random values, how dense will they be near X? 2. Has to be positive, but can be greater than 1 (density ≠ probability) 3. Classic "bell curve" is a probability density, more likely in the middle, less likely at the edges. 6. Probability distribution 1. Uniform = equally likely: flip a coin, roll a die 2. Binomial = flip a coin several times, can be *unfair*, so heads & tails not equal * Good for a choice: does player take an action, make a purchase, etc. 3. Gaussian/Normal = classic bell curve, more likely near mean * Good for something that takes a specific value (expected ARPU) 4. Poisson = chance of (rare) things happening over time * Number purchases per day, chance of a player having a random encounter in a fixed amount of time, time until a loot box pays out 4. Using to tune 1. Target rounds to battle 2. Target amount of damage player/enemy will receive during battle 3. Target loot or health pick-up probabilities A-B Testing =========== 1. Bayesian Statistics 1. Have *prior* distribution P(A) = your belief on the probability of A 2. Observe actions from a group of players P(B|A) 3. Update to get *posterior* distribution P(A|B) 4. Use posterior as your new prior = update what you believe in the face of new data 2. Bayesian Updates 1. Generally want prior & posterior to have the same distribution, but observation can be different 2. If B has a binomial distribution, P(A) and P(A|B) will have *Beta* distribution 1. Will say Beta is the *conjugate prior* for Binomial, or that you're using the *Beta-Binomial* model 2. Beta has two parameters: α & β. 3. Update: add #successes to α & number of failures to β 4. Simple equations for mean and variance 3. Conjugate prior for Normal/Gaussian distribution is Normal/Gaussian * Say you're using the *Normal-Normal* model 4. Conjugate prior for Poisson is Gamma: *Gamma-Poisson* 3. Using 1. Give one group option A, 2nd group option B 2. Track & update distribution for each group 3. Graph mean and *credibility interval* [(Example from Swrve GDC talk)](swrve-ab.jpg) 4. If bands pull apart, choose higher one 5. No decision until there's some separation between bands (may never) 6. Consider A/A or 3-group A/A/B test, see to if A groups appear to differ * Shows problems with group size or group consistency Other Resources =============== 1. [Difficulty levels](https://www.gamedeveloper.com/design/difficulty-is-difficult-designing-for-hard-modes-in-games) 1. Time limits 2. Resource availability 3. Enemy count 4. AI aggression 5. Damage dynamics 2. [Input, response, context](https://www.gamedeveloper.com/design/game-feel-the-secret-ingredient) 1. Platformer example: 1. 20h complete game 2. 1-2h boss fight 3. 30m access new area or ability 4. 5m complete minor objective 5. Jump success, acrobatic maneuvers 2. Input system 1. Control type 1. L/R or analog 2. Control in air 2. Effect of context 1. Double/triple jump, wall jump 2. Feedback 3. [Beta-Binomial A-B testing](https://www.gdcvault.com/play/1020201/A-B-Testing-for-Game) * Game-specific with nice dashboards 4. [A-B Testing Overview](https://towardsdatascience.com/why-you-should-switch-to-bayesian-a-b-testing-364557e0af1a) 5. [Binomial & Gaussian](https://towardsdatascience.com/how-to-do-bayesian-a-b-testing-fast-41ee00d55be8) * Has both equations and sample code 6. [Bayesian modeling for Poisson](https://www.y1zhou.com/series/bayesian-stat/bayesian-stat-bayesian-inference-poisson/) 7. [Game live ops](https://www.devtodev.com/education/articles/en/351/a-b-testing-in-liveops)