As given in the specification, we are given a confidence interval on the expected percentages of specific behaviors (like we accept the accumulated number of times for fish turning around from 50 to 150 to prove a 10% chance). Sometimes when we run the tests on those percentages, the accumulated number we got might falls out of the confidence interval (then in such case, p-value would be smaller than 5% indicating this result as an outlier in the model). May I ask how would our unit test results be assessed? Would our unit test be run for several times to prove that it works most of the time, since this “outlier” scenario is possible to happen?
This is more than a 5% confidence interval, it’s a 99.99997% confidence interval. If you’ve used R before, see:
> pbinom(150, 1000, .10) - pbinom(50, 1000, .10)  0.9999997
(Because I like probability, I can even explain in STA347 terms:)
- Each time you are checking if the bubble should spawn, you’re really making a trial of a Bernoulli(p=0.1) random variable
- The sum of these random variables (by definition) forms a Binomial(n = 1000, p = 0.1) random variable (if $$Z_1, Z_2, …, Z_n$$ have $$Z_i ~ Bernoulli(p )$$, then $$Z_1 + Z_2 + \dots + Z_n ~ Binomial(n, p)$$
- pbinom(x, n, p) yields $$P(X \leq x)$$ where $$X ~ Binomial(n, p)$$, so $$pbinom(150, 1000, .10) - pbinom(50, 1000, .10)$$ is really $$P(50 < x <= 150)$$
Right! Sorry that’s my fault on calculation.
Thank you for reminding
Then may I ask will our test be run for more than one time? I know it’s hard to get the out-of-interval result, but this truly happened once on my computer…(I promise I didn’t change any code of the test!)
Java doesn’t actually use “true randomness”, so I guess it’s possible that the random number generator really didn’t like you – you can make your test deterministic by making the random you use have a “seed” value (e.g.,
(new Random(207)).nextDouble() will always return the same thing) or just increasing the number of iterations to over 1000.
Looks good! Thank you