I'm trying to edit a bayesian python code for $A/B$ test analysis.

I'm using uninformative priors as a beta distribution, so my $\alpha$ & $\beta$ parameters are $&$ for both control & test for the first observation of the data.

My question is, Should I use cumulative visitors & conversions for each day as my likelihood or visitors & conversions for each day separately since I'm updating the previous days' data in priors?

So my doubt is am I updating previous information in both my likelihood & prior?

If the goal is to obtain $Pr(\theta | Y_1, Y_2, \ldots, Y_n)$ where the $Y$s are the cumulative data, then a simply application of Bayes Rule shows that: $$Pr(\theta | Y_1, Y_2, \ldots, Y_n) = \frac{Pr( Y_1, Y_2, \ldots, Y_n | \theta) Pr(\theta)}{Pr(Y_1, Y_2, \ldots, Y_n)}$$ And under the assumption of independence of the $Y$s (being consecutive observations across time): $$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2) \cdot \ldots \cdot Pr(Y_n)$$ and also $$Pr(Y_1, Y_2, \ldots, Y_n | \theta) = Pr(Y_1| \theta) \cdot Pr( Y_2| \theta) \cdot \ldots \cdot Pr(Y_n| \theta)$$ So you see that yes you can run the analyses sequentially if the time series is independent.

Alternately, if there is a time series autocorrelated effect, you can rely on the general factorization of the likelihood $$Pr(Y_1, Y_2, \ldots, Y_n) = Pr(Y_1) \cdot Pr( Y_2 | Y_1) \cdot \ldots \cdot Pr(Y_n | Y_{n-1} \ldots, Y_2, Y_1)$$ So by simply modeling the autocorrelation using some form of ARIMA or something, you can run a sequential analysis.

To rephrase what i'm looking for - My confusion is do I need to update my previous days' data both in the likelihood and in the prior?

I'm using informative priors as previous days' data from the AB test result.

But i'm also using the likelihood as the cumulative conversion data.

@rettib it depends on the question you're trying to answer. Using the informative prior gives you conditional inference on theta, E.g.

Given the evidence suggests the odds ratio has this distribution, the present data give us belief about the distribution of theta.

That is, a process which has only two possible outcomes.

Tags: , ,