Stochastic inflation describes the global structure of the inflationary universe by modeling the super-Hubble dynamics as a stochastic system of matter fields coupled to gravity where the sub-Hubble field fluctuations induce the stochastic noise. The subsequent dynamics is ultralocal, allowing us to neglect spatial derivatives and treat each Hubble patch as a separate universe. This provides a natural framework in which to discuss probabilities on the space of solutions and initial conditions. In this article we derive an evolution equation for this probability for arbitrary matter systems by using detailed balance as a guiding principle, and demonstrate that this is consistent with a first principles derivation. At no point do we need to assume slow roll or quasi-de Sitter geometry, and so our results are directly applicable to non-slow roll models such a k-inflation and DBI-inflation. We discuss in general terms the conditions for eternal inflation to set in, volume weighting, and the description of stochastic tunneling a la Hawking Moss.