Gibbs Sampling is a way of simulating a random vector with a complex distribution if we have a simple conditional distribution of each coordinate given the other coordinates. The conditional distributions may be easier to sample than the resulting joint distribution because that is how the variables are defined. Suppose (X1, ... , Xp) is the random vector that we have to simulate. The Gibbs sampler follows the algorithm:
1] Initialize values of the random vector: (X1(0), ... , Xp(0)).
2] Let (X1(k), ... , Xp(k)) be the current value of the random vector (X1, ... , Xp). Simulate the new realization X1(k+1) of variable X1 out of the conditional distribution f( X1 | X2 = X2(k), ... , Xp = Xp(k)).
3] Now (X1(k+1), X2(k), ... , Xp(k)) is the current value of the random vector (X1, ... , Xp). Simulate the new realization X2(k+1) of variable X2 out of the conditional distribution f( X2 | X1 = X1(k+1), X3 = X3(k), ... , Xp = Xp(k)).
4] Repeat step 3 for variables X3, ... , Xp until you obtain the realization (X1(k+1), ... , Xp(k+1)).
5] Repeat steps 2 - 4 multiple times until the simulated distribution of vector (X1, ... , Xp) converges to its true distribution.
GIBBS SAMPLER REFERENCES
Robert, C. P., & Casella, G. (2004). Monte Carlo Statistical Methods (2nd ed). New York: Springer.
Casella, G., & George, E. I. (1992). Explaining the Gibbs sampler. The American Statistician 46 (3): pp. 167–174.
Liu, J. S. (2008). Monte Carlo Strategies in Scientific Computing. Springer.
Brooks, S., Gelman, A., Jones, G., & Meng, X. (2011). Handbook of Markov Chain Monte Carlo. Chapman & Hall / CRC.
Gilks, W. R., Richardson, S., & Spiegelhalter, D. (ed) (1995). Markov Chain Monte Carlo in Practice. Chapman & Hall / CRC.
Berg, B. A. (2004). Markov Chain Monte Carlo Simulations and Their Statistical Analysis (With Web-Based Fortran Code). Hackensack, NJ: World Scientific.
Lemieux, C. (2009). Monte Carlo and Quasi-Monte Carlo Sampling. Springer.
Bishop, C. M. (2006). Pattern Recognition and Machine Learning. Springer.
GIBBS SAMPLER RESOURCES