The most frustrating aspect of working in computational probability/statistics is that it's basically impossible to construct algorithms that actually return well-defined probabilistic quantities and this results in no end of chaos.

At best algorithms can return approximations with quantifiable error, but to understand when approximations are useful you have to learn enough math to understand the exact result and how the algorithmic approximation relates to that exact result. Many do not do this.

More commonly programmers project the heuristics that prove successful in other computing problems -- pattern matching, type consistency, unit testing, relying on compiler errors, etc -- but these test only the algorithm and not the relevance of the algorithm to a stats problem.

Unaware of these subtleties many end up conceptually replacing the algorithm for the output being approximated, assuming that algorithmic properties are inherent and well-defined features of the underlying probabilistic/statistical system.

Needless to say this generalizes...poorly. Even worse: without formal knowledge of what is being approximated the poor generalization performance itself is easy to ignore and naive applications drift ever so steadily away from any well-defined mathematical objective.

A short thread on what I think is particularly useful intuition for the application of computational statistics.

Deterministic methods, like variational Bayes, utilize _rigid_ approximations. Ultimately these methods try to find the best way to wedge an approximating solid into the desired solid. If the two shapes are close enough then you can get a good fit.

But this rigidity also means that the approximating solid can't always contort to fit into the desired solid. If the two shapes don't match up well enough then we will end up in lots of awkward configurations no matter how hard we push.

In particular it's hard to quantify how good the fit of the approximate solid into the desired solid might be without knowing the shape of the desired solid already, which isn't possible in practice. This is one reason why these methods have such limited empirical diagnostics.

Stochastic, methods, on the other hand, utilize more fluid approximations that are able to take the shape of their container given enough time. In this respect we can think of them like a liquid or gas filling the desired shape.