OK, so observing things has an effect on them, fine and dandy. Well, a bunch of physicists started criticizing the uncertainty principle, saying things like, "Oh, your equipment just sucks at these tiny levels," or, "Hey, you're just using too clumsy a method to make your observation." In other words, the methods and equipment were being criticized, and everyone thought we just needed to invent more precise equipment, or discover more precise methods - until Einstein came along. In 1935, the physicists Einstein, Podolsky, and Rosen wrote a paper outlining what is now called the EPR paradox. This paper said, in a nutshell, "No, you're doing it wrong!" The upshot of it was that the observations we were obtaining were not due to shoddy equipment or anything of the kind; rather, the critical physicists didn't properly understand the uncertainty principle, and so weren't taking it seriously. An experiment was laid out that could settle the matter empirically by testing quantum entanglement - but Einstein had no idea as to how such an experiment could be conducted.
Enter John Bell, who was able to devise a method for performing the experiment in 1964. This class of experiment became known as "Bell test experiments," and has shown some interesting and confusing things about the world we live in. To make an already-long story medium-length, I'll skip the technical stuff and get right to brass tacks. Basically, we take two quantum-entangled particles, let's call them A & B. We measure something about A, say its mass, and then we measure something about B, say its velocity. Then we go to measure the mass of A again - and much to our surprise and philosophical chagrin, A's mass has changed. What this means is that, in a state of quantum uncertainty, things aren't just too subtle to measure without changing - they are actually probabilistic (more technical and more surprising, quantum entanglement is an observable phenomenon!). The research strongly suggests that there is actually an element of irreducible randomness to reality.
There are, of course, various interpretations for these results. Any finite set of data has an infinite set of possible interpretations, after all. These interpretations fall into three main lines: the "hidden variables" interpretation (endorsed by Einstein, but losing ground), the "many worlds" interpretation (which has a variety of metaphysical & epistemological convolutions all its own), and the "Copenhagen interpretation" (which posits metaphysical chance). The many worlds and Copenhagen interpretations are epistemologically indistinguishable, as I shall soon demonstrate, and also seem to be "fallback positions" should the hidden variables interpretation be soundly defeated at some point. Since the hidden variables interpretation is the simplest, I shall explain it first: these apparently probabilistic events are in reality governed entirely by some as-yet undiscovered set of principles which, upon their discovery and elaboration, shall show quantum uncertainty to be an illusion. It leaves a lot to be desired, sure, but what it's got going for it is that it's really close to the default position of science (as opposed to, say, fucking magic). But this view is losing ground, as our ever-closer and ever-more-careful observations of the world have been tending to lead us away from a fully deterministic world model.
The Copenhagen interpretation (there are actually several, the term is fraught with ambiguity) posits metaphysical chance as the explanation for these observations. Reality, on this view, contains some irreducibly random component that is fundamentally incompatible with a fully deterministic worldview. "Chance," it would seem, is not merely a way of talking about the limitations of our knowledge, but a fundamental part of the Universe. Let us use the metaphor of a deck of cards. In the everyday world, every deck of cards has a certain order which is determined entirely by the processes of its previous arrangements (whether by shuffling or otherwise). If one knew the initial conditions of the deck (or the conditions of the deck at any time T) and was able to precisely observe and track the deck's arrangement through time up until the present, then one could know with certainty the arrangement of the deck at present. It so happens that our inability to observe and track such minute events as the shuffling of a deck of cards prevents us from knowing its current arrangement, despite the fact that we know how the last hand went. In opposition to the hidden variables interpretation, the Copenhagen interpretation is the idea that not only are we simply incapable of tracking the shuffling of our quantum deck, all the cards are blank until we look at them. In a state of quantum uncertainty, prior to any measurement, there is no fact of the matter. Period. This is what is meant by metaphysical chance.
But it goes deeper than that. As the Bell experiments show, a "fact of the matter" emerges when an observation is made. As a young, naive physicist, I once postulated that matter behaved as waves until some sort of interaction behaved as a "call function," saying more or less, "Hey, you! Collapse your waveform, already! You're supposed to be particulate for this!" The problem with this position is that all matter in the Universe is always interacting with other matter in the Universe. It doesn't matter how much we isolate an electron from other charges, or the atmosphere, or cosmic background radiation - we have found no way to isolate an environment from ambient gravity, which means that it's not mere interactions with other matter that make the difference. No, what seems to be the "special" thing about the observer effect is that the observer is conscious. These are the interactions that matter, and this is friggin' strange. But enough on the Copenhagen interpretation for now.
Straddling the fence between the two is the fully deterministic and fully probablilistic mind-bender known as the many worlds hypothesis. In short, this is going to be weird. In "less short," the many worlds hypothesis is the view that all possible futures actually happen "on top of" each other in the Universe; or, put differently, that any "world-state" can be captured by an internally consistent mathematical model, and the "real world" is composed of all such models. This interpretation, I think, is best explained by a diagram:
This is the world!
The above image represents an extremely simplified (but complete) "many worlds" world. Also, I'm really sorry1 that sloppy MS Paint drawings are so much fun for me. OK, so T represents some initial condition, like the beginning of the universe or something, and the vertical axis represents time - time goes down here, because for something to "go down" means for it to "occur." Or give head. Your choice. Anyway, each horizontal line represents the smallest meaningful interval of time, which is the time between quantum events. Now, here's what's crazy: an observer at A can look back and see a single path going back to T, but looking forward sees all possible futures, a couple of which end abruptly, some of which go on for a while to be radically different from each other. But at A, our observer has but a single "option" open: go to A' or go to A'' (for the benefit of the uninitiated, these are pronounced "A prime" and "A double-prime," respectively). To which path does our observer travel down? Both. In fact, our observer at A travels to all endpoints in the world (assuming that death and destruction are scrupulously avoided along the way, of course).
Now, let us posit our observer at A'. Our observer still sees but a single path going back to T, but can only look forward to all paths proceeding from A', and none of the paths proceeding from A''. Vice versa for an observer posited at A''. So which is the "real" observer from A, just one quantum event earlier? Well, this is just the paradox of the ship of Theseus in fancy dress: our observer at A becomes both our observer at A' and our observer at A'', and neither of the derivative observers has any privileged claim to being "the" observer from A. In fact, if we want to play rigid designator with our identities, they're all the "same" observer. Time for a pop quiz! Suppose that you, personally are at A', going to one of the two points designated B: even with perfect knowledge of the possible states of the world, as well as the laws of causality and probability, can you predict which point B you will go to?
The clever reader will note that this is a trick question: you go down both. It is only by conceit of hindsight at either of the points B that you can describe a singular "you" as having come from A'. Similarly, the "you" at A travels to all points B and C, and only by conceit of hindsight at any of these may you say that "you" travelled down any particular path, because the "whole you" goes down all of them. Now here's the upshot of the diagram: since the "whole thing" is the complete world, every quantum event goes every way it can, and the act of observing quantum states (i.e. collapsing the waveforms out of quantum uncertainty) serves only to tell us what path we have just come down. Under the many worlds hypothesis, all the quantum states that can possibly happen, do actually happen2. This means that an "un-simplified" world diagram would be a nigh-infinite-dimensional idea space describing all logically possible states of the Universe, all of which are equally real, all of which truly exist as much as any other. Capisce?
I told you it was weird.
As promised, my penultimate note will be on the epistemological indistinguishability of the Copenhagen interpretation and the many worlds hypothesis. If you've stuck with me this far, but didn't understand that last sentence... well, you know what? I'm willing to say that that's entirely my fault - I'm trying to write this for the layman, after all. What I mean is that, if either the Copenhagen interpretation or the many worlds hypothesis was true, we could not tell the difference between them. The reason for this is that, under the many worlds hypothesis, we can only describe the future in terms of a probabilistic idea space (this is in fact what the many worlds hypothesis says reality is like). However, in hindsight, we will see only a series of quantum events that played out probabilistically. But under the Copenhagen interpretation, we can only describe the future in terms of probabilities, and if we were to fully describe all possible futures, this would be exactly the same as the many worlds idea space. And looking back, we would also see a series of quantum events that played out probabilistically - not due to some conceit of hindsight, but because this is what the world is actually like. To strain the language of convenience between two mutually exclusive possibilities, "we" would "experience" only one "actual path" down the idea space, but could only describe this path by looking backward, which looks the same under either model. Or, in other words, if you're standing at A', how could you possibly know there's actually an observer at A'' who diverged from you at A, as opposed to there being only you at A' based on a cosmic roll of the die?
Pro tip: you could do no such thing.
OK, so: here are my questions for physicists, at long last! First, how could we really give the hidden variables interpretation a sound thrashing? And suppose we did manage to falsify it, how could we then distinguish between the Copenhagen interpretation and the many worlds hypothesis? Furthermore, if the many worlds hypothesis is true, then how are we to reconcile this with the laws of thermodynamics (specifically, the second one)? Moreover, if the many worlds hypothesis is true, then that gives every logically consistent counterfactual a concrete referent, so why do I want to punch David Lewis in the face? (Sorry, I couldn't resist the esoteric philosophy joke.) And if the Copenhagen interpretation is true, does this mean that there simply was no fact of the matter regarding the Universe as a whole prior to conscious observation? (And if so, how did conscious observers, in fact, come about?) Finally, doesn't the Copenhagen interpretation necessarily exclude an omniscient deity? I mean, I'm pretty sure it does. Just sayin'.
I know it's been a long post, and my deepest thanks to those who stuck it out with me. I hope you can see, however, that the last paragraph would have made exactly zero sense to non-physicists without nearly all of the foregoing. Anyway, enjoy involuntarily reevaluating your conception of reality! Catch you next time around!
1. At this point, I started drawing my MS Paint diagram, and I should really point out that MS Paint is like playing with crayons for me - it makes me giddy. As schoolgirls are giddy. Point is, I couldn't resist the silliness in the following paragraph. Hell, I still can't. It's got that whimsical MS Paint diagram above it, playin' at quantum physics! That's just too cute!
2. Full disclosure: the green, blue, pink, purple, and brown lines were all drawn just for fun. I actually drew out the whole diagram before I had even planned what to write (I'm a bad writer, I know), and I drew most of it because I felt like it. I'm telling you: like playing with crayons.