Cautions with small samples
Watch out for over-interpreting the results from small samples. Our brains seemed hard-wired to over-interpret the information in small samples and believe they hold more knowledge about how the real world works than they possibly can. Believable anecdotes are often interpreted as if they give substantial data about a topic; coincidences are given mystical or causal interpretations; and patterns seen in small samples are thought to be the forerunner of a shift in paradigm.
The Gambler's Fallacy Section
The Law of Large Numbers does not work by compensation for a run of bad luck. Yet expecting that an independent sequence of trials will be self-correcting is a common misconception. You go to a casino and see red come up five times in a row at the roulette table - should you now bet on black because "it is due." Clearly, the roulette wheel has no memory and each spin provides the same chances for red or black. Succumbing to the misconception that the wheel will somehow compensate for an unusual run of events is known as the "Gambler's Fallacy." It is the flip side of another misconception exemplified by seeing red come up five times in a row at the roulette table and deciding to bet on red because "it is hot."
Outside of the casino, the Gambler's Fallacy is often seen in the habits of small investors in the stock market. If the market is down five days in a row - should you now buy into the market because it is due for an upswing? If the market is up five days in a row - should you now sell your stocks because it is due for a downturn? The chance that the stock market goes up on a randomly chosen day is a little better than 50% (perhaps closer to 51%). However, whether the market will rise on a given day is essentially independent of what happened the day before - it is still 51% regardless of whether the market was up or down the day before. Like the roulette wheel, seeing stocks rise 5 days in a row does not seem to change the chances on what will happen on that sixth day.
Beware of the Anecdote
Anecdotes will often seem compelling, since telling the story of an individual case can make an issue come to life. But moving from a few anecdotes to the general principle is tricky on two fronts - the sample is both small and not chosen randomly from the population.
Coincidences Happen Section
A friend has just purchased a new car - a silver-colored Subaru. A few days later she was stopped at a stoplight and the car in front of her was also a silver-colored Subaru. She looks at that car's license plate and sees that the letters are her initials and the numbers are her ATM PIN number. If you hear about an event like that sometime this year that seems like it has a miraculously small probability of occurring, don't be too shocked - such things are bound to happen someday to someone somewhere. In evaluating the rarity of surprising events, keep in mind all of the times you've had the opportunity for an unusual event to be relayed to you when one was not. You probably have many friends or acquaintances that might have told you about a weird coincidence and 364 days in the year when you did not hear about one. Silver Subarus may be a low percentage of the cars on the road but having just purchased one, you become much more likely to notice them in traffic. The letters and numbers on a license plate being her initials and ATM PIN may seem extremely rare but what if the letters had been a three letter word that she had just used in conversation (about 6% of all three-letter combinations form words) and what if the numbers were her birthday month and day or the last four digits of her social security number or the hour and minute on the clock, or .... Any of these would have been seen as remarkable and the totality of remarkable things may be unremarkable.
Our mind likes to find patterns but the chance processes that generate the pattern are usually independent of the past pattern of results. A stretch of six tosses of a coin that land HHHHHH is just as likely as the sequence TTHHTH. The coin does not care that we see a pattern in the first sequence but not in the second one.
There is no "Law of Small Numbers" as small samples are subject to unpredictable volatility and hence not necessarily representative of the population.