Essence of Information

This post draws significantly on William Poundstone

I. INFORMATION

Information, unlike, say water, can be compressed so that we can fit more it through a communication channel. This implies that messages are like sponges; preserve the substance and you can squeeze out the air. Morse code accomplishes this by frequent letters having efficient symbols, and infrequent letters having inefficient symbols.

What is the "substance" of a message? What is the part of a message you cannot do without?

We now know that information exists only when the sender is saying something that the recipient doesn't already know and cannot predict. Because true information is unpredictable, it is essentially a series of random events like spins or a roulette wheel, or rolls of a dice.

The essence of information is its improbability.

For information, this means we encode information (compress) sufficiently that the chance of "noisy" errors virtually non-existent, no matter how noisy the communication channel.

Which leads, of course, to physicist John Kelly's paper "Information Theory and Gambling".

The best gambling strategy (or so we can debate) is the one which affords the highest compound return consistent with no risk of going brok. Just as it is possible to send messages through a communication channel with virtually no chance of error, its is possible for a bettor to compound wealth at a certain maximum rate with virtually no risk of ruin.

We wager: Edge/odds

Edge is the amount of profit we expect to win, on average assuming we can make the same wager over and over with the same probabilities.

Odds is the profit if we win. Odds is not a good measure of probability, because odds are set by market forces (i.e. everyone else's probability of winning).

When the edge is zero, you don't bet. Equivocation is the measure of ambiguity (the noisy channel). Equivocation describes the chance that the message we receive is wrong.

Gmax = R

Where G is the growth rate of the gamblers money (a way of saying compound rate of return) and R is the information transmission rate in Shannon's theory. Money = information.