| : |
|
|
- Suppose you
could discern market trends, speed up
time to see where those trends were
going, then make a bet on what you
discovered. That's how geeks in suits --
known as rocket scientists -- are raking
it in now. Wired's Kevin Kelly reports on
hacking financial markets.
- ------------------------------------------------------------------------
- "Tell me
about the future," I begged. I was
sitting on a sofa in the guru's office. I
trekked to this high mountain outpost a
couple of years ago to arrive at one of
the planet's power points, the national
research labs at Los Alamos, New Mexico.
The guru's office was decorated with
colorful posters of past conferences that
traced the almost mythical career of this
high-tech legend: from a maverick physics
student who formed an underground band of
hippie hackers to break the bank at Las
Vegas with a wearable computer, to a
principal character in a renegade band of
scientists who invented the accelerating
science of chaos by studying a dripping
faucet, to a founding father of the
artificial life movement, to the head of
a small lab investigating the new science
of complexity in an office kitty-corner
to the museum of atomic weapons at Los
Alamos -- the office I had trekked to.
- The guru,
Doyne Farmer, looked like Ichabod Crane
in a bolo tie. Tall, bony, probably
thirty-something, Doyne (pronounced Doan)
was about to embark on his next
remarkable adventure. He was starting a
company to beat the odds on Wall Street
by predicting stock prices with computer
simu-lations. He was going to hack the
global economy.
- Money is just
a type of information, a pattern that,
once digitized, becomes subject to
persistent programmatic hacking by the
mathematically skilled. As the
information of money swishes around the
planet, it leaves in its wake a history
of its flow, and if any of that complex
flow can be anticipated, then the hacker
who cracks the pattern will become a rich
hacker.
- Some sort of
financial hacking has been around as long
as computers. The art reached an apex in
the late 1980s when powerful PCs allowed
any numskull to click on a computer and
perform quick and massive trades under
certain favorable conditions. When a
heavily computerized stock market
collapsed in October 1987, this
"programmed trading" was
blamed.
- In the
subsequent years, as the stock market
rebounded, the increasing computerization
of financial trading was ignored. More
and more of the everyday tide of money
flowed in digital bits, and more and more
sophisticated games could be played with
them. The traditional financial gamble of
derivatives is the latest to be
computerized.
- A derivative
is sort of a bet on a bet, or a
speculation squared. Really complex
derivatives may give you, say, the option
of buying milk at a certain price in New
Zealand while simultaneously selling oil
in Taiwan. Third- and fourth-order
derivatives -- those betting on an option
based on a bet that hinges on another
gamble -- up the complexity and
incomprehensibility of these financial
instruments.
- Derivatives
are only made possible by the immense
number-crunching power of 1990s desktop
computers. Yet exotic bets form an
increasingly major part of the world
economy. The bulk of the approximately
US$14 trillion that is entangled in
derivatives is three times as much money
as is tied up in the ordinary stocks and
bonds that these esoteric gambles are
derived from. When the stock market
shuddered in early 1994, computerized
derivatives were blamed.
- But I wasn't
entreating Doyne Farmer because I was
interested in derivatives. "I've
been thinking about the future, and I
have one question," I told Farmer.
- "You
want to know if IBM is gonna be up or
down," Farmer suggested with a wry
smile.
- "No. I
want to know why the future is so hard to
predict."
- Of course I
wouldn't mind knowing if IBM is going up
or down. Might be quite handy every now
and then. But hacking the global economy
big time wasn't going to be any fun
unless it could be done regularly,
dependably, and on a scale where real
money could be wagered. What I wanted to
know was, can you really predict
something as complex and squirrelly as
the stock market? Can you really predict
the future at all?
- Farmer thinks
so. He likes to use a favorite example
when explaining the anatomy of a
prediction. "Here, catch this!"
he says, tossing you a ball. You grab it.
"You know how you caught that?"
he asks. "By prediction."
- Farmer
contends you have a model in your head of
how baseballs fly. You could predict the
trajectory of a high fly using Newton's
classic equation f=ma, but your brain
doesn't stock up on elementary physics
equations. Rather, it builds a model
directly from experiential data. A
baseball player watches a thousand
baseballs come off a bat, and a thousand
times lifts his gloved hand, and a
thousand times adjusts his guess with his
mitt. Without his knowing how, his brain
gradually compiles a model of where the
ball lands -- a model almost as good as
f=ma, but not as generalized. It's based
entirely on a collection of hand-eye data
from past catches. In the field of logic
such a process is known as induction, in
contradistinction to the deduction
process that leads to f=ma.
- In the early
days of astronomy, before the advent of
Newton's f=ma, planetary events were
predicted on Ptolemy's model of nested
circular orbits -- wheels within wheels.
Because the central premise upon which
Ptolemy's theory was founded (that all
heavenly bodies orbited the Earth) was
wrong, his model needed mending every
time new astronomical observations
delivered more exact data for a planet's
motions. But wheels-within-wheels was a
model amazingly robust to amendments.
Each time better data arrived, another
layer of wheels inside wheels inside
wheels was added to adjust the model. For
all its serious faults, this baroque
simulation worked and
"learned." Ptolemy's
simple-minded scheme served well enough
to regulate the calendar and make
practical celestial predictions for 1,400
years.
- An
outfielder's empirically based
"theory" of missiles is
reminiscent of the latter stages of
Ptolemaic epicyclic models. If we parsed
an outfielder's "theory" we
would find it to be incoherent, ad hoc,
convoluted, and approximate. But it would
also be evolvable. It's a rat's nest of a
theory, but it works and improves. If we
humans had to wait until each of our
minds figured out f=ma (and half of f=ma
is worse than nothing), no one would ever
catch anything. Even knowing the equation
now doesn't help. "You can do the
flying baseball problem with f=ma, but
you can't do it in the outfield in real
time," says Farmer.
- "Now
catch this!" Farmer says as he
releases an inflated balloon. It
ricochets around the room in a wild,
drunken zoom. No one ever catches it.
It's a classic illustration of chaos -- a
system with sensitive dependence on
initial conditions. Imperceptible changes
in the launch can amplify into enormous
changes in flight direction. Although the
f=ma law still holds sway over the
balloon, other forces such as propulsion
and air lift push and pull, generating an
unpredictable trajectory. In its chaotic
dance, the careening balloon mirrors the
unpredictable waltz of sunspot cycles,
the Ice Age's temperatures, epidemics,
the flow of water down a tube, and --
more to the point -- the flux of the
stock market.
- But is the
balloon really unpredictable? If you
tried to solve the equations for the
balloon's crazy flitter, its path would
be nonlinear, therefore almost
unsolvable, and therefore unforeseeable.
Yet, a teenager reared on Nintendo could
learn how to catch the balloon,
sometimes. Not infallibly, but at a rate
better than chance. After a couple dozen
tries, the teenage brain begins to mold a
theory -- an intuition, an induction --
based on the data. After a thousand
balloon takeoffs, his brain has modeled
some aspect of their flight. It cannot
predict precisely where a balloon will
land, but it detects a direction the
missile favors, say, to the rear of the
launch or following a certain pattern of
loops. Perhaps over time, the balloon
catcher hits 10 percent more than chance
would dictate.
- If balloon
catching paid $1,000 a hit, what more do
you need? In some games, one doesn't
require much information to make a
prediction that is useful. While running
from lions, or investing in stocks, the
tiniest edge over raw luck is
significant.
- Farmer calls
it "gambling with a positive
edge." He believes that by gambling
with a positive edge he can crack the
stock market. "The nice thing about
markets is that you don't really have to
predict very much to do an awful
lot," he says.
- Plotted on
the gray end-pages of a newspaper, the
graphed journey of the stock market as it
rises and falls has just two dimensions:
time and price. For as long as there has
been a stock market, investors have
scrutinized that wavering two-dimensional
black line in the hopes of discerning
some pattern that might predict its
course. Even the vaguest, if reliable,
hint in direction would lead to a pot of
gold. Pricey financial newsletters
promoting this or that method for
forecasting the chart's future are a
perennial fixture in the stock market
world. Practitioners are known as
chartists.
- In the '70s
and '80s chartists had modest success in
predicting currency markets because, one
theory says, the strong role of central
banks and treasuries in currency markets
constrained the variables so that they
could be described in relatively simple
linear equations. (In a linear equation,
a solution can be expressed in a graph as
a straight line.) As more and more
chartists exploited the easy linear
equations and successfully spotted
trends, the market became less
profitable. Naturally, forecasters began
to look at the wild and woolly places
where only chaotic nonlinear equations
ruled. In nonlinear systems, the outcome
is not proportional to the input. Most
complexity in the world -- including all
markets -- are nonlinear.
- More
importantly, all classical economic
understanding has been based on the
premise that the collective action of a
market is not biased toward particular
players or patterns but operates
consistently and uniformly throughout the
marketplace. An unbiased marketplace was
seen to be maximally efficient in
allotting resources and rewards; any
inefficiency in the marketplace would be
exploited by players until it
disappeared. Indeed, the efficiency of a
random market was the cornerstone of
modern economic theory.
- But with the
advent of cheap, industrial-strength
computers, mathematicians have been able
to understand certain aspects of
nonlinearity and are now questioning the
premise of random markets. While
traditional economists elaborate ever
more complicated explanations of how
markets cannot be predicted, money -- big
money -- is being made by math jockeys
who extract reliable patterns out of the
nonlinearity of financial prices and then
bet on them. The computer nerds who
decipher these esoteric methods, and who
have invented computerized derivatives,
are called "rocket scientists"
or "quants," short for
quantitative analysts. These geeks in
suits, working in the basements of
trading companies, are the capitalist
hackers of the '90s.
- The aim of
the quants is to use sophisticated
computer programs and mathematical
formulae to play with market data and to
try to predict the future a bit. Since
all that is required to play the game is
a Unix workstation and a math degree, a
lot of the action takes place off the big
city trading floors. At Wall Street
Analytics, based at the center of Silicon
Valley in Palo Alto, California, rocket
scientists devised a program (list price
$50,000), running on PCs, to compute
scenarios of mortgage-pool investments.
By ingeniously regrouping pools of
mortgages and then extrapolating the
profits under different "what
if" conditions (much like a
spreadsheet but with massive
criss-crossing feedback of the
variables), the software is able to
isolate crummy mortgages (nicknamed
"toxic waste") from more
profitable ones. The computer plays out
hundreds of scenarios of different
"slices" of the mortgage pool
in rapidly fluctuating financial
environments. One moment a particular
mortgage is toxic, the next it's golden.
The quants use the computer to sift
through this seething mass of interlinked
considerations to select the most
profitable buy at any given moment.
- Given the
propeller-headed math needed to build
this incredibly complex forecast, it is
not surprising that Wall Street Analytics
was started by ex-physicists.
"Investing is increasingly becoming
dominated by mathematicians, electrical
engineers, and programmers," says
Adrian Cooper, vice president of Wall
Street Analysts. "As a grad student
in physics you spend five years banging
your head against an abstract problem
that no one can solve. This prepares you
for working with intractable problems on
Wall Street." And the pay can pop
the eyes of a postdoc. Salaries on Wall
Street can be three times those of
physicists. According to Science
magazine, "Two of the four students
who received doctorates in theoretical
physics from Harvard went off to jobs on
Wall Street.... Of the twenty or so
students who received theoretical physics
doctorates over the last five years from
Stanford University, only two or three
[classmates say] are still in physics;
they can name eight or nine who are
working in finance."
- I sought out
Doyne Farmer, former mathematical
physicist (he banged his head against the
insoluble problem of systems with
infinite numbers of dimensions while in
graduate school), because he is currently
one of the financial world's hottest
rocket scientists. Together with
colleagues from his earlier mathematical
adventures, Farmer moved his filing
cabinets from Los Alamos and set up an
office in a small, four-room house in
adobe-adorned Santa Fe. Using their own
custom workstations and software, these
ex-physicists are directly wired into the
financial trading markets in Chicago.
They are building the Supercollider of
Finance.
- Farmer is
betting that they can take what they
learn from dissecting the trajectories of
colliding bits of digitized money and use
this information to play the market.
- The
two-dimensional chart of stock market
prices over time hides a hodge-podge of
forces driving the price line. A true
graph would include an axis for every
influence and would thus become an
unpicturable, thousand-armed monster.
- Mathematicians
struggle with ways to tame these
monsters, which they call
"high-dimensional" systems. A
mere 100 variables create a humongous
swarm of possibilities. Because each
behavior impinges upon the 99 others, it
is impossible to examine one parameter
without examining the whole interacting
swarm at once. Even a simple
three-variable model of weather, say,
touches back upon itself in strange
loops, breeding chaos and making any kind
of linear prediction unlikely. (The
failure to predict weather led to the
discovery of chaos theory in the first
place.)
- Pop wisdom
says that chaos theory proves that these
high-dimensional complex systems -- such
as the weather, the economy, army ants,
and, of course, stock prices -- are
intrinsically, no-way-around-it
unpredictable. So ironclad is the popular
assumption that in common perception any
design for predicting the outcome of a
complex system is considered naive or
mad.
- But chaos
theory is vastly misunderstood. It has
another face. Farmer suggests chaos is
like a hit (vinyl) record with two sides.
The lyrics to the hit side go: "By
the laws of chaos, initial order can
unravel into raw unpredictability. You
can't predict far." But the flip
side goes: "By the laws of chaos,
things that look completely disordered
may be predictable over the short term.
You can predict short."
- In other
words, the character of chaos carries
both good news and bad news. The bad news
is that very little, if anything, is
predictable far into the future. The good
news -- the flip side of chaos -- is that
in the short term, more may be more
predictable than it first seems. Both the
long-term, unpredictable nature of the
high-dimensional systems and the
short-term, predictable nature of
low-dimensional systems derive from the
fact that "chaos" is not the
same thing as "randomness."
"There is order in chaos,"
Farmer says.
- Farmer should
know. He was an original pioneer into the
dark frontier of chaos before it jelled
into a scientific theory and faddish
field of study. In the hip California
town of Santa Cruz in the 1970s, Doyne
Farmer and friend Norm Packard co-founded
a commune of nerd hippies who practiced
collective science. They shared a house,
meals, cooking, and credit on scientific
papers.
- As the
"Chaos Cabal," the band
investigated the weird physics of
dripping faucets and other seemingly
random generating devices. Farmer in
particular was obsessed with the roulette
wheel. He was convinced that there must
be hidden order in the apparently random
spinning of the wheel. If one could
discern secret order among the spin-ning
chaos, then . . . why, one could get rich
. . . very rich.
- In 1977, long
before the birth of commercial
microcomputers such as the Apple, the
Chaos Cabal built a set of handcrafted,
programmable, tiny microcomputers into
the bottoms of three ordinary leather
shoes. The computers were keyboarded with
toes; their function was to predict the
toss of a roulette ball. The home-brew
computers ran code devised by Farmer
based on the group's study of a purchased
secondhand Las Vegas roulette wheel set
up in one of the commune's crowded
bedrooms. Farmer's computer algorithm was
based not on the mathematics of roulette
but on the physics of the wheel. In
essence, the cabal's code simulated the
entire rotating roulette wheel and
bouncing ball inside the chip in the
shoe. And it did this in a minuscule 4
Kbytes of memory, in an era when
computers were behemoths demanding
24-hour air conditioning and an attendant
priesthood.
- On more than
one occasion, the science commune played
out the flip side of chaos in a scene
like this: Wired up at the casino, one
person (usually Farmer) wore a pair of
magic shoes to calibrate the roulette
operator's flick of the wheel, the speed
of the bouncing ball, and the tilt of the
wheel's wobble. Nearby, a cabal cohort
wore the third magic shoe that was linked
to Farmer's by radio signals and placed
the actual bet on the table.
- Earlier,
using his toes, Farmer had tuned his
algorithm to the idiosyncrasies of a
particular wheel in the casino. Now, in
the mere fifteen seconds or so between
the drop of the ball and its decisive
stop, his shoe-computer simulated the
full chaotic run of the ball. About a
million times faster than it took the
real ball to land in a numbered cup,
Farmer's prediction machinery buzzed out
in small taps the ball's future
destination on his right big toe. Typing
with his left big toe, Farmer transmitted
that information to his partner, who
"heard" it on the bottom of his
feet, and then, with a poker face, pushed
the chips onto the predetermined squares
before the ball stopped.
- When
everything worked, the chips won. The
system never predicted the exact winning
number: the cabal members were realists.
Their prediction machinery forecast a
small neighborhood of numbers -- one
octave section of the wheel -- as the
bettable destination of the ball. The
gambling partner spread the bets over
this neighborhood as the ball finished
spinning. Out of the bunch, one won.
While the companion bets lost, the
neighborhood as a whole would win often
enough to beat the odds. And make money.
- The group
sold the quasi-legal system to other
gamblers because of unreliability in the
hardware. But Farmer learned three
important things about predicting the
future from this adventure:
- First, you
can milk underlying patterns inherent in
chaotic systems to make good predictions.
- Second, you
don't need to look very far ahead to make
a useful prediction.
- And third,
even a little bit of information about
the future can be valuable.
- With these
lessons firmly in mind, Farmer, together
with five other physicists (one of them a
former Chaos Cabal member) engineered a
start-up company to crack every gambler's
dream: Wall Street. They would use
high-powered computers. They would stuff
them with experimental nonlinear dynamics
and other esoteric rocket-scientist
formulas. They would hack the financial
world with the help of banks; it would be
very legal, too.
- They would .
. . (drum roll, please) . . . predict the
future. With a bit of bravado, the old
gang hung out their new shingle: the
Prediction Company.
- The guys in
the Prediction Company figure that
looking ahead a few days into the
financial market future is all that is
needed to make big bucks. Prediction
machinery need not see like a prophet to
be of use. It needs only to detect
limited patterns -- almost any pattern --
out of a camouflage of randomness and
complexity.
- According to
Farmer, there are two kinds of
complexity: inherent and apparent.
Inherent complexity is the
"true" complexity of chaotic
systems. It leads to dark
unpredictability. The other kind of
complexity is the complement of chaos --
apparent complexity obscuring exploitable
order.
- Farmer draws
a square in the air. Going up the square
increases apparent complexity; going
across the square increases inherent
complexity. "Physics normally works
down here," Farmer says, pointing to
the bottom corner of low complexity for
both sorts, home of the easy problems.
"Out there," pointing to the
opposite upper corner, "it's all
hard. But we are now sliding up to here,
where it gets interesting -- where the
apparent complexity is high, but the true
complexity is still low. Up here, complex
problems have something in them you can
predict. And those are exactly the ones
we are looking for in the stock
market."
- With crude
computer tools that take advantage of the
flip side of chaos, the Prediction
Company hopes to knock off the easy
problems in financial markets.
- "We are
using every method we can find,"
says partner Norman Packard, a former
Chaos Cabalist. The idea is to throw
proven pattern-finding strategies of any
stripe at the data and "keep
pounding on them" to optimize the
algorithms. Find the merest hint of a
pattern, and then exploit the daylights
out of it. The mind-set here is that of a
gambler: any positive edge is an
advantage.
- Farmer and
Packard's motivating faith that chaos
possesses a flip side firm enough to bank
on is based on their own experience.
Nothing overcomes doubts like the
tangible money they won from their
experiments with the Las Vegas roulette
wheel.
- In addition
to experience, Farmer and Packard place a
lot of faith in the well-respected
theories they invented during their years
in chaos research. Now they are testing
their wildest, most controversial theory
yet. They believe, against the unbelief
of most economists, that "the market
is not random." They hold that
certain regions of otherwise complicated
financial phenomena can be predicted
accurately. Packard calls these areas
"pockets of predictability" or
"local predictability." In
other words, the distribution of
unpredictability is not uniform
throughout systems. Most of the time,
most of a complex system may not be
forecastable, but some small part of it
may be for short times. In hindsight,
Packard believes local predictability is
what allowed the Santa Cruz Chaos Cabal
to make money forecasting the approximate
path of a roulette ball.
- If there are
pockets of predictability, they will
surely be buried under a haystack of
gross unpredictability. The signal of
local predictability can be masked by a
swirling mess of noise from a thousand
other variables. The Prediction Company's
thirteen quants and rocket scientists use
a mixture of old and new, high-tech and
low-tech search techniques to scan this
combinatorial haystack. Their software
examines the mathematically
high-dimensional space of financial data
and searches for local regions -- any
local region -- that might match
low-dimensional patterns they can
predict. They search the financial cosmos
for hints of order, any order.
- They do this
in real time, or what might be called
hyperreal time. Just as the simulated
bouncing roulette ball in the
shoe-computer comes to rest before the
real ball does, the Prediction Company's
simulated financial patterns are played
out faster than they happen on Wall
Street. They reenact a simplified portion
of the stock market in a computer. When
they detect the beginnings of a wave of
unfolding local order, they simulate it
faster than real life and then bet on
where they think the wave will
approximately end.
- David
Berreby, writing in the March 1993 issue
of Discover, puts the search for pockets
of predictability in terms of a lovely
metaphor: "Looking at market chaos
is like looking at a raging white-water
river filled with wildly tossing waves
and unpredictably swirling eddies. But
suddenly, in one part of the river, you
spot a familiar swirl of current, and for
the next five or ten seconds you know the
direction the water will move in that
section of the river."
- Sure, you
can't predict where the water will go a
half-mile downstream, but for five
seconds -- or five hours on Wall Street
-- you can predict the unfolding show.
That's all you really need to be useful
(or rich). Find any pattern and exploit
it. The Prediction Company's algorithms
grab a fleeting bit of order and exploit
this ephemeral archetype to make money.
Farmer and Packard emphasize that while
economists are obliged by their
profession to unearth the cause of such
patterns, gamblers are not bound so. The
exact reason why a pattern forms is not
important for the Prediction Company's
purposes. In inductive models -- the kind
the Prediction Company constructs -- the
abstracted causes of events are not
needed, just as they aren't needed for an
outfielder's internalized ballistic
notions or for a dog to catch a tossed
stick.
- Rather than
worry about the dim relationships between
causes and effects in these massively
swarmy systems crowded with circular
causality, Farmer says, "The key
question to ask in beating the stock
market is, what patterns should you pay
attention to?" Which ones disguise
order? Learning to recognize order, not
causes, is the key.
- Before a
model is used to bet with, Farmer and
Packard test it with backcasting. In
backcasting techniques (commonly used by
professional futurists) a model is built
withholding the most recent data from the
human managing the model. Once the system
finds order in past data, say from the
'80s, it is fed the record of the last
several years. If it can accurately
predict the 1993 outcome, based on what
it found in the '80s, then the pattern
seeker has won its wings. Farmer:
"The system makes twenty models. We
run them each through a sieve of
diagnostic statistics. Then the six of us
will get together to select the one to
run live." Each round of model
building may take days on the company's
computers. But once local order is
detected, a prediction based on it can be
spun in milliseconds.
- For the final
step -- running it live with bundles of
real money in its fists -- one of the
PhDs still has to hit the Enter button.
This act thrusts the algorithm into the
big-league world of very fast,
mind-boggling big bucks. Cut loose from
theory, running on automatic, the
fleshed-out algorithm hums under the
murmurs of its creators in Santa Fe:
"Trade, sucker, trade!"
- "If we
can earn 5 percent better than what the
market does, then our investors will make
money," Packard says. He clarifies
that number by explaining that they can
predict 55 percent of market moves, that
is, 5 percent more than by random
guessing, but that when they do guess
right their result can be 200 percent
better. The fat-cat financial backers who
invest in the Prediction Company
(currently O'Connor & Associates
which, pending approval, will become part
of a new subsidiary of Swiss Bank
Corporation) get exclusive use of the
algorithms in exchange for payments
according to the performance of the
predictions. "We have
competitors," Packard states with a
smile. "I know of four other
companies with the same thing in
mind" -- capturing patterns in chaos
with nonlinear dynamics and predicting
from them. "Two of them are up and
going. Some involve friends."
- One
competitor trading real money is
Citibank. Since 1990, British
mathematician Andrew Colin has been
evolving trading algorithms. His
forecasting program randomly generates
several hundred hypotheses of which
parameters influence currency data, and
then tests the hundreds against the last
five years of data. The most likely
influences are sent to a computer neural
net, which juggles the weight of each
influence to better fit the data
rewarding the best combinations in order
to produce better guesses. The neural net
system keeps feeding the results back in
so that the system can hone its guess in
a type of learning. When a model fits the
past data, it is sent out into the
future. In 1992 The Economist said,
"After two years of experiments, Dr.
Colin reckons his computer can make
returns of 25 percent a year on its
notional dealing capital.... That is
several times more than most human
traders hope to make."
- Midland Bank
in London has eight rocket scientists
working on prediction machinery. In their
scheme, computers breed algorithms.
However, just as at the Prediction
Company, humans evaluate them before
"hitting the Return button."
Midland Bank's computers were trading
real money by late 1993.
- Investors
like to ask Farmer how he can prove he
can make money in markets with the
advantage of only a small bit of
information. As an "existence
proof" Farmer points to people such
as George Soros, who earn millions year
after year trading currencies and whatnot
on Wall Street. Successful traders,
sniffs Farmer, "are pooh-poohed by
the academics as being extremely lucky --
but the evidence goes the other
way." Human traders unconsciously
learn how to spot patterns of local
predictability streaking through the
ocean of random data. The traders make
millions of dollars because they detect
patterns (which they cannot articulate),
then make an internal model (of which
they are unconscious), in order to make
predictions (for which they are rewarded
or punished, sharpening the feedback
loop). They have no more idea of what
their model or theory is than of how they
catch fly balls. They just do. Yet both
kinds of models were empirically
constructed in the same inductive
Ptolemaic way. And that's how the
Prediction Company employs computers to
build models of the flow of money -- from
the data up.
- "Our
predictions are not as certain as a
physicist's prediction of the motion of
planets," Farmer says, "but
they don't have to be. They are less
certain than the laws of motion, but much
more than just random."
- But the laws
of economics may be more certain than
some think. In the early '70s economists
Fischer Black and Myron Scholes came up
with an equation which explains why
derivatives work. The Black-Scholes
equation was quickly adopted by Wall
Street quants to calculate the value of
options in complex derivative dealings. A
lot of them being ex-physicists, they
noticed that the structure of the
equation was parallel to the equation
used in physics to describe the
dissipation of heat. Manipulating this
heavy economic math was just like hauling
heavy physics math. Indeed, Chicago-based
O'Connor & Associates, the
well-heeled firm that funded the
Prediction Company's start-up, made
multimillions from quants running the
Black-Scholes equation in the derivatives
market.
- Says Farmer,
"If we are successful on a broad
basis in what we are doing, it will
demonstrate that machines are better
forecasters than people, and that
algorithms are better economists than
Milton Friedman. Already, traders are
hesitant about this stuff. They feel
threatened by it."
- The hard part
is keeping it simple. Says Farmer,
"The more complex the problem is,
the simpler the models that you end up
having to use. It's easy to fit the data
perfectly, but if you do that, you
invariably end up just fitting to the
flukes. The key is to generalize."
- Prediction
machinery is ultimately theory-making
machinery -- devices for generating
abstractions and generalizations.
Prediction machinery chews on the mess of
seemingly random chicken-scratch data
produced by complex and living things. If
there is a sufficiently large stream of
data over time, the device can discern a
small bit of pattern. Slowly, the
technology shapes an internal ad hoc
model of how the data might be produced.
The apparatus shuns
"overfitting" the pattern on
specific data and leans to the fuzzy fit
of a somewhat imprecise generalization.
Once it has a general fit -- a theory --
it can make a prediction. In fact,
prediction is the whole point of
theories. "Prediction is the most
useful, the most tangible, and, in many
respects, the most important consequence
of having a scientific theory,"
Farmer declares. Manufacturing a theory
is a creative act that human minds excel
at, although, ironically, we have no
theory of how we do it. Farmer calls this
mysterious general-pattern-finding
ability "intuition." It's the
exact technology "lucky" Wall
Street traders use.
- Prediction
machinery is found in biology, too. Dogs
don't do math, yet dogs can be trained to
predictively calculate the path of a
Frisbee and catch it precisely.
Intelligence and smartness in general is
fundamentally prediction machinery.
- Farmer
confessed to a private gathering of
business CEOs, "Predicting markets
is not my long-term goal. Frankly, I'm
the kind of guy who has a hard time
opening to the financial page of the Wall
Street Journal. "For an unrepentant
ex-hippie, that's no surprise. Farmer
sees himself working for five years on
the problem of predicting the stock
market, scoring big time, and then moving
on to more interesting problems -- such
as real artificial life, artificial
evolution, and artificial intelligence.
Financial forecasting, like roulette, is
just another hard problem. "We are
interested in this because our dream is
to produce prediction machinery that will
allow us to predict lots of different
things" -- weather, global climate,
epidemics -- "anything generating a
lot of data we don't understand
well."
- "Ultimately,"
says Farmer, "we hope to imbue
computers with a crude form of
intuition."
-
- So what
happens when everyone is net surfing the
world economy? It'll get more difficult
to stay ahead, and that will prompt many
novices to drop out, leaving the field to
those with the biggest iron, the newest
schemes, and the keenest, most nimble
insights. The only thing the Prediction
Company has over its upcoming competitors
is a two-year lead.
- By late 1993,
the Prediction Company reported success
in predicting markets with
"computerized intuition" in
live trading. They refused, however, to
comment about their performance. Their
agreement with their investors prohibits
them from talking about this, as much as
the talkative ex-physicists are dying to
do so. They were told: "Do not say
anything about performance --
anything!" They won't say how much
money they trade, or exactly where.
(Farmer asked me to let him vet these
closing paragraphs because "if I
inadvertently tell you something I
shouldn't, it could blow our contract
with the Swiss Bank Corporation. You know
the Swiss -- they are secretive to the
point of paranoia.") Indeed, at his
request, I removed the hints of what kind
of markets they track that slipped out
during his conversations with me.
- He would say,
though, that the studies they have done
prove "by rigorous scientific
standards" that financial markets
can in fact be beaten. "We really
have found statistically significant
patterns in financial data. There are
pockets of predictability. We have
learned a lot, and I would love to be
able to describe it all to the world, to
write a technical book laying out the
knowledge we have accumulated on how to
extract the weak signals that exist in
financial markets and trade on them -- a
kind of Theory of Financial Prediction.
Maybe call it How to Beat the Market. But
given the quantity of money involved, I'm
sure our partners will never let us do
that, and I have just enough belief in
efficient markets myself that I probably
agree with them."
- ------------------------------------------------------------------------
- This story is
adapted from Kevin Kelly's new book
"Out of Control: The Rise of
Neo-Biological Civilization,"
published by Addison-Wesely in June 1994.
"Out of Control" is about how
machines are becoming biological so that
we can manage their increasing
complexity.
- ------------------------------------------------------------------------
- Kevin Kelly
(kk@well.com) is executive editor of
Wired.
- ------------------------------------------------------------------------
|