Tag Archives: quantitative finance

Quantitative Finance, The Wilmott Magazine

Functional Programming

Functional programming is the programming methodology that puts great emphasis on statelessness and religiously avoids side effects of one function in the evaluation any other function. Functions in this methodology are like mathematical functions. The conventional programming style, on the other hand, is considered “imperative” and uses states and their changes for accomplishing computing tasks.

Adapting this notion of functional programming may sound like regressing back to the pre-object-oriented age, and sacrificing all the advantages thereof. But there are practitioners, both in academia and in the industry, who strongly believe that functional languages are the only approach that ensures stability and robustness in financial and number crunching applications.

Functional languages, by definition, are stateless. They do everything through functions, which return results that are, well, functions of their arguments. This statelessness immediately makes the functions behave like their mathematical counterparts. Similarly, in a functional language, variable behave like mathematical variables rather than labels for memory locations. And a statement like x = x + 1 would make no sense. After all, it makes no sense in real life either.

This strong mathematical underpinning makes functional programming the darling of mathematicians. A piece of code written in a functional programming language is a set of declarations quite unlike a standard computer language such as C or C++, where the code represents a series of instructions for the computer. In other words, a functional language is declarative — its statements are mathematical declarations of facts and relationships, which is another reason why a statement like x = x + 1 would be illegal.

The declarative nature of the language makes it “lazy,” meaning that it computes a result only when we ask for it. (At least, that is the principle. In real life, full computational laziness may be difficult to achieve.) Computational laziness makes a functional programming language capable of handling many situations that would be impossible or exceedingly difficult for procedural languages. Users of Mathematica, which is a functional language for symbolic manipulation of mathematical equations, would immediately appreciate the advantages of computational laziness and other functional features such as its declarative nature. In Mathematica, we can carry out an operation like solving an equation for instance. Once that is done, we can add a few more constraints at the bottom of our notebook, scroll up to the command to solve the original equation and re-execute it, fully expecting the later constraints to be respected. They will be, because a statement appearing at a later part in the program listing is not some instruction to be carried out at a later point in a sequence. It is merely a mathematical declaration of truism, no matter where it appears.

This affinity of functional languages toward mathematics may appeal to quants as well, who are, after all, mathematicians of the applied kind. To see where the appeal stems from, let us consider a simple example of computing the factorial of an integer. In C or C++, we can write a factorial function either using a loop or making use of recursion. In a functional language, on the other hand, we merely restate the mathematical definition, using the syntax of the language we are working with. In mathematics, we define factorial as:

n! = left{begin{array}{ll}1 & n=1 \n times (n-1)! & textrm{Otherwise}end{array}right.

And in Haskell (a well known functional programming language), we can write:

bang 1 = 1
bang n = n * bang (n-1)

And expect to make the call bang 12 to get the factorial of 12.

This example may look artificially simple. But we can port even more complicated problems from mathematics directly to a functional language. For an example closer to home, let us consider a binomial pricing model, illustrating that the ease and elegance with which Haskell handles factorial do indeed extend to real-life quantitative finance problems as well.

Sections

Quantitative Finance, The Wilmott Magazine

Magic of Object Oriented Languages

Nowhere is the dominance of paradigms more obvious than in object oriented languages. Just take a look at the words that we use to describe some their features: polymorphism, inheritance, virtual, abstract, overloading — all of them normal (or near-normal) everyday words, but signifying notions and concepts quite far from their literal meaning. Yet, and here is the rub, their meaning in the computing context seems exquisitely appropriate. Is it a sign that we have taken these paradigms too far? Perhaps. After all, the “object” in object oriented programming is already an abstract paradigm, having nothing to do with “That Obscure Object of Desire,” for instance.

We do see the abstraction process running a bit wild in design patterns. When a pattern calls itself a visitor or a factory, it takes a geekily forgiving heart to grant the poetic license silently usurped. Design patterns, despite the liberties they take with our sensitivities, add enormous power to object oriented programming, which is already very powerful, with all the built in features like polymorphism, inheritance, overloading etc.

To someone with an exclusive background in sequential programming, all these features of object oriented languages may seem like pure magic. But most of the features are really extensions or variations on their sequential programming equivalents. A class is merely a structure, and can even be declared as such in C++. When you add a method in a class, you can imagine that the compiler is secretly adding a global function with an extra argument (the reference to the object) and a unique identifier (say, a hash value of the class name). Polymorphic functions also can be implemented by adding a hash value of the function signature to the function names, and putting them in the global scope.

The real value of the object oriented methodology is that it encourages good design. But good programming discipline goes beyond mere adaptation of an object oriented language, which is why my first C++ teacher said, “You can write bad Fortran in C++ if you really want. Just that you have to work a little harder to do it.”

For all their magical powers, the object oriented programming languages all suffer from some common weaknesses. One of their major disadvantages is, in fact, one of the basic design features of object oriented programming. Objects are memory locations containing data as laid down by the programmer (and the computer). Memory locations remember the state of the object — by design. What state an object is in determines what it does when a method is invoked. So object oriented approach is inherently stateful, if we can agree on what “state” means in the object oriented context.

But in a user interface, where we do not have much control over the sequence in which various steps are executed, we might get erroneous results in stateful programming depending on what step gets executed at a what point in time. Such considerations are especially important when we work with parallel computers in complex situations. One desirable property in such cases is that the functions return a number solely based on their arguments. This property, termed “purity,” is the basic design goal of most functional languages, although their architects will concede that most of them are not strictly “pure.”

Sections

Quantitative Finance, The Wilmott Magazine

Paradigms All the Way

Paradigms permeate almost all aspects of computing. Some of these paradigms are natural. For instance, it is natural to talk about an image or a song when we actually mean a JPEG or an MP3 file. File is already an abstraction evolved in the file-folder paradigm popularized in Windows systems. The underlying objects or streams are again abstractions for patterns of ones and zeros, which represent voltage levels in transistors, or spin states on a magnetic disk. There is an endless hierarchy of paradigms. Like the proverbial turtles that confounded Bertrand Russell (or was it Samuel Johnson?), it is paradigms all the way down.

Some paradigms have faded into the background although the terminology evolved from them lingers. The original paradigm for computer networks (and of the Internet) was a mesh of interconnections residing in the sky above. This view is more or less replaced by the World Wide Web residing on the ground at our level. But we still use the original paradigm whenever we say “download” or “upload.” The World Wide Web, by the way, is represented by the acronym WWW that figures in the name of all web sites. It is an acronym with the dubious distinction of being about the only one that takes us longer to say than what it stands for. But, getting back to our topic, paradigms are powerful and useful means to guide our interactions with unfamiliar systems and environments, especially in computers, which are strange and complicated beasts to begin with.

A basic computer processor is deceptively simple. It is a string of gates. A gate is a switch (more or less) made up of a small group of transistors. A 32 bit processor has 32 switches in an array. Each switch can be either off representing a zero, or on (one). And a processor can do only one function — add the contents of another array of gates (called a register) to itself. In other words, it can only “accumulate.”

In writing this last sentence, I have already started a process of abstraction. I wrote “contents,” thinking of the register as a container holding numbers. It is the power of multiple levels of abstraction, each of which is simple and obvious, but building on whatever comes before it, that makes a computer enormously powerful.

We can see abstractions, followed by the modularization of the abstracted concept, in every aspect of computing, both hardware and software. Groups of transistors become arrays of gates, and then processors, registers, cache or memory. Accumulations (additions) become all arithmetic operations, string manipulations, user interfaces, image and video editing and so on.

Another feature of computing that aids in the seemingly endless march of the Moore’s Law (which states that computers will double in their power every 18 months) is that each advance seems to fuel further advances, generating an explosive growth. The first compiler, for instance, was written in the primitive assembler level language. The second one was written using the first one and so on. Even in hardware development, one generation of computers become the tools in designing the next generation, stoking a seemingly inexorable cycle of development.

While this positive feedback in hardware and software is a good thing, the explosive nature of growth may take us in wrong directions, much like the strong grown in the credit market led to the banking collapses of 2008. Many computing experts now wonder whether the object oriented technology has been overplayed.

Sections

Quantitative Finance, The Wilmott Magazine

Zeros and Ones

Computers are notorious for their infuriatingly literal obedience. I am sure anyone who has ever worked with a computer has come across the lack of empathy on its part — it follows our instructions to the dot, yet ends up accomplishing something altogether different from what we intend. We have all been bitten in the rear end by this literal adherence to logic at the expense of commonsense. We can attribute at least some of the blame to our lack of understanding (yes, literal and complete understanding) of the paradigms used in computing.

Rich in paradigms, the field of computing has a strong influence in the way we think and view the world. If you don’t believe me, just look at the way we learn things these days. Do we learn anything now, or do we merely learn how to access information through browsing and searching? Even our arithmetic abilities have eroded along with the advent of calculators and spreadsheets. I remember the legends of great minds like Enrico Fermi, who estimated the power output of the first nuclear blast by floating a few pieces of scrap paper, and like Richard Feynman, who beat an abacus expert by doing binomial expansion. I wonder if the Fermis and Feynmans of our age would be able to pull those stunts without pulling out their pocket calculators.

Procedural programming, through its unwarranted reuse of mathematical symbols and patterns, has shaped the way we interact with our computers. The paradigm that has evolved is distinctly unmathematical. Functional programming represents a counter attack, a campaign to win our minds back from the damaging influences of the mathematical monstrosities of procedural languages. The success of this battle may depend more on might and momentum rather than truth and beauty. In our neck of the woods, this statement translates to a simple question: Can we find enough developers who can do functional programming? Or is it cheaper and more efficient to stick to procedural and object oriented methodologies?

Sections

Columns, Quantitative Finance, The Wilmott Magazine

Hedging Dilemma

1 Comment

Ever wonder why those airfares are quick to climb, but slow to land? Well, you can blame the risk managers.

When the oil price hit $147 a barrel in July ’08, with all the pundits predicting sustained $200 levels, what would you have done if you were risk managing an airline’s exposure to fuel? You would have ran and paid an arm and a leg to hedge it. Hedging would essentially fix the price for your company around $150 level, no matter how the market moved. Now you sit back and relax, happy in the knowledge that you saved your firm potentially millions of dollars.

Then, to your horror, the oil price nosedives, and your firm is paying $100 more than it should for each barrel of oil. (Of course, airlines don’t buy WTI, but you know what I mean.) So, thanks to the risk managers’ honest work, airlines (and even countries) are now handing over huge sums of money to energy traders. Would you rather be a trader or a risk manager?

And, yes, the airfares will come down, but not before the risk managers take their due share of flak.

Sections

Quantitative Finance, The Wilmott Magazine, Work and Life

Ambition vs. Greed

4 Comments

Growing up in a place like India, I was told early in life that ambition was a bad thing to have. It had a negative connotation closer to greed than drive in its meaning. I suspect this connotation was rather universal at some point in time. Why else would Mark Anthony harp on Brutus calling Caesar ambitious?

Greed, or its euphemistic twin ambition, probably had some role to play in the pain and suffering of the current financial turmoil and the unfolding economic downturn. But, it is not just the greed of Wall Street. Let’s get real. Jon Steward may poke fun at the twenty something commodity trader earning his thirty million dollar bonus by pushing virtual nothingness around, but nobody complained when they were (or thought they were) making money. Greed is not confined to those who ran fifty billion dollar Ponzi schemes; it is also in those who put their (and other people’s) money in such schemes expecting a too-good-to-be-true rate of returns. They were also made of the sterner stuff.

Let’s be honest about it. We in the financial industry are in the business of making money, for others and for ourselves. We don’t get into this business for philanthropic or spiritual reasons. We get into it because we like the rewards. Because we know that “how to get rich quick” or “how to get even richer” is the easiest sell of all.

We hear a lot about how the CEOs and other fat cats made a lot of money while other normal folks suffered. It is true that the profits were “private” while the losses are public, which is probably why the bailout plan did not get much popular support. But with or without the public support, bailout plan or not, like it or not, the pain is going to be public.

Sure, the CEOs of financial institutions with their private jets and eye-popping bonuses were guilty of ambition, but the fat cats didn’t all work in a bank or a hedge fund. It is the legitimization of greed that fueled this debacle, and nobody is innocent of it.

Sections

Columns, Physics, Quantitative Finance, The Wilmott Magazine

Chaos and Uncertainty

The last couple of months in finance industry can be summarized in two words — chaos and uncertainty. The aptness of this laconic description is all too evident. The sub-prime crisis where everybody lost, the dizzying commodity price movements, the pink slip syndrome, the spectacular bank busts and the gargantuan bail-outs all vouch for it.

The financial meltdown is such a rich topic with reasons and ramifications so overarching that all self-respecting columnists will be remiss to let it slide. After all, a columnist who keeps his opinions to himself is a columnist only in his imagination. I too will share my views on causes and effects of this turmoil that is sure to affect our lives more directly than anybody else’s, but perhaps in a future column.

The chaos and uncertainty I want to talk about are of different kind — the physics kind. The terms chaos and uncertainty have a different and specific meanings in physics. How those meanings apply to the world of finance is what this column is about.

Symmetries and Patterns

Physicists are a strange bunch. They seek and find symmetries and patterns where none exists. I remember once when our brilliant professor, Lee Smolin, described to us how the Earth could be considered a living organism. Using insightful arguments and precisely modulated articulation, Lee made a compelling case that the Earth, in fact, satisfied all the conditions of being an organism. The point in Lee’s view was not so much whether or the Earth was literally alive, but that thinking of it as an organism was a viable intellectual pattern. Once we represent the Earth in that model, we can use the patterns pertaining to organism to draw further predictions or conclusions.

Expanding on this pattern, I recently published a column presenting the global warming as a bout of fever caused by a virus (us humans) on this host organism. Don’t we plunder the raw material of our planet with the same abandon with which a virus usurps the genetic material of its host? In addition to fever, typical viral symptoms include sores and blisters as well. Looking at the cities and other eye sores that have replaced pristine forests and other natural landscapes, it is not hard to imagine that we are indeed inflicting fetid atrocities to our host Earth. Can’t we think of our city sewers and the polluted air as the stinking, oozing ulcers on its body?

While these analogies may sound farfetched, we have imported equally distant ideas from physics to mathematical finance. Why would stock prices behave anything like a random walk, unless we want to take Bush’s words (that “Wall Street got drunk”) literally? But seriously, Brownian motion has been a wildly successful model that we borrowed from physics. Again, once we accept that the pattern is similar between molecules getting bumped around and the equity price movements, the formidable mathematical machinery and physical intuitions available in one phenomenon can be brought to bear on the other.

Looking at the chaotic financial landscape now, I wonder if physics has other insights to offer so that we can duck and dodge as needed in the future. Of the many principles from physics, chaos seems such a natural concept to apply to the current situation. Are there lessons to be learned from chaos and nonlinear dynamics that we can make use of? May be it is Heisenberg’s uncertainty principle that holds new insights.

Perhaps I chose these concepts as a linguistic or emotional response to the baffling problems confronting us now, but let’s look at them any way. It is not like the powers that be have anything better to offer, is it?

Chaos Everywhere

In physics, chaos is generally described as our inability to predict the outcome of experiments with arbitrarily close initial conditions. For instance, try balancing your pencil on its tip. Clearly, you won’t be able to, and the pencil will land on your desktop. Now, note this line along which it falls, and repeat the experiment. Regardless of how closely you match the initial conditions (of how you hold and balance the pencil), the outcome (the line along which it falls) is pretty much random. Although this randomness may look natural to us — after all, we have been trying to balance pencils on their tips ever since we were four, if my son’s endeavours are anything to go by — it is indeed strange that we cannot bring the initial conditions close enough to be confident of the outcome.

Even stranger is the fact that similar randomness shows up in systems that are not quite as physical as pencils or experiments. Take, for instance, the socio-economic phenomenon of globalization, which I can describe as follows, admittedly with an incredible amount of over-simplification. Long time ago, we used to barter agricultural and dairy products with our neighbours — say, a few eggs for a litre (or was it pint?) of milk. Our self-interest ensured a certain level of honesty. We didn’t want to get beaten up for adding white paint to milk, for instance. These days, thanks to globalization, people don’t see their customers. A company buys milk from a farmer, adds god knows what, makes powder and other assorted chemicals in automated factories and ships them to New Zealand and Peru. The absence of a human face in the supply chain and in the flow of money results in increasingly unscrupulous behaviour.

Increasing chaos can be seen in the form of violently fluctuating concentrations of wealth and fortunes, increasing amplitudes and frequency of boom and bust cycles, exponential explosion in technological innovation and adaptation cycles, and the accelerated pace of paradigm shifts across all aspects of our lives.

It is one thing to say that things are getting chaotic, quite another matter to exploit that insight and do anything useful with it. I won’t pretend that I can predict the future even if (rather, especially if) I could. However, let me show you a possible approach using chaos.

One of the classic examples of chaos is the transition from a regular, laminar flow of a fluid to a chaotic, turbulent flow. For instance, when you open a faucet slowly, if you do it carefully, you can have a pretty nice continuous column of water, thicker near the top and stretched thinner near the bottom. The stretching force is gravity, and the cohesive forces are surface tension and inter-molecular forces. As you open the faucet still further, ripples begin to appear on the surface of the column which, at higher rates of flow, rip apart the column into complete chaos.

In a laminar flow, macroscopic forces tend to smooth out microscopic irregularities. Like gravity and surface tension in our faucet example, we have analogues of macroscopic forces in finance. The stretching force is probably greed, and the cohesive ones are efficient markets.

There is a rich mathematical framework available to describe chaos. Using this framework, I suspect one can predict the incidence and intensity of financial turmoils, though not their nature and causes. However, I am not sure such a prediction is useful. Imagine if I wrote two years ago that in 2008, there would be a financial crisis resulting in about one trillion dollar of losses. Even if people believed me, would it have helped?

Usefulness is one thing, but physicists and mathematicians derive pleasure also from useless titbits of knowledge. What is interesting about the faucet-flow example is this: if you follow the progress two water molecules starting off their careers pretty close to each other, in the laminar case, you will find that they end up pretty much next to each other. But once the flow turns turbulent, there is not telling where the molecules will end up. Similarly, in finance, suppose two banks start off roughly from the same position — say Bear Stearns and Lehman. Under normal, laminar conditions, their stock prices would track similar patterns. But during a financial turbulence, they end up in totally different recycle bins of history, as we have seen.

If whole financial institutions are tossed around into uncertain paths during chaotic times, imagine where two roughly similar employees might end up. In other words, don’t feel bad if you get a pink slip. There are forces well beyond your control at play here.

Uncertainty Principle in Quantitative Finance

The Heisenberg uncertainty principle is perhaps the second most popular theme from physics that has captured the public imagination. (The first one, of course, is Einstein’s E = mc2.) It says something seemingly straightforward — you can measure two complementary properties of a system only to a certain precision. For instance, if you try to figure out where an electron is (measure its position, that is) more and more precisely, its speed becomes progressively more uncertain (or, the momentum measurement becomes imprecise).

Quantitative finance has a natural counterpart to the uncertainty principle — risks and rewards. When you try to minimize the risks, the rewards themselves go down. If you hedge out all risks, you get only risk-free returns. Since risk is the same as the uncertainty in rewards, the risk-reward relation is not quite the same as the uncertainty principle (which, as described in the box, deals with complementary variables), but it is close enough to draw some parallels.

To link the quantum uncertainty principle to quantitative finance, let’s look at its interpretation as observation altering results. Does modelling affect how much money we can make out of a product? This is a trick question. The answer might look obvious at first glance. Of course, if we can understand and model a product perfectly, we can price it right and expect to reap healthy rewards. So, sure, modelling affects the risk-reward equation.

But, a model is only as good as its assumptions. And the most basic assumption in any model is that the market is efficient and liquid. The validity of this assumption (or lack thereof) is precisely what precipitated the current financial crisis. If our modelling effort actually changes the underlying assumptions (usually in terms of liquidity or market efficiency), we have to pay close attention to the quant equivalent of the uncertainty principle.

Look at it this way — a pyramid scheme is a perfectly valid money making model, but based on one unfortunate assumption on the infinite number of idiots at the bottom of the pyramid. (Coming to think of it, the underlying assumption in the sub-prime crisis, though more sophisticated, may not have been that different.) Similar pyramid assumptions can be seen in social security schemes, as well. We know that pyramid assumptions are incorrect. But at what point do they become incorrect enough for us to change the model?

There is an even more insidious assumption in using models — that we are the only ones who use them. In order to make a killing in a market, we always have to know a bit more than the rest of them. Once everybody starts using the same model, I think the returns will plummet to risk-free levels. Why else do you think we keep inventing more and more complex exotics?

Summing up…

The current financial crisis has been blamed on many things. One favourite theory has been that it was brought about by the greed in Wall Street — the so-called privatization of profits and socialization of losses. Incentive schemes skewed in such a way as to encourage risk taking and limit risk management must take at least part of the blame. A more tempered view regards the turmoil as a result of a risk management failure or a regulatory failure.

This column presents my personal view that the turmoil is the inevitable consequence of the interplay between opposing forces in financial markets — risk and rewards, speculation and regulation, risk taking and risk management and so on. To the extent that the risk appetite of a financial institute is implemented through a conflict between such opposing forces, these crises cannot be avoided. Worse, the intensity and frequency of similar meltdowns are going to increase as the volume of transactions increases. This is the inescapable conclusion from non-linear dynamics. After all, such turbulence has always existed in the real economy in the form cyclical booms and busts. In free market economies, selfishness and the inherent conflicts between selfish interests provide the stretching and cohesive forces, setting the stage for chaotic turbulence.

Physics has always been a source of talent and ideas for quantitative finance, much like mathematics provides a rich toolkit to physics. In his book, Dreams of a Final Theory, Nobel Prize winning physicist Steven Weinberg marvels at the uncanny ability of mathematics to anticipate physics needs. Similarly, quants may marvel at the ability of physics to come up with phenomena and principles that can be directly applied to our field. To me, it looks like the repertoire of physics holds a few more gems that we can employ and exploit.

Box: Heisenberg’s Uncertainty Principle

Where does this famous principle come from? It is considered a question beyond the realms of physics. Before we can ask the question, we have to examine what the principle really says. Here are a few possible interpretations:

  • Position and momentum of a particle are intrinsically interconnected. As we measure the momentum more accurately, the particle kind of “spreads out,” as George Gamow’s character, Mr. Tompkins, puts it. In other words, it is just one of those things; the way the world works.
  • When we measure the position, we disturb the momentum. Our measurement probes are “too fat,” as it were. As we increase the position accuracy (by shining light of shorter wavelengths, for instance), we disturb the momentum more and more (because shorter wavelength light has higher energy/momentum).
  • Closely related to this interpretation is a view that the uncertainty principle is a perceptual limit.
  • We can also think of the uncertainly principle as a cognitive limit if we consider that a future theory might surpass such limits.

The first view is currently popular and is related to the so-called Copenhagen interpretation of quantum mechanics. Let’s ignore it for it is not too open to discussions.

The second interpretation is generally understood as an experimental difficulty. But if the notion of the experimental setup is expanded to include the inevitable human observer, we arrive at the third view of perceptual limitation. In this view, it is actually possible to “derive” the uncertainty principle, based on how human perception works.

Let’s assume that we are using a beam of light of wavelength lambda to observe the particle. The precision in the position we can hope to achieve is of the order of lambda. In other words, Delta x approx lambda. In quantum mechanics, the momentum of each photon in the light beam is inversely proportional to the wavelength. At least one photon is reflected by the particle so that we can see it. So, by the classical conservation law, the momentum of the particle has to change by at least this amount(approx constant/lambda) from what it was before the measurement. Thus, through perceptual arguments, we get something similar to the Heisenberg uncertainty principle

Delta x.Delta p approx constant

We can make this argument more rigorous, and get an estimate of the value of the constant. The resolution of a microscope is given by the empirical formula 0.61lambda/NA, where NA is the numerical aperture, which has a maximum value of one. Thus, the best spatial resolution is 0.61lambda. Each photon in the light beam has a momentum 2pihbar/lambda, which is the uncertainty in the particle momentum. So we get Delta x.Delta p approx 4hbar, approximately an order of magnitude bigger than the quantum mechanical limit.

Through more rigorous statistical arguments, related to the spatial resolution and the expected momentum transferred, it may possible to derive the Heisenberg uncertainty principle through this line of reasoning.

If we consider the philosophical view that our reality is a cognitive model of our perceptual stimuli (which is the only view that makes sense to me), my fourth interpretation of the uncertainty principle being a cognitive limitation also holds a bit of water.
About the Author

The author is a scientist from the European Organization for Nuclear Research (CERN), who currently works as a senior quantitative professional at Standard Chartered in Singapore. More information about the author can be found at his blog: http//www.Thulasidas.com. The views expressed in this column are only his personal views, which have not been influenced by considerations of the firm’s business or client relationships.

Columns, Quantitative Finance, The Wilmott Magazine

Software Nightmares

To err is human, but to really foul things up, you need a computer. So states the remarkably insightful Murphy’s Law. And nowhere else does this ring truer than in our financial workplace. After all, it is the financial sector that drove the rapid progress in the computing industry — which is why the first computing giant had the word “business” in its name.

The financial industry keeps up with the developments in the computer industry for one simple reason. Stronger computers and smarter programs mean more money — a concept we readily grasp. As we use the latest and greatest in computer technology and pour money into it, we fuel further developments in the computing field. In other words, not only did we start the fire, we actively fan it as well. But it is not a bad fire; the positive feedback loop that we helped set up has served both the industries well.

This inter-dependency, healthy as it is, gives us nightmarish visions of perfect storms and dire consequences. Computers being the perfect tools for completely fouling things up, our troubling nightmares are more justified than we care to admit.

Models vs. Systems

Paraphrasing a deadly argument that some gun aficionados make, I will defend our addiction to information technology. Computers don’t foul things up; people do.

Mind you, I am not implying that we always mess it up when we deploy computers. But at times, we try to massage our existing processes into their computerised counterparts, creating multiple points of failure. The right approach, instead, is often to redesign the processes so that they can take advantage of the technology. But it is easier said than done. To see why, we have to look beyond systems and processes and focus on the human factors.

In a financial institution, we are in the business of making money. We fine-tune our reward structure in such a way that our core business (of making money, that is) runs as smoothly as possible. Smooth operation relies on strict adherence to processes and the underlying policies they implement. In this rigid structure, there is little room for visionary innovation.

This structural lack of incentive to innovate results in staff hurrying through a new system rollout or a process re-engineering. They have neither the luxury of time nor the freedom to slack off in the dreaded “business-as-usual” to do a thorough job of such “non-essential” things.

Besides, there is seldom any unused human resource to deploy in studying and improving processes so that they can better exploit technology. People who do it need to have multi-facetted capabilities (business and computing, for instance). Being costly, they are much more optimally deployed in the core business of making more money.

Think about it, when is the last time you (or someone you know) got hired to revamp a system and the associated processes? The closest you get is when someone is hired to duplicate a system that is already known to work better elsewhere.

The lack of incentive results in a dearth of thought and care invested in the optimal use of technology. Suboptimal systems (which do one thing well at the cost of everything else) abound in our workplace. In time, we will reach a point where we have to bite the bullet and redesign these systems. When redesigning a system, we have to think about all the processes involved. And we have to think about the system while designing or redesigning processes. This cyclic dependence is the theme of this article.

Systems do not figure in a quant’s immediate concern. What concerns us more is our strongest value-add, namely mathematical modelling. In order to come up with an optimal deployment strategy for models, however, we need to pay attention to operational issues like trade workflow.

I was talking to one of our top traders the other day, and he mentioned that a quant, no matter how smart, is useless unless his work can be deployed effectively and in a timely manner. A quant typically delivers his work as a C++ program. In a rapid deployment scenario, his program will have to plug directly into a system that will manage trade booking, risk measurements, operations and settlement. The need for rapid deployment makes it essential for the quants to understand the trade lifecycle and business operations.

Life of a Trade

Once a quant figures out how to price a new product, his work is basically done. After coaxing that stochastic integral into a pricing formula (failing which, a Crank-Nicholson or Monte Carlo), the quant writes up a program and moves on to the next challenge.

It is when the trading desk picks up the pricing spreadsheet and books the first trade into the system that the fun begins. Then the trade takes on a life of its own, sneaking through various departments and systems, showing different strokes to different folks. This adventurous biography of the trade is depicted in Figure 1 in its simplified form.

At the inception stage, a trade is conceptualized by the Front Office folks (sales, structuring, trading desk – shown in yellow ovals in the figure). They study the market need and potential, and assess the trade viability. Once they see and grab a market opportunity, a trade is born.

Fig. 1: Life of a Trade

Even with the best of quant models, a trade cannot be priced without market data, such as prices, volatilities, rates and correlations and so on. The validity of the market data is ensured by Product Control or Market Risk people. The data management group also needs to work closely with Information Technology (IT) to ensure live data feeds.

The trade first goes for a counterparty credit control (the pink bubbles). The credit controllers ask questions like: if we go ahead with the deal, how much will the counterparty end up owing us? Does the counterparty have enough credit left to engage in this deal? Since the credit exposure changes during the life cycle of the trade, this is a minor quant calculation on its own.

In principle, the Front Office can do the deal only after the credit control approves of it. Credit Risk folks use historical data, internal and external credit rating systems, and their own quantitative modelling team to come up with counterparty credit limits and maximum per trade and netted exposures.

Right after the trade is booked, it goes through some control checks by the Middle Office. These fine people verify the trade details, validate the initial pricing, apply some reasonable reserves against the insane profit claims of the Front Office, and come up with a simple yea or nay to the trade as it is booked. If they say yes, the trade is considered validated and active. If not, the trade goes back to the desk for modifications.

After these inception activities, trades go through their daily processing. In addition to the daily (or intra-day) hedge rebalancing in the Front Office, the Market Risk Management folks mark their books to market. They also take care of compliance reporting to regulatory bodies, as well as risk reporting to the upper management — a process that has far-reaching consequences.

The Risk Management folks, whose work is never done as Tracy Chapman would say, also perform scenario, stress-test and historical Value at Risk (VaR) computations. In stress-tests, they apply a drastic market movement of the kind that took place in the past (like the Asian currency crisis or 9/11) to the current market data and estimate the movement in the bank’s book. In historical VaR, they apply the market movements in the immediate past (typically last year) and figure out the 99 percentile (or some such pre-determined number) worst loss scenario. Such analysis is of enormous importance to the senior management and in regulatory and compliance reporting. In Figure 1, the activities of the Risk Management folks are depicted in blue bubbles.

In their attempts to rein in the ebullient traders, the Risk Management folks come across in their adversarial worst. But we have to remind ourselves that the trading and control processes are designed that way. It is the constant conflict between the risk takers (Front Office) and the risk controllers (Risk Management) that implements the risk appetite of the bank as decided by the upper management.

Another group that crunches the trade numbers every day from a slightly different perspective are the Product Control folks, shown in green in Figure 1. They worry about the daily profit and loss (P/L) movements both at trade and portfolio level. They also modulate the profit claims by the Front Office through a reserving mechanism and come up with the so called unrealized P/L.

This P/L, unrealized as it is, has a direct impact on the compensation and incentive structure of Front Office in the short run. Hence the perennial tussle over the reserve levels. In the long term, however, the trade gets settled and the P/L becomes realized and nobody argues over it. Once the trade is in the maturity phase, it is Finance that worries about statistics and cash flows. Their big picture view ends up in annual reports and stake holders meetings, and influences everything from our bonus to the CEO’s new Gulfstream.

Trades are not static entities. During the course of their life, they evolve. Their evolution is typically handled by Middle Office people (grey bubbles) who worry about trade modifications, fixings, knock-ins, knock-outs etc. The exact name given to this business unit (and indeed other units described above) depends on the financial institution we work in, but the trade flow is roughly the same.

The trade flow that I described so far should ring alarm bells in a quant heart. Where are the quants in this value chain? Well, they are hidden in a couple of places. Some of them find home in the Market Risk Management, validating pricing models. Some others may live in Credit Risk, estimating peak exposures, figuring out rating schemes and minimising capital charges.

Most important of all, they find their place before a trade is ever booked. Quants teach their home banks how to price products. A financial institution cannot warehouse the risk associated with a trade unless it knows how much the product in question is worth. It is in this crucial sense that model quants drive the business.

In a financial marketplace that is increasingly hungry for customized structures and solutions, the role of the quants has become almost unbearably vital. Along with the need for innovative models comes the imperative of robust platforms to launch them in a timely fashion to capture transient market opportunities.

In our better investment banks, such platforms are built in-house. This trend towards self-reliance is not hard to understand. If we use a generic trading platform from a vendor, it may work well for established (read vanilla) products. It may handle the established processes (read compliance, reporting, settlements, audit trails etc.) well. But what do we do when we need a hitherto unknown structure priced? We could ask the vendor to develop it. But then, they will take a long time to respond. And, when they finally do, they will sell it to all our competitors, or charge us an arm and a leg for exclusivity thereby eradicating any associated profit potential.

Once a vended solution is off the table, we are left with the more exciting option of developing in-house system. It is when we design an in-house system that we need to appreciate the big picture. We will need to understand the whole trade flow through the different business units and processes as well as the associated trade perspectives.

Trade Perspectives

The perspective that is most common these days is trade-centric. In this view, trades are the primary objects, which is why conventional trading systems keep track of them. Put bunch of trades together, you get a portfolio. Put a few portfolios together, you have a book. The whole Global Markets is merely a collection of books. This paradigm has worked well and is probably the best compromise between different possible views.

But the trade-centric perspective is only a compromise. The activities of the trading floor can be viewed from different angles. Each view has its role in the bigger scheme of things in the bank. Quants, for instance, are model-centric. They try to find commonality between various products in terms of the underlying mathematics. If they can reuse their models from one product to another, potentially across asset classes, they minimize the effort required of them. Remember how Merton views the whole world as options! I listened to him in amazement once when he explained the Asian currency crisis as originating from the risk profile of compound options — the bank guarantees to corporate clients being put options, government guarantees to banks being put options on put options.

Unlike quants who develop pricing models, quantitative developers tend to be product-centric. To them, it doesn’t matter too much even if two different products use very similar models. They may still have to write separate code for them depending on the infrastructure, market data, conventions etc.

Traders see their world from the asset class angle. Typically associated with a particular trading desks based on asset classes, their favourite view cuts across models and products. To traders, all products and models are merely tools to making profit.

IT folks view the trading world from a completely different perspective. Theirs is a system-centric view, where the same product using the same model appearing in two different systems is basically two different beasts. This view is not particularly appreciated by traders, quants or quant developers.

One view that all of us appreciate is the view of the senior management, which is narrowly focussed on the bottom line. The big bosses can prioritise things (whether products, asset classes or systems) in terms of the money they bring to the shareholders. Models and trades are typically not visible from their view — unless, of course, rogue traders lose a lot of money on a particular product or by using a particular model. Or, somewhat less likely, they make huge profits using the same tricks.

When the trade reaches the Market Risk folks, there is a subtle change in the perspective from a trade-level view to a portfolio or book level view. Though mathematically trivial (after all, the difference is only a matter of aggregation), this change has implications in the system design. Trading systems have to maintain a robust hierarchical portfolio structure so that various dicing and slicing as required in the later stages of the trade lifecycle can be handled with natural ease.

The busy folks in the Middle Office (who take care of trade validations and modifications) are obsessed with trade queues. They have a validation queue, market operation queue etc. Again, the management of queues using status flags is something we have to keep in mind while designing an in-house system.

When it comes to Finance and their notions of cost centres, the trade is pretty much out of the booking system. Still, they manage trading desks and asset classes cost centres. Any trading platform we design has to provide adequate hooks in the system to respond to their specific requirements as well.

Quants and the Big Picture

Most quants, especially at junior levels, despise the Big Picture. They think of it as a distraction from their real work of marrying stochastic calculus to C++. Changing that mindset to some degree is the hidden agenda behind this column.

As my trader friends will agree, the best model in the world is worthless unless it can be deployed. Deployment is the fast track to the big picture — no point denying it. Besides, in an increasingly interconnected world where a crazy Frenchman’s actions instantly affect our bonus, what is the use of denying the existence of the big picture in our nook of the woods? Instead, let’s take advantage of the big picture to empower ourselves. Let’s bite the bullet and sit through a “Big Picture 101.”

When we change our narrow, albeit effective, focus on the work at hand to an understanding of our role and value in the organization, we will see the potential points of failure of the systems and processes. We will be prepared with possible solutions to the nightmarish havoc that computerized processes can wreak. And we will sleep easier.

Columns, Quantitative Finance, The Wilmott Magazine

Quant Talent Management

The trouble with quants is that it is hard to keep them anchored to their moorings. Their talent is in high demand for a variety of reasons. The primary reason is the increasing sophistication of the banking clients, who demand increasingly more structured products with specific hedging and speculative motives. Servicing their demand calls for a small army of quants supporting the trading desks and systems.

Since structured products are a major profit engine on the trading floor of most banks, this demand represents a strong pull factor for quants from competing institutions. There is nothing much most financial institutions can do about this pull factor, except to pull them back in with offers they can’t refuse.

But we can try to eliminate the push factors that are hard to identify. These push factors are often hidden in the culture, ethics and the way things get done in institutions. They are, therefore, specific to the geographical location and the social settings where the banks operate.

Performance Appraisal — Who Needs It?

Performance appraisal is a tool for talent retention, if used wisely. But, if misused, it can become a push factor. Are there alternatives that will aid in retaining and promoting talent?

As it stands now, we go through this ordeal of performance appraisal at least once every year. Our career progression, bonus and salary depend on it. So we spend sleepless nights agonizing over it.

In addition to the appraisal, we also get our “key performance indicators” or KPIs for next year. These are the commandments we have to live by for the rest of the year. The whole experience of it is so unpleasant that we say to ourselves that life as an employee sucks.

The bosses fare hardly better though. They have to worry about their own appraisals by bigger bosses. On top of that, they have to craft the KPI commandments for us as well — a job pretty darned difficult to delegate. In all likelihood, they say to themselves that their life as a boss sucks!

Given that nobody is thrilled about the performance appraisal exercise, why do we do it? Who needs it?

The objective behind performance appraisal is noble. It strives to reward good performance and punish poor shows — the old carrot and stick management paradigm. This objective is easily met in a small organization without the need for a formal appraisal process. Small business owners know who to keep and who to sack. But in a big corporate body with thousands of employees, how do you design a fair and consistent compensation scheme?

The solution, of course, is to pay a small fortune to consultants who design appraisal forms and define a uniform process — too uniform, perhaps. Such verbose forms and inflexible processes come with inherent problems. One problem is that the focus shifts from the original objective (carrot and stick) to fairness and consistency (one-size-fits-all). Mind you, most bosses know who to reward and who to admonish. But the HR department wants the bosses to follow a uniform process, thereby increasing everybody’s workload.

Another, more insidious problem with this consultancy driven approach is that it is necessarily geared towards mediocrity. When you design an appraisal process to cater to everybody, the best you can hope to achieve is to improve the average performance level by a bit. Following such a process, the CERN scientist who invented the World Wide Web would have fared badly, for he did not concentrate on his KPIs and wasted all his time thinking about file transfers!

CERN is a place that consistently produces Nobel laureates. How does it do it? Certainly not by following processes that are designed to make incremental improvements at the average level. The trick is to be a center for excellence which attracts geniuses.

Of course, it is not fair to compare an average bank with CERN. But we have to realize that the verbose forms, which focus on averages and promote mediocrity, are a poor tool for innovation management, especially when we are trying to retain and encourage excellence in quant talent.

A viable alternative to standardized and regimented appraisal processes is to align employee objectives with those of the institutions and leave performance and reward management to bosses. With some luck, this approach may retain fringe geniuses and promote innovation. At the very least, it will alleviate some employee anxiety and sleepless nights.

To Know or Not To Know

One peculiar push factor in the Asian context is the lack of respect for technical knowledge. Technical knowledge is not always a good thing in the modern Asian workplace. Unless you are careful, others will take advantage of your expertise and dump their responsibilities on you. You may not mind it as long as they respect your expertise. But, they often hog the credit for your work and present their ability to evade work as people management skills.

People management is better rewarded than technical expertise. This differentiation between experts and middle-level managers in terms of rewards is a local Asian phenomenon. Here, those who present the work seem to get the credit for it, regardless of who actually performs it. We live in a place and time where articulation is often mistaken for accomplishments.

In the West, technical knowledge is more readily recognized than smooth presentations. You don’t have to look beyond Bill Gates to appreciate the heights to which technical expertise can take you in the West. Of course, Gates is more than an expert; he is a leader of great vision as well.

Leaders are different from people managers. Leaders provide inspiration and direction. They are sorely needed in all organizations, big and small.

Unlike people mangers, quants and technical experts are smart cookies. They can easily see that if they want to be people managers, they can get started with a tie and a good haircut. If the pickings are rich, why wouldn’t they?

This Asian differentiation between quants and managers, therefore, makes for a strong push factor for some quants who find it worthwhile to hide their technical skills, get that haircut, grab that tie, and become a people manager. Of course, it comes down to your personal choice between fulfilment and satisfaction originating from technical authority on the one hand, and convenience and promotions arising from people skills on the other.

I wonder whether we have already made our choices, even in our personal lives. We find fathers who cannot get the hang of changing diapers household chores. Is it likely that men cannot figure out washing machines and microwaves although they can operate complicated machinery at work? We also find ladies who cannot balance their accounts and estimate their spending. Is it really a mathematical impairment, or a matter of convenience? At times, the lack of knowledge is as potent a weapon as its abundance.

How Much is Talent Worth?

Banks deal in money. Our profession in finance teaches us that we can put a dollar value to everything in life. Talent retention is no different. After taking care of as much of the push factors as we can, the next question is fairly simple: How much does it take to retain talent?

My city-state of Singapore suffers from a special disadvantage when it comes to talent management. We need foreign talent. It is nothing to feel bad about. It is a statistical fact of life. For every top Singaporean in any field — be it finance, science, medicine, sports or whatever — we will find about 500 professionals of equal calibre in China and India. Not because we are 500 times less talented, just that they have 500 times more people.

Coupled with overwhelming statistical supremacy, certain countries have special superiority in their chosen or accidental specializations. We expect to find more hardware experts in China, more software gurus in India, more badminton players in Indonesia, more entrepreneurial spirit and managerial expertise in the west.

We need such experts, so we hire them. But how much should we pay them? That’s where economics comes in — demand and supply. We offer attractive expatriate packages that the talents would bite.

I was on an expatriate package when I came to Singapore as a foreign talent. It was a fairly generous package, but cleverly worded so that if I became a “local” talent, I would lose out quite a bit. I did become local a few years later, and my compensation diminished as a consequence. My talent did not change, just the label from “foreign” to “local.”

This experience made me think a bit about the value of talent and the value of labels. The local quant talents, too, are beginning to take note of the asymmetric compensation structure associated with labels. This asymmetry and the consequent erosion of loyalty introduce another push factor for the local quant talents, as if one was needed.

The solution to this problem is not a stricter enforcement of the confidentiality of salaries, but a more transparent compensation scheme free of anomalies that can be misconstrued as unfair practices. Otherwise, we may see an increasing number of Asian nationals using Singapore-based banks as a stepping stone to greener pastures. Worse, we may see (as indeed we do, these days) locals seeking level playing fields elsewhere.

We need to hire the much needed talent whatever it costs; but let’s not mistake labels for talent.

Handling Goodbyes

Losing talent is an inevitable part of managing it. What do you do when your key quant hands in the dreaded letter? It is your worst nightmare as a manager! Once the dust settles and the panic subsides, you should ask yourself, what next?

Because of all the pull and push factors discussed so far, quant staff retention is a challenge. New job offers are becoming increasingly more irresistible. At some stage, someone you work closely with — be it your staff, your boss or a fellow team member — is going to say goodbye. Handling resignations with tact and grace is no longer merely a desirable quality, but an essential corporate skill today.

We do have some general strategies to deal with resignations. The first step is to assess the motivation behind the career choice. Is it money? If so, a counter offer is usually successful. Counter offers (both making them and taking them) are considered ineffective and in poor taste. At least, executive search firms insist that they are. But then, they would say that, wouldn’t they?

If the motivation behind the resignation is the nature of the current or future job and its challenges, a lateral movement or reassignment (possibly combined with a counter offer) can be effective. If everything fails, then it is time to bid goodbye — amicably.

It is vitally important to maintain this amicability — a fact often lost on bosses and HR departments. Understandably so because, by the time the counter offer negotiations fail, there is enough bitterness on both sides to sour the relationship. Brush those wounded feelings aside and smile through your pain, for your paths may cross again. You may rehire the same person. Or, you may end up working with him/her on the other side. Salvage whatever little you can for the sake of positive networking.

The level of amicability depends on corporate culture. Some organizations are so cordial with deserting employees that they almost encourage desertion. Others treat the traitors as the army used to — with the help of a firing squad.

Both these extremes come with their associated perils. If you are too cordial, your employees may treat your organization as a stepping stone, concentrating on acquiring only transferable skills. On the other extreme, if you develop a reputation for severe exit barriers in an attempt to discourage potential traitors, you may also find it hard to recruit top talent.

The right approach lies somewhere in between, like most good things in life. It is a cultural choice that an organization has to make. But regardless of where the balance is found, resignation is here to stay, and people will change jobs. Change, as the much overused cliché puts it, is the only constant.

Summing Up…

In a global market that demands ever more customization and structuring, there is an unbearable amount of pull factor for good quants. Quant talent management (acquisition and retention) is almost as challenging as developing quant skills yourself.

While powerless against the pull factor, banks and financial institutions should look into eliminating hidden push factors. Develop respect and appreciation for hard-to-replace talents. Invent innovative performance measurement metrics. Introduce fair and transparent compensation schemes.

When it all fails and the talent you so long to retain leaves, handle it with tact and grace. At some point in the future, you may have to hire them. Or worse, you may want to get hired by them!

Columns, Quantitative Finance, The Wilmott Magazine

Benford and Your Taxes

Nothing is certain but death and taxes, they say. On the death front, we are making some inroads with all our medical marvels, at least in postponing it if not actually avoiding it. But when it comes to taxes, we have no defense other than a bit of creativity in our tax returns.

Let’s say Uncle Sam thinks you owe him $75k. In your honest opinion, the fair figure is about the $50k mark. So you comb through your tax deductible receipts. After countless hours of hard work, fyou bring the number down to, say, $65k. As a quant, you can estimate the probability of an IRS audit. And you can put a number (an expectation value in dollars) to the pain and suffering that can result from it.

Let’s suppose that you calculate the risk of a tax audit to be about 1% and decide that it is worth the risk to get creative in you deduction claims to the tune of $15k. You send in the tax return and sit tight, smug in the knowledge that the odds of your getting audited are fairly slim. You are in for a big surprise. You will get well and truly fooled by randomness, and IRS will almost certainly want to take a closer look at your tax return.

The calculated creativity in tax returns seldom pays off. Your calculations of expected pain and suffering are never consistent with the frequency with which IRS audits you. The probability of an audit is, in fact, much higher if you try to inflate your tax deductions. You can blame Benford for this skew in probability stacked against your favor.

Skepticism

Benford presented something very counter-intuitive in his article [1] in 1938. He asked the question: What is the distribution of the first digits in any numeric, real-life data? At first glance, the answer seems obvious. All digits should have the same probability. Why would there be a preference to any one digit in random data?

figure1
Figure 1. The frequency of occurrence of the first digits in the notional amounts of financial transactions. The purple curve is the predicted distribution. Note that the slight excesses at 1 and 5 above the purple curve are expected because people tend to choose nationals like 1/5/10/50/100 million. The excess at 8 is also expected because it is considered a lucky number in Asia.

Benford showed that the first digit in a “naturally occurring” number is much more likely to be 1 rather than any other digit. In fact, each digit has a specific probability of being in the first position. The digit 1 has the highest probability; the digit 2 is about 40% less likely to be in the first position and so on. The digit 9 has the lowest probability of all; it is about 6 times less likely to be in the first position.

When I first heard of this first digit phenomenon from a well-informed colleague, I thought it was weird. I would have naively expected to see roughly same frequency of occurrence for all digits from 1 to 9. So I collected large amount of financial data, about 65000 numbers (as many as Excel would permit), and looked at the first digit. I found Benford to be absolutely right, as shown in Figure 1.

The probability of the first digit is pretty far from uniform, as Figure 1 shows. The distribution is, in fact, logarithmic. The probability of any digit d is given by log(1 + 1 / d), which is the purple curve in Figure 1.

This skewed distribution is not an anomaly in the data that I happened to look at. It is the rule in any “naturally occurring” data. It is the Benford’s law. Benford collected a large number of naturally occurring data (including population, areas of rivers, physical constants, numbers from newspaper reports and so on) and showed that this empirical law is respected.

Simulation

As a quantitative developer, I tend to simulate things on a computer with the hope that I may be able to see patterns that will help me understand the problem. The first question to be settled in the simulation is to figure out what the probability distribution of a vague quantity like “naturally occurring numbers” would be. Once I have the distribution, I can generate numbers and look at the first digits to see their frequency of occurrence.

To a mathematician or a quant, there is nothing more natural that natural logarithm. So the first candidate distribution for naturally occurring numbers is something like RV exp(RV), where RV is a uniformly distributed random variable (between zero and ten). The rationale behind this choice is an assumption that the number of digits in naturally occurring numbers is uniformly distributed between zero and an upper limit.

Indeed, you can choose other, fancier distributions for naturally occurring numbers. I tried a couple of other candidate distributions using two uniformly distributed (between zero and ten) random variables RV1 and RV2: RV1 exp(RV2) and exp(RV1+RV2). All these distributions turn out to be good guesses for naturally occurring numbers, as illustrated in Figure 2.

figure2
Figure 2. The distribution of the first digits in the simulation of “naturally occurring” numbers, compared to the prediction.

The first digits of the numbers that I generated follow Benford’s law to an uncanny degree of accuracy. Why does this happen? One good thing about computer simulation is that you can dig deeper and look at intermediate results. For instance, in our first simulation with the distribution: RV exp(RV), we can ask the question: What are the values of RV for which we get a certain first digit? The answer is shown in Figure 3a. Note that the ranges in RV that give the first digit 1 are much larger than those that give 9. About six times larger, in fact, as expected. Notice how pattern repeats itself as the simulated natural numbers “roll over” from the first digit of 9 to 1 (as an odometer tripping).

figure3a
Figure 3a. The ranges in a uniformly distributed (between 0 and 10) random variable RV that result in different first digits in RV exp(RV). Note that the first digit of 1 occurs much more frequently than the rest, as expected.

A similar trend can be seen in our fancier simulation with two random variables. The regions in their joint distributions that give rise to various first digits in RV1 exp(RV2) are shown in Figure 3b. Notice the large swathes of deep blue (corresponding to the first digit of 1) and compare their area to the red swathes (for the first digit 9).

figure3b
Figure 3b. The regions in the joint distribution of two uniformly distributed (between 0 and 10) random variables RV1 and RV2 that result in different first digits in RV1 exp(RV2).

This exercise gives me the insight I was hoping to glean from the simulation. The reason for the preponderance of smaller digits in the first position is that the distribution of naturally occurring numbers is usually a tapering one; there is usually an upper limit to the numbers, and as you get closer to the upper limit, the probably density becomes smaller and smaller. As you pass the first digit of 9 and then roll over to 1, suddenly its range becomes much bigger.

While this explanation is satisfying, the surprising fact is that it doesn’t matter how the probability of natural distributions tapers off. It is almost like the central limit theorem. Of course, this little simulation is no rigorous proof. If you are looking for a rigorous proof, you can find it in Hill’s work [3].

Fraud Detection

Although our tax evasion troubles can be attributed to Benford, the first digit phenomenon was originally described in an article by Simon Newcomb [2] in the American Journal of Mathematics in 1881. It was rediscovered by Frank Benford in 1938, to whom all the glory (or the blame, depending on which side of the fence you find yourself) went. In fact, the real culprit behind our tax woes may have been Theodore Hill. He brought the obscure law to the limelight in a series of articles in the 1990s. He even presented a statistical proof [3] for the phenomenon.

In addition to causing our personal tax troubles, Benford’s law can play a crucial role in many other fraud and irregularity checks [4]. For instance, the first digit distribution in the accounting entries of a company may reveal bouts of creativity. Employee reimbursement claims, check amounts, salary figures, grocery prices — everything is subject to Benford’s law. It can even be used to detect market manipulations because the first digits of stock prices, for instance, are supposed to follow the Benford distribution. If they don’t, we have to be wary.

Moral

figure4
Figure 4. The joint distribution of the first and second digits in a simulation, showing correlation effects.

The moral of the story is simple: Don’t get creative in your tax returns. You will get caught. You might think that you can use this Benford distribution to generate a more realistic tax deduction pattern. But this job is harder than it sounds. Although I didn’t mention it, there is a correlation between the digits. The probability of the second digit being 2, for instance, depends on what the first digit is. Look at Figure 4, which shows the correlation structure in one of my simulations.

Besides, the IRS system is likely to be far more sophisticated. For instance, they could be using an advanced data mining or pattern recognition systems such as neural networks or support vector machines. Remember that IRS has labeled data (tax returns of those who unsuccessfully tried to cheat, and those of good citizens) and they can easily train classifier programs to catch budding tax evaders. If they are not using these sophisticated pattern recognition algorithms yet, trust me, they will, after seeing this article. When it comes to taxes, randomness will always fool you because it is stacked against you.

But seriously, Benford’s law is a tool that we have to be aware of. It may come to our aid in unexpected ways when we find ourselves doubting the authenticity of all kinds of numeric data. A check based on the law is easy to implement and hard to circumvent. It is simple and fairly universal. So, let’s not try to beat Benford; let’s join him instead.

References
[1] Benford, F. “The Law of Anomalous Numbers.” Proc. Amer. Phil. Soc. 78, 551-572, 1938.
[2] Newcomb, S. “Note on the Frequency of the Use of Digits in Natural Numbers.” Amer. J. Math. 4, 39-40, 1881.
[3] Hill, T. P. “A Statistical Derivation of the Significant-Digit Law.” Stat. Sci. 10, 354-363, 1996.
[4] Nigrini, M. “I’ve Got Your Number.” J. Accountancy 187, pp. 79-83, May 1999. http://www.aicpa.org/pubs/jofa/may1999/nigrini.htm.

Photo by LendingMemo