Rise of the algorithms

In Robert Harris’s novel The Fear Index a brilliant physicist creates an artificially intelligent trading algorithm that is fabulously successful and ruthlessly smart. Analysing real-time data from all over the internet, it shorts an airline’s stock before one of its planes crashes, deliberately causes the flash crash in New York and even sends a German cannibal to eat its creator. In other words, it is the ultimate trader.

The algorithm, Vixal-4, doesn’t seem like much of a fiction after visiting Progress Software’s convention last month in Boston — aptly named Progress Revolution. For the past 10 years, hedge funds and proprietary traders have been using the company’s Apama platform to build high-frequency algorithmic trading applications that do something very similar to Harris’s creation: detect sophisticated real-time patterns and turn them into trading signals.

The fictional Vixal does this across the whole of the internet, scanning blogs and social media for scraps of information that will help to predict the direction of the market, and learning all the time. Needless to say, it soon realises that underemployed PhDs and overpaid sales guys are a major drag on its P&L and sets about killing them off, starting at the top. Now that’s what I call alpha.

So far, few real-world trading algos have shown signs of wanting to become evil silicone overlords, but even a simple typo in a piece of code can cause havoc these days — and the opportunity for such problems is growing as venues and algos proliferate. Fragmentation also makes it easier to hide manipulation.

Regulators seem woefully resourced to cope with the challenges, but John Bates, Progress’s chief technology officer, is confident that his company can provide the tools they need to tap into multiple markets and monitor them in real time, as well as putting in place safeguards to make sure that algorithms are behaving within their normal parameters. As a result, he is strongly opposed to knee-jerk regulation.

“Firms themselves have got to adopt best practices, but I don’t think there’s any need to ban or restrict algorithmic or high-frequency trading,” he said during an interview on the sidelines of the conference. “Rather than fearing this stuff and banning it, let’s have free, open markets and police them really well. The technology exists.”

The US Commodity Futures Trading Commission (CFTC) could be moving in that direction, said Bates, with the new swap execution facilities being introduced as part of the Dodd-Frank financial reform package. And he should know. Bates sits on the CFTC’s technology advisory commission and recently joined the UK Foresight Committee, a government-sponsored body looking into the future of high-frequency and algorithmic trading. In 2008, Institutional Investor named him one of the 30 most influential people in finance.

Another influential speaker brought up the threat posed by terrorists. Rudy Giuliani, the former mayor of New York and US presidential candidate, mentioned that the New York Stock Exchange was considered the number one target for terrorism before the attacks of September 11 2001. In the intervening 10 years, Bates says that the terrorist threat to financial markets has moved from the physical to the virtual. Policing, meanwhile, has not.

“You could do much more damage moving the market through manipulation,” he said. “We’ve seen that one mis-parameterised algorithm in the right market can cause a flash crash, so you can maybe do bigger things than that if you were trying. It does concern me because we’re all so reliant on the underlying capital markets for so much.”

Clearly, there are many reasons for regulators to up their game when it comes to market surveillance.

Like Vixal, today’s event-processing solutions can also do much more than just analyse financial markets.

Businesses, governments and individuals all leave a rich trail of data as they go about their business, and making sense of it all in real-time can be like trying to drink from a fire hose — which gives rise to opportunities for companies that can use technology to process that data to help them become more responsive, according to Bates.

Event-processing technology is already being used in a number of industries to help save costs, increase sales and improve service, often all at once.

In retail, for example, Bates envisages a virtual pricing model that would customise deals for each individual, “like a vendor at the souk”. When someone buys an iPad, his telecom provider or credit card issuer could determine his propensity to buy a case as well, or to go for lunch nearby, and propose a deal. They could flash time-limited offers as you walk by certain stores or entice you out of the office with happy-hour drinks deals.

Indeed, much of this is already happening, and not just in the US. Progress’s software solutions can typically plug into existing systems and analyse their data streams without the wholesale rip-and-replace that typically comes with major IT decisions, which makes it much easier for companies to implement.

Cellphone users in Turkey, for example, will soon be getting bugged with live spam (technically, “real-time situation-based promotions”) thanks to a partnership between Progress and Turkcell. Banks in Asia are also in discussions with Progress, and other customers include shipping firms and airlines, which use Apama solutions to optimise complex operations.

Military buyers are also said to be interested for similar reasons — a fact that could have added spice to Harris’s bestseller, perhaps in the form of an egomaniacal, out-of-control algo on a Nimitz-class aircraft carrier.

But Bates is sure about one thing; the best way to police these new threats (real and imagined) will be with technology rather than inflexible and unresponsive regulation. That will be good for traders — or, at least, for those that aren’t replaced by machines.