Free Site Registration

Re-Engineering J.P. Morgan

Why High-Performance Computing Can Be A Bear

Why High-Performance Computing Can Be A Bear


February 7, 2012
By Tom Steinert-Threlkeld

 J.P. Morgan had a product up its sleeve for its broker-dealers on August 4.

That was the date it put the servicing of more than one million accounts onto a new broker-dealer platform, known as Morgan Communications or MORCOM. The online system  was to serve both retail and institutional brokers.

The Web 2.0 initiative is part of one of four main thrusts of a half-billion-dollar Strategic Reengineering Program that, in effect, got its start with the May 2008 completion of the acquisition of Bear Stearns, the collapsed Wall Street investment bank.

This is how the acquisition of Bear in a fire sale at the height of the credit crisis led to a re-engineering of the technical operations of J.P. Morgan, the investment banking business of JPMorgan Chase, the nation’s second-largest banking firm.

Bear Stearns had a retail broker workstation that served its correspondents’ clearance business as well as brokers that served high net worth individuals. J.P. Morgan had spent a couple years and $100 million enhancing that workstation and Web platform to make it available to Chase’s retail brokers, as well.

The trick up its sleeve had better be an ace, for J.P. Morgan. The launch was coming at the height of the U.S. Congress’ debate on the national debt and budget deficit. Greece was running out of cash, stoking fears of a breakdown in Europe’s efforts to stave off a debt crisis that would ripple through the continent.

Then, after markets closed on Friday August 5, Standard & Poor’s lowered its credit rating on long-term debt of the United States to 'AA+' from 'AAA.'

In the first full week of operation of this Private Client Work Station Strategy, the Dow Jones Industrial Average moved at least 400 points on four consecutive days, for the first time in history.

“The preparation behind the early August migration began three years ago,” said Joe Triarsi, global head of Broker-Dealer Services at J.P. Morgan.  “With a number of dress rehearsals, extensive integration and performance testing, we were able to successfully transfer client accounts on time without disruption.” 

The new system for serving its clients held up. “Talk about intestinal fortitude and a confidence you can deliver things,’’ said Peter Cherasia, Global Head of Markets Strategies for J.P. Morgan, a former Bear technologist now responsible for Markets Strategies which incorporates front office technology, e-commerce, quantitative analytics and strategic investments.  “We delivered.”



The broker-dealer platform is just one of 24 work streams in the four-pillar strategic re-engineering program (SRP) that has been in the works at J.P. Morgan in the past two-and-a-half years.

The bottoms-up and top-down review compared all existing systems side-by-side and included functional, architectural and business reviews.

The four pillars include:

Derivatives Trading Platform Program: A rationalization of core trading platforms.

Back Office Reengineering Program: Improving front-to-back office systems.

Derivatives Program: Upgrading derivatives clearing operations and achieving ‘straight-through processing,’ also in front-to-back systems.

Critical Platforms Reengineering Program: Initiatives to lower overall expenses and increase the speed of bringing new services to market.

That all four might be needed was almost inevitable.

When it took over Bear in a fire sale that foreshadowed the worldwide credit crisis that would erupt in September 2008, J.P. Morgan had to write 400 programs just to make sure its own systems could interact with Bear Stearns’ trading, clearance and settlement systems.

The combined company didn’t need to run two clearance and settlement platforms for stock transactions, for instance. And it certainly didn’t need 14 different platforms for trading derivative securities, from futures and options to currency swaps.

Plus, it needed to upgrade its order management system, in the process. At the time – the second half of 2008 – J.P. Morgan’s system could only handle 40 orders a second. The state of the art? More than 3,000 a second.

In 2008, chief administrative officer for the Investment Bank and former chief financial officer, Paul Compton, and Cherasia went to chief administrative officer of JPMorgan Chase Frank Bisignano and chairman Jamie Dimon and asked for $50 million over two years to re-architect the company’s main trading systems.

The result was a “detailed yet very easy conversation around what the right thing to do was for the firm,’’ said Cherasia. Two meetings later, the Strategic Reengineering Program got its start.



The first part was to rationalize what was called the Equities Core Infrastructure.

That involved taking the nearly 400 different market data and transaction feeds and connections between the systems they served to make sure that details of every stock order and stock trade went through one large set of pipes. That was “so that we can dip into that order flow, internalize liquidity and provide price improvements to our clients,’’ Cherasia said.

Creating one big stream of liquidity from all parts of the company would let J.P. Morgan compress the process of allocating shares of stock to different accounts, achieve speedier and less costly settlements, minimize “breaks” in transactions and the reconciliation of and details and, in general, consolidate operating expenses.

This was no small matter.

In 2008, JP Morgan Chase, overall, had spent $4.3 billion on technology, communications and equipment. Adding and integrating Bear Stearns’ technical infrastructure could escalate that, potentially dramatically.

By the end of 2009, “we had taken a little bit more than a billion dollars out of that” spend, said Cherasia, with the re-engineering and consolidation project.

The overall JP Morgan Chase spend in 2009: $4.6 billion. And in 2010? $4.7 billion.

The strategic reengineering program began in earnest with 14 different trading platforms for derivatives. Four systems were eliminated right off the bat. But the rest of the plan for streamlining systems did not get finished until the tail end of 2009 – and got formalized with the two-meeting launch of the strategic re-engineering program authorized by Bisignano and Dimon in November 2009.

Another five systems were cut out in 2010 and 2011. The firm’s target is two platforms: one for fixed-income derivatives and one for equities derivatives.

For now, “we are comfortable with running five in the interim that are well coordinated and integrated,’’ Cherasia said.

Even with this rationalization, the company did not have a single, unified system for processing transactions.

“We had a collection of different clearance and settlement systems,’’ said Cherasia. “Some were owned by ‘back office technology.’ Some were owned by the businesses. Some were shared services.”

Complicating matters, J.P. Morgan’s investor services businesses were fairly independent. Every business had the ability to trade every product, any product. 

Equity derivatives might be used in trades for swaps. Credit products might be used in customized ways to provide hedges for clients. That created tremendous complexity in the middle office.  That, in effect, meant the company had to have an equity derivatives middle office, a credit derivatives middle office and a foreign exchange middle office – for each business.

So Dimon initiated a plan that created a single desk in each asset class that would act as the primary market maker.  Any desk that wanted to trade that product had to face the primary market-making desk.  That desk could take the other side of the trade. Or that desk could pass it on to the Street. By the time the new hierarchy was complete, trades between desks could be handled in an automated fashion. In limited cases, brokers could handle some trades on their own. But, by and large, everything went to the primary market makers.

And the middle offices got reorganized. General ledger entries and postings were handled by a single core processing platform for derivatives, a single platform for currencies, a single platform for credit instruments, a single platform for stocks.

Clearing and settlement activities, in addition, got consolidated onto one platform. That saved $30 million a year. And capacity went up ten-fold.

All in all, 24 streams of work got started at the end of 2009, covering changes in technology and operations that would affect not just trading, clearing and settlement, but human resources, legal affairs, compliance, risk management, finance, tax services, auditing, real estate, regional operations and the like.

Twelve streams of work are completed and 12 are still in progress.

The derivatives core processing platform overhaul, for instance, is midway through its program and is scheduled to be completed by the end of 2013. 



It’s not that easy. Creating – or changing over – always-on, high-performance systems is hard, Cherasia notes.

Look at September 2008. J.P. Morgan was already in the midst of moving 600,000 derivatives positions over from Bear Stearns – when Lehman Brothers went out of business. “Lehman was interconnected through their derivatives contracts and collateral to everybody on the Street,’’ said Cherasia.

Lehman filed for bankruptcy on September 15 of that year. That was two days after the second most destructive hurricane to ever make landfall, Hurricane Ike, hit Houston, Texas.

That happened to be where J.P. Morgan had a major development center. Windows were broken throughout the company’s 75-story tower, downtown. Nine hundred technology and operations staff members were displaced. So were 800 other staffers covering 600 lines of investment banking business.  “It was a nightmare,’’ said Cherasia. With staff already displaced, “Lehman was an absolute stop dead for the derivatives position migration.’’

But ultimately about 2 million derivatives positions got winnowed to 1 million, through the Strategic Re-Engineering Project.

All told, the three-year project is expected to generate $300 million a year in savings to the bank and $1 billion of business benefits.

Plus, it’s allowed Cherasia and his team to start thinking that maybe they, and not Goldman Sachs, are becoming the best technology and operations team in capital markets.

Among the core developments of its re-engineering event are its derivatives and securities trading platform, known as Athena.

The system relies on a classic form of computer science programming known as an “acyclic dependency graph” that interconnects risk calculations, securities prices, durations of prices, yields, cash flows and reference data.

What the graph allows Athena to do is determine at every moment just what “leaf in the tree” has to be recalculated. This makes for what Cherasia calls “lazy recalcs.”

“Things only need to be recalculated when you need to recalculate them,’’ he said. Processing cycles are saved.

The technology team also has built its platform out of Python, a general-purpose, high-level programming language whose design philosophy emphasizes code readability. The reference implementation of Python, known as CPython, is free and open source software.

This allows J.P. Morgan to recruit talent already familiar with its basic programming language. That “allows our quants to come in and build their models directly on the platform, very quickly,’’ Cherasia said, “which allows us to leverage those products broadly across the platform.”

The code that gets produced, in turn, is all connected to an object-oriented database that is replicated in real time across every one of its major trading centers, in what Cherasia calls “ring fashion.”

This lets quants and technologists, he said, to work directly with traders to solve problems – and deliver updates every day.



In addition to the re-engineering project, J.P. Morgan is pursuing high-performance computing, to reduce cost and speed of bring services to market, while improving performance.

This includes ‘’virtual server’’ farms and the use of advanced chip technologies, including field programmable gate arrays (FPGAs) and graphics processing units (GPUs).  This improves the speed and efficiency of running quantitative models.

J.P. Morgan has accelerated much of the work that ends up on its “virtual servers” through the use of the graphical processing units. These are the kinds of computer chips that first got widely used in video games, where speed and complexity, as well as constantly changing colors, backgrounds and interactions are paramount.

Such chips modify the style of the processor to meet the style of the code, Cherasia notes.

“We started our work with GPU’s and we were able to basically drop the cost of our main equity derivative trading system in half,’’ he said. And increase computational speed more than 30 percent.

FPGAs, in turn, helped J.P. Morgan to rework how it computes complex risk scenarios for exotic and hybrids books that spanned more than one asset class. FPGAs give the programmer the ability to change, then fine-tune the computer to fit the algorithm. When applied to hybrid credit derivatives models, this reduced the compute time for risk by a factor of more than 200 times and the end-to-end system time by a factor of more than 130 times. In the case of complex foreign exchange models the latest improvements are 260 times and 50 times, respectively. Such chips modify the style of the processor to meet the style of the code, Cherasia notes.

You start with one conceptual break. That leads to another that leads to exploration, some failures and some success.

But, in the end, “success breeds more success and leads you to the next thing,’’ he said. And makes not only for a “better operating environment,’’ but “continuing opportunities to set a business apart, by investing in technology.”                                      STM




DOWNLOAD full-color PDF of FIELD REPORT here