On this page · 6 sections
Starting at about 8:01 on the morning of Wednesday, August 1, 2012, an internal system at Knight Capital began sending automated emails to a group of employees. Each one reported an error, labeled "Power Peg disabled." By the time the stock market opened at 9:30, 97 of them had gone out.
The messages weren't designed to work as alarms, and the people who got them generally didn't read them.
Over the next 45 minutes, Knight's computers fired millions of orders into the market. They were trying to fill just 212 ordinary customer orders. When the firm finally stopped the flood, it had bought and sold shares in 154 stocks, more than 397 million shares in all, and was sitting on billions of dollars of stock that nobody had asked it to buy. Selling that pile off left Knight with a loss of more than $460 million, the figure in the federal government's own investigation. The company's first estimate, put out the next morning, was about $440 million.
Knight wasn't a fly-by-night shop. In 2011 and 2012 it handled roughly a tenth of all trading in US-listed stocks. Within a week, it was fighting for its life.
A middleman with a very fast router
Most people haven't heard of a firm like Knight, and that's fine. It was a middleman. Other brokerage firms sent it their customers' orders, and Knight did the buying and selling, sometimes with its own money on the line. It was based in Jersey City, New Jersey, and it had built its business on technology.
Part of that technology was a piece of software called SMARS, which the government's investigators describe as an automated, high-speed order router. Here's why a router is needed. When you tell a brokerage app to buy 100 shares of a company, your order doesn't always get filled in one piece. Shares are for sale in several places at once, so the software cuts your order into smaller pieces and sends each piece to wherever it can be filled. The big original order is called the "parent." The little pieces are "children." When enough children have been filled to add up to the parent, the software is supposed to stop.
That last part is the whole story.
Years earlier, Knight had used a feature inside SMARS called Power Peg. It kept a running count of how many shares of a parent order had already been bought or sold, and it used that count to know when to quit sending children. Knight stopped using Power Peg in 2003. It didn't delete the code. The code just sat there on Knight's servers (the computers that run software for everyone else), switched off, still there and still able to be called.
In 2005, Knight moved the part that does the counting to a different spot earlier in the program. Knight didn't retest Power Peg afterward to see whether it still worked with the counter moved. Picture a vending machine that's supposed to stop dispensing once it has handed over what you paid for. Someone moves the part that does the counting, and nobody checks that the machine still stops. As long as no one presses the old button, it doesn't matter.
That's where things stood for seven years.
One of eight
The old button got a new reason to be pressed in the summer of 2012. The New York Stock Exchange was about to launch something called the Retail Liquidity Program, set to begin on August 1. It was a new way for orders from everyday investors to be handled, and Knight wanted its customers to be able to use it. That meant new code inside SMARS.
The plan was to put the new code where Power Peg used to be and delete Power Peg for good. There was a catch. The new code reused a setting that used to switch Power Peg on. Think of it as a marker attached to each order that tells the software which way to handle it. In the new version, that marker would wake up the new Retail Liquidity Program code. In the old version, on any computer that still had it, the very same marker would wake up Power Peg.
So the whole plan depended on every computer getting the new version.
SMARS ran on eight servers. Starting July 27, Knight rolled the new code out in stages, a few servers at a time on successive days. At some point in that process, one technician didn't copy the new code to one of the eight. Nobody caught it. Knight had no second person reviewing the rollout, and no written procedure that required one. Nobody at the firm realized that server still had Power Peg on it and hadn't received the new code.
Seven of the eight servers were fine. The eighth was still running the old code.
Forty-five minutes
When the market opened on August 1, orders carrying the reused marker started arriving. The seven updated servers handled them correctly. Any order with the marker that landed on the eighth woke up Power Peg, which sent out child orders in a rapid-fire sequence without regard to how many shares had already been bought or sold. Another part of Knight's system actually knew the parent orders had been filled. It just didn't pass that along to SMARS. The router kept going because nothing told it to stop.
The SEC's order finds that from 212 incoming orders, SMARS produced millions of child orders and 4 million executions. An execution is a trade that actually went through. Over roughly 45 minutes, that works out to something like 1,500 completed trades every second, on average, and no customer had asked for any of them.
Outside the firm, what came next looked like weather. A rush of unusually heavy trading tore through dozens of stocks on the New York Stock Exchange in early trading, and Knight told its clients to send their orders to other firms. The exchange began reviewing trades in 148 stock symbols made between 9:30 and 10:15. A handful of stocks were halted. By the investigators' count, in 75 of the stocks Knight's trades made up more than a fifth of all trading and helped push prices up or down by more than 5 percent. In 37 of those, Knight accounted for more than half of all trading, and prices moved by more than 10 percent.
Traders described wild swings in prices. Phil Orlando, chief equity strategist at the investment firm Federated Investors, put the mood on his side of the market in two words: "We're panicked."
The stocks caught up in it included household names such as Ford, RadioShack and American Airlines. In the end, the exchange canceled trades in just six stocks, the ones where deals had been done at prices 30 percent or more away from where they'd opened that day. Everything else stood, which meant Knight had to live with the trades it had made.
Inside the firm, people noticed. At the open, senior staff could see an unusually large pile of positions building up in an internal account known as the 33 Account, and they couldn't quickly tell where it was coming from. The account temporarily held several kinds of positions, including shares from trades the system couldn't match to an order, and it had a limit of $2 million. That limit wasn't linked to any automatic control on the firm's overall exposure, so nothing in the system stopped orders on its own. The main tool for watching the firm's risk relied entirely on a human being to look at it and know the limits by heart. It also lagged during heavy trading, so its reports were inaccurate.
Then came the attempt to fix it. In one try, Knight took the new code off the seven servers where it had been installed correctly. That made things worse. With the new code gone, those seven servers were now running the same old Power Peg that had been waking up on the eighth, and more incoming orders began triggering it. Knight had no supervisory procedures for responding to an incident like this one, and the SEC later found Knight needed clear guidance for its technology staff on when to disconnect a malfunctioning system from the market.
By the time the orders stopped, Knight had ended up holding a net long position of about $3.5 billion in 80 stocks, meaning it owned that much more than it had sold. In another 74 stocks it was on the other side, short about $3.15 billion, meaning it had sold that much more than it owned. It had to unwind all of it. If you spread the final loss evenly across those 45 minutes, it comes to more than $10 million a minute, though the actual loss was locked in as the firm sold off the positions it hadn't wanted.
The weekend nobody wanted
On Thursday morning, Knight announced that the trouble had come from its installation of trading software, that the software had been removed, that its clients hadn't been negatively affected by the bad orders, and that the damage was about $440 million. The announcement also admitted the company's capital base had been "severely impacted," and that it was looking for financing. Its shares, above $10 before the mishap, were by then around $3, down about half on the day.
Over the following days, the story moved from the trading floor to conference rooms. Knight's chief executive, Thomas Joyce, had just had knee surgery. He went into the office on crutches. The New York Stock Exchange temporarily moved Knight's market-making duties in more than 600 securities to Getco, a trading firm that also ended up investing in the rescue. By Sunday, restructuring lawyers were reportedly preparing a possible bankruptcy filing, working on it until about 4 a.m. Monday.
A rescue came together in pieces. Jefferies, the investment bank, put together a group of investors. Over the weekend, the group included Blackstone, Getco, TD Ameritrade, Stifel and Stephens. In Washington, senior SEC officials worked through the night monitoring Knight's finances in offices without air-conditioning, because the building's cooling had been programmed to shut off before midnight to save energy. Joyce went home at about 9 p.m. Sunday while lawyers reached a deal over Chinese food.
On Monday, August 6, Knight announced $400 million in new financing. Jefferies put in $125 million of it. The investors got special shares that could be converted into ordinary shares at $1.50 each, which added up to about 73 percent of the company, and three new seats on its board. Anyone who owned Knight shares before the mishap now held a much smaller slice of the company. Its stock closed that Monday down 24.2 percent, at $3.07.
The company's announcement thanked clients, employees and partners for their "steadfastness during a brief yet difficult period" and promised Knight was "getting back to business as usual."
In July 2013, Knight's parent company combined with GETCO Holding Company, another trading firm, to form a new company called KCG Holdings.
What the government found
On October 16, 2013, about 14 months later, the Securities and Exchange Commission, the agency that polices the US stock market, announced what it had found. Knight agreed to pay a $12 million penalty to settle charges that it had broken a rule requiring brokers to put safeguards on their access to the market, including safeguards against bad orders from their own automated systems. The rule, known as the market access rule, had been adopted in 2010. This was the first time the SEC had brought a case under it.
Knight agreed to the order without admitting or denying what the SEC found. It was censured, told to stop breaking the rules involved, and required to hire an independent consultant to go through its software development and its risk controls. The order also found that Knight had mislabeled some of its sell orders from August 1, in violation of separate rules about short sales.
The order reads as a list of missing safety nets. Knight had no check to compare the orders coming out of SMARS against the orders going in. It had no procedures to halt SMARS when it behaved oddly. The $2 million limit on the 33 Account wasn't linked to an automatic control. It had no written procedures for rolling out SMARS code, no second-person review of a rollout, and no protocol for testing unused code left on its live servers. And the 97 emails that went out before the open had pointed straight at the problem, at a moment when there was time to fix it. The order doesn't name the technician who missed the server.
The loss figure moved a little as the accounting settled. The SEC puts it at more than $460 million. In its year-end financial statements, Knight reported a $457.6 million trading loss from the incident, plus $7.0 million in related legal and professional fees.
One line in the SEC's order, buried in the middle of the findings, points at how small the fix might have been. A written procedure requiring a simple double-check of the new code's rollout, the investigators wrote, "could have identified that a server had been missed and averted the events of August 1."
References
- SEC, In the Matter of Knight Capital Americas LLC, Order Instituting Administrative and Cease-and-Desist Proceedings, Release No. 70694, Oct 16, 2013: https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf
- SEC press release 2013-222, "SEC Charges Knight Capital With Violations of Market Access Rule," Oct 16, 2013: https://www.sec.gov/newsroom/press-releases/2013-222
- Knight Capital Group press release, "Knight Capital Group Provides Update Regarding August 1st Disruption To Routing In NYSE-listed Securities," Aug 2, 2012: https://www.sec.gov/Archives/edgar/data/1060749/000119312512332176/d391111dex991.htm
- Knight Capital Group press release, "Knight Capital Group Completes $400 Million Equity Financing Agreement," Aug 6, 2012: https://www.sec.gov/Archives/edgar/data/1060749/000119312512338098/d392396dex991.htm
- Knight Capital Group, Form 8-K, Aug 2012 (pre-tax loss and the Aug 6 investor agreement): https://www.sec.gov/Archives/edgar/data/1060749/000119312512336167/d392288d8k.htm
- Knight Capital Group annual report note 3, "August 1, 2012 Technology Issue" (trading loss of $457.6 million and $7.0 million in related fees): https://www.sec.gov/Archives/edgar/data/1060749/000119312513087636/R12.htm
- Ben Protess and Nathaniel Popper, "Quick Lunge for a Lifeline Helped Knight Capital Skirt Collapse," The New York Times (DealBook), Aug 6, 2012: https://dealbook.nytimes.com/2012/08/06/knight-capital-skirts-collapse-as-investors-offer-lifeline/
- Reuters, "Knight Capital gets $400 million rescue, shares tumble," Aug 6, 2012: https://jp.reuters.com/article/world/knight-capital-gets-400-million-rescue-shares-tumble-idUSBRE8750HT/
- MarketWatch, "Some U.S. stock market trading disrupted," Aug 1, 2012: http://www.marketwatch.com/story/some-us-stock-market-trading-disrupted-2012-08-01
- BBC News, "NYSE looks into 'irregular trading' in 140 stocks," Aug 1, 2012: https://www.bbc.com/news/business-19087748
- Roland Jones, NBC News, "Knight Trading: Glitch cost us $440 million," Aug 2, 2012: https://www.nbcnews.com/business/markets/knight-trading-glitch-cost-us-440-million-flna919627
- Max Viscio, CNBC, "Knight Glitches Latest in String of Trading Snafus," Aug 1, 2012: https://www.cnbc.com/2012/08/01/knight-glitches-latest-in-string-of-trading-snafus.html