Skip to content
Menu
Finabase Learn

How Wall Street really works Lesson 13 of 14

Hedge funds and AI · Quants and high-frequency trading

Knight Capital 2012: the 45-minute glitch that lost $460 million

How a missed server at Knight Capital sent millions of orders on August 1, 2012, the $460 million loss and rescue, the SEC’s market access case and what it changed.

On this page
  1. The setting
  2. What went wrong
  3. Forty-five minutes
  4. The loss and the rescue
  5. The SEC case
  6. What it changed
  7. Lessons for investors
  8. How Finabase does it
  9. Frequently asked questions
  10. Summary
  11. Check your understanding
  12. Exercise

On the morning of August 1, 2012, Knight Capital was one of the largest traders of US stocks. Its aggregate trading, for itself and its customers, was about a tenth of all trading in listed US equities. Within the first hour of trading it had bought and sold so many shares by mistake that it was fighting for survival. In roughly 45 minutes after the opening bell, one of its order routers sent millions of orders into the market, received more than 4 million executions in 154 stocks and left the firm holding positions worth billions of dollars that nobody had meant to buy or sell. Knight lost more than $460 million.

No market move caused the loss, and no trader made a bad bet. A routine software release had missed one of eight servers, and an old, unused piece of code on that server woke up. This lesson traces what happened from the SEC's own findings, how the firm was rescued within a week, why the SEC charged it under the Market Access Rule, and what the episode taught about controls on automated trading.

The setting

Knight Capital Americas was a broker-dealer based in Jersey City, New Jersey, and a big market maker in US stocks. Much of its business was filling retail orders sent by other brokers. One of its systems, SMARS, was a high-speed algorithmic router. It took a "parent" order from Knight's other systems and, depending on available liquidity, sent one or more "child" orders to outside venues until the parent was filled. SMARS alone generally accounted for 1% or more of all trading in listed US stocks.

The New York Stock Exchange was launching a Retail Liquidity Program for retail orders on August 1, 2012, a 12-month pilot the SEC had approved that July with an exemption from the sub-penny rule. To take part, Knight changed its order handling, including new code in SMARS.

What went wrong

The SEC's 2013 order lays out the chain of events.

  1. Old code was never removed. SMARS still contained a function called Power Peg that Knight had stopped using in 2003. It was dead code, but still present and callable.
  2. A 2005 change broke it silently. In 2005 Knight moved the part of SMARS that counted how many shares of a parent order had been filled, the check that told Power Peg to stop. Power Peg was never retested after the move.
  3. A flag was reused. The new Retail Liquidity Program code reused the flag that had once switched on Power Peg. The plan was to delete Power Peg so that the flag would now trigger the new logic.
  4. One server was missed. Starting on July 27, 2012, Knight installed the new code server by server. A technician did not copy it to one of the eight SMARS servers, no second person reviewed the deployment, and Knight had no written procedure requiring such a review.
One missed server: how the August 1, 2012 failure started Server 1 new RLP code Server 2 new RLP code Server 3 new RLP code Server 4 new RLP code Server 5 new RLP code Server 6 new RLP code Server 7 new RLP code Server 8 old code: Power Peg still live An order carrying the repurposed flag reaches one of the eight SMARS servers Servers 1–7: the flag switches on the new Retail Liquidity Program logic. Orders behave. Server 8: the same flag switches on Power Peg, unused since 2003. Its fill counter had been moved in 2005, so it never sees that the parent order is filled and keeps sending child orders. While staff searched for the cause, the new code was removed from servers 1–7; they then did the same. Source: SEC order 34-70694 (October 16, 2013), paragraphs 13–16 and 27.
How the failure started, as described in the SEC order of October 16, 2013. Server numbering is illustrative; the order says only that one of eight servers was missed.

On August 1, orders carrying the reused flag reached all eight servers. Seven handled them correctly. On the eighth, the flag woke Power Peg, which sent child orders for each parent order and, because its fill counter had been moved, never noticed when the parent was complete. It kept sending. Other parts of Knight's system knew the parent orders were filled; that information did not reach SMARS.

There had been a warning. From 8:01 a.m., an internal system sent Knight staff automated emails that named SMARS and an error described as "Power Peg disabled." Ninety-seven of them arrived before the 9:30 a.m. open. They were not designed as alerts and nobody acted on them.

Knight Capital, July 27, 2012 to October 16, 2013 Jul 27, 2012 New code for NYSE’s Retail Liquidity Program deployed to SMARS servers in stages; one of eight servers is missed Aug 1, 8:01 a.m. Automated “BNET reject” emails mention “Power Peg disabled”; 97 of them arrive before the open, none is acted on 9:30 a.m. Market opens; the eighth server starts sending child orders for 212 parent orders without stopping ~45 minutes Over 4 million executions, 397 million shares, 154 stocks; $3.5 billion net long and $3.15 billion net short Aug 2, 2012 Knight reports a realized pre-tax loss of about $440 million Aug 6, 2012 $400 million of preferred stock sold to six firms, convertible at $1.50 a share Jul 1, 2013 Knight Capital Group and GETCO combine as KCG Holdings Oct 16, 2013 SEC order: Rule 15c3-5 violated; $12 million penalty, censure and an independent consultant; loss put at $460 million Sources: SEC order 34-70694 and press release 2013-222; Knight Capital Group Forms 8-K, August 2 and 6, 2012.
Knight Capital from deployment to settlement. Compiled from SEC order 34-70694 and Knight Capital Group Forms 8-K of August 2 and 6, 2012.

Forty-five minutes

For 212 incoming parent orders, SMARS sent millions of child orders. In about 45 minutes Knight got more than 4 million executions in 154 stocks for more than 397 million shares. The trades were large enough to move prices: the SEC found that in 75 of the stocks Knight's executions were more than 20% of volume and contributed to price moves of more than 5%, and in 37 of those the price moved more than 10% while Knight was more than half the volume. Some other traders got worse prices because of it, and some got better ones.

The firm's controls did not stop it:

  • No check on the router's output. Nothing compared the orders leaving SMARS with the orders that had entered it, and there was no procedure to halt SMARS for its own abnormal behavior.
  • A price limit that did not apply. SMARS capped a parent order's limit price at 9.5% through the national best bid or offer, but the cap did not cover orders received before the open for the opening auction, which these 212 were.
  • Limits that were not linked to order entry. The executions piled up in an internal account, the 33 Account, that had a $2 million gross position limit. The limit was not connected to any automated control, so nothing stopped new orders once it was breached.
  • Monitoring by eye. Knight's main risk tool showed positions after the fact, gave no automatic alerts, did not display the limits and lagged in heavy volume.
  • No incident plan. Staff tried to find the problem while the system stayed connected. One fix, removing the new code from the seven good servers, made it worse: the reused flag now woke Power Peg on those servers too.

When the orders finally stopped, Knight held a net long position of about $3.5 billion in 80 stocks and a net short position of about $3.15 billion in 74 stocks. Many of the unintended sales were short sales that were marked as long and made without locating shares to borrow, which were separate violations of Regulation SHO.

Knight’s August 1 positions and what followed, millions of dollars Net long position, 80 stocks $3,500 million Net short position, 74 stocks $3,150 million Loss on the positions over $460 million Rescue financing, Aug 6 $400 million SEC penalty, Oct 2013 $12 million Sources: SEC order 34-70694; Knight Capital Group Form 8-K, August 6, 2012. Approximate values as stated there.
Approximate values from SEC order 34-70694 and Knight Capital Group’s Form 8-K of August 6, 2012.

The loss and the rescue

The next day, August 2, Knight said in a filing that a technology issue at the open, related to its installation of trading software, had sent numerous erroneous orders in NYSE-listed stocks and produced a realized pre-tax loss of about $440 million. Its capital base, it said, had been severely affected, although its broker-dealers still met their net capital requirements. The SEC later put the loss at more than $460 million.

To put that in proportion: the positions it had to unwind were more than 14 times the eventual loss, and the loss was larger than the rescue that followed. On August 6, 2012, Knight announced $400 million of equity financing from six firms, Jefferies, Blackstone, GETCO, Stephens, Stifel and TD Ameritrade, in preferred shares convertible into common stock at $1.50 a share. The company said the money let it resume normal operations immediately. Less than a year later, on July 1, 2013, Knight Capital Group and GETCO combined to form KCG Holdings.

The SEC case

On October 16, 2013, the SEC settled charges against Knight Capital Americas under Rule 15c3-5, the Market Access Rule adopted in November 2010. The rule requires a broker-dealer with access to exchanges to have risk controls reasonably designed to prevent erroneous orders and orders that exceed pre-set credit or capital thresholds, and to review those controls regularly with a yearly certification by the chief executive. Direct market access (DMA) explains how the rule came about.

The SEC found that from at least July 14, 2011, when the rule's first compliance date arrived, through August 1, 2012, Knight's controls fell short in several ways: no erroneous-order control at the router immediately before orders went to market, no firm-wide capital thresholds linked to order entry, inadequate written descriptions of its controls, weak governance of code deployment and of unused code, no incident procedures, reviews that missed these gaps, and a 2012 CEO certification that said only that Knight had processes in place to comply, rather than certifying that its controls complied. It also noted an October 2011 incident in which a desk kept quoting from test data after a weekend disaster-recovery test and lost nearly $7.5 million; Knight had fixed that one system without asking whether its controls were adequate everywhere.

Knight agreed, without admitting or denying the findings, to a censure, a cease-and-desist order, a $12 million penalty and an independent consultant to review its software development and risk controls. The SEC's press release called it the first enforcement action under the Market Access Rule.

What it changed

  • Controls must sit in front of the order, not behind it. The case made concrete what Rule 15c3-5 means: automated, pre-trade limits on order size, price, duplicate orders and firm-wide capital, able to block an order before it reaches the market, rather than a screen someone watches after the fills arrive.
  • Software deployment is a risk control. Code review, a second check on deployments, testing of old code and removal of dead code are part of a trading firm's risk management, as the consultant undertaking in the order spelled out.
  • A way to stop, and an incident plan. The order faulted Knight for staying connected while its staff searched for the cause, and the consultant had to review its incident response protocols along with its routers and capital limits.
  • Wider rules for market infrastructure. In November 2014 the SEC adopted Regulation SCI, which requires self-regulatory organizations such as exchanges, certain alternative trading systems, plan processors and certain clearing agencies to have written policies for their technology, to fix and report systems problems and to test business continuity. It covers market infrastructure, not broker-dealers like Knight, for which the Market Access Rule remains the main rule.

Lessons for investors

The losses fell on Knight itself, although the SEC found that other traders in the affected stocks got worse prices, or better ones, than they would have otherwise. The episode matters to anyone who trades. It shows that much of the liquidity behind a quote comes from automated firms whose errors can move prices in seconds, and that a firm handling a tenth of the market can be brought down by a deployment step. The same logic of hard, automatic limits applies at the scale of one account: Daily loss limits: when to stop trading for the day shows how an individual trader can set a stop for a bad day before it arrives. For the market-wide circuit breakers and single-stock pauses that limit the damage of runaway orders, see Trading halts and Limit Up-Limit Down (LULD) pauses explained; for the other great algorithmic scare of the period, The 2010 Flash Crash: what happened on May 6 and what changed.

How Finabase does it

Finabase is not a broker-dealer and does not send orders to any market, so the Market Access Rule does not apply to it. Its paper account at /trading fills simulated market and limit orders against live prices with virtual money. Nothing placed there reaches an exchange, which is what makes it a safe place to practise order entry and sizing.

Frequently asked questions

What happened to Knight Capital in 2012?

On August 1, 2012, a software deployment error made one of Knight's order routers send millions of unintended orders in about 45 minutes. The firm built up billions of dollars of unwanted positions and lost more than $460 million.

How much did Knight Capital lose?

The SEC put the loss at more than $460 million. Knight's own first estimate, on August 2, 2012, was a realized pre-tax loss of about $440 million.

Was Knight Capital fined?

Yes. In October 2013 Knight Capital Americas agreed to pay a $12 million penalty to settle SEC charges that it violated the Market Access Rule, without admitting or denying the findings.

Did Knight Capital go bankrupt?

No. Six firms provided $400 million of equity financing on August 6, 2012, and in July 2013 Knight Capital Group combined with GETCO to form KCG Holdings.

Summary

Knight Capital's 2012 failure began with a missed server and a reused flag that woke dormant code, and grew because nothing stopped a router that kept sending orders. In about 45 minutes the firm got more than 4 million executions, built roughly $3.5 billion of long and $3.15 billion of short positions, and lost more than $460 million. A $400 million rescue kept it alive until its merger with GETCO. The SEC's $12 million settlement, its first under the Market Access Rule, turned the episode into a lasting standard: automated trading needs automated, pre-trade limits, disciplined software releases and the ability to stop first and investigate second.

Check your understanding

4 questions on this lesson. Each answer comes with the reason, and nothing is scored.

  1. What started Knight Capital’s August 1, 2012 failure, according to the SEC?

    1. New code missed one of eight servers, and a reused flag switched on old, retired code there
    2. A trader placed a deliberately oversized bet that went wrong
    3. An exchange outage sent Knight’s orders to the wrong venue
    Show the answer

    A. New code missed one of eight servers, and a reused flag switched on old, retired code there The deployment left the old Power Peg code on the eighth SMARS server. The flag that now meant “use the new retail program logic” still meant “run Power Peg” there, and Power Peg no longer knew when a parent order was filled.

  2. Knight had a $2 million position limit on the account where the stray executions piled up. Why did it not stop the orders?

    1. The limit was not linked to any automated control on order entry, so breaching it blocked nothing
    2. The limit applied only to short positions
    3. Exchanges are required to ignore broker position limits during the opening auction
    Show the answer

    A. The limit was not linked to any automated control on order entry, so breaching it blocked nothing A limit someone has to notice on a screen is not a pre-trade control. The SEC found Knight’s capital thresholds were not tied to order entry, which is what Rule 15c3-5 requires: controls that reject orders before they reach the market.

  3. Knight ended up net long about $3.5 billion and net short about $3.15 billion, and lost over $460 million. Roughly how large were the unintended positions compared with the loss?

    1. About 14 times the loss
    2. About the same as the loss
    3. About 140 times the loss
    Show the answer

    A. About 14 times the loss $3.5 billion plus $3.15 billion is $6.65 billion, about 14.5 times $460 million. Positions that size can be unwound only at a cost, and the cost exceeded the $400 million the firm raised a week later.

  4. Under which rule did the SEC charge Knight in October 2013, and what did the settlement include?

    1. The Market Access Rule (Rule 15c3-5); a $12 million penalty, a censure and an independent consultant
    2. The order protection rule (Rule 611); a $460 million fine
    3. Regulation SCI; a ban on market making
    Show the answer

    A. The Market Access Rule (Rule 15c3-5); a $12 million penalty, a censure and an independent consultant It was the SEC’s first case under Rule 15c3-5, adopted in 2010. Knight settled without admitting or denying the findings. Regulation SCI came in 2014 and covers market infrastructure such as exchanges, not broker-dealers like Knight.

Exercise

One task to do in Finabase. Nothing is graded or saved.

Before you place a paper trade in the paper trading account, write down two limits of your own: the most shares you will buy in one order and the highest price you will pay. Then enter the order as a limit order at that size and price, and check the order screen against your note before you confirm it.

Open paper trading
What you should find

Your note is a pre-trade control, the kind the SEC found missing at Knight: a check on size and price that sits in front of the order, not a screen watched after the fills arrive. A limit price caps what one order can pay. A market order has no cap, and Knight’s 9.5% price limit did not cover the 212 opening orders that ran away.

Sources and further reading

  1. SEC — In the Matter of Knight Capital Americas LLC, order instituting proceedings, Release 34-70694 (October 16, 2013)
  2. SEC — Press release 2013-222: SEC charges Knight Capital with violations of market access rule (October 16, 2013)
  3. Knight Capital Group — Form 8-K, Exhibit 99.1 press release (August 2, 2012): technology issue and pre-tax loss of about $440 million
  4. Knight Capital Group — Form 8-K, Exhibit 99.1 press release (August 6, 2012): $400 million equity financing
  5. eCFR — 17 CFR 240.15c3-5, Risk management controls for brokers or dealers with market access
  6. SEC — Press release 2014-260: SEC adopts Regulation SCI (November 19, 2014)