A partnership with academia

Building knowledge for trade and development

Vi Digital Library - Text Preview

Quantification of the High Level of Endogeneity and of Structural Regime Shifts in Commodity Markets

Discussion paper by Filimonov,Vladimir; Bicchetti, David; Maystre, Nicolas; Sornette ,Didier, 2013

Download original document (English)

This paper proposes a “reflexivity” index that quantifies the relative importance of short-term endogeneity/reflexivity for several commodity futures markets (corn, oil, soybeans, sugar, and wheat) and a benchmark equity futures market. The authors use a reflexivity index that is defined as the average ratio of the number of price moves that are due to endogenous interactions to the total number of all price changes, which also include exogenous events. Estimated reflexivity levels are obtained by calibrating the Hawkes self-excited conditional Poisson model on time series of price changes. The Hawkes model accounts simultaneously for the co-existence and interplay between the exogenous impact of news and the endogenous mechanism by which past price changes may influence future price changes. Robustness tests show that the index provides a ‘pure’ measure of endogeneity that is independent of the rate of activity, order size, volume or volatility. The authors find an overall increase of the reflexivity index since the middle of the first decade of the 21st century to October 2012, which implies that, on a monthly basis, at least 60–70 per cent of commodity price changes are now due to self-generated activities rather than novel information, compared to 20–30 per cent earlier. While their reflexivity index is defined on short-time windows (10 minutes) and thus does not capture long-term memory, they discover striking coincidence between its dynamics and that of the price hikes and abrupt falls that developed since 2006 and culminated in early 2009.

No. 212


November 2013


QUANTIFICATION OF THE HIGH LEVEL OF ENDOGENEITY


AND OF STRUCTURAL REGIME SHIFTS IN


COMMODITY MARKETS






Quantification of the high LeveL of endogeneity
and of StructuraL regime ShiftS in


commodity marketS


Vladimir Filimonov, David Bicchetti, Nicolas Maystre and Didier Sornette


No. 212
November 2013


Acknowledgements: The authors would like to thank Heiner Flassbeck, James Hamilton, Scott Irwin, Mika
Kastenholz, Jörg Mayer, Regis Moneret and two anonymous reviewers for helpful comments and insights on
earlier drafts of this paper, as well as participants in the International Conference on Understanding International
Commodity Price Fluctuations at the International Monetary Fund in March 2013. A condensed version of this
UNCTAD Discussion Paper has been accepted for publication in the Journal of International Money and Finance.


UNCTAD/OSG/DP/2013/2




ii


The opinions expressed in this paper are those of the authors and are not to be taken as the official views
of the UNCTAD secretariat or its member States. The designations and terminology employed are also
those of the authors.


UNCTAD Discussion Papers are read anonymously by at least one referee, whose comments are taken
into account before publication.


Comments on this paper are invited and may be addressed to the author, c/o the Publications Assistant,
Macroeconomic and Development Policies Branch (MDPB), Division on Globalization and Development
Strategies (DGDS), United Nations Conference on Trade and Development (UNCTAD), Palais des Nations,
CH-1211 Geneva 10, Switzerland; e-mail: gdsinfo@unctad.org; fax no: +41 22 917 0274. Copies of
Discussion Papers may also be obtained from this address.


UNCTAD Discussion Papers are available on the UNCTAD website at http://unctad.org.


JEL classification: G10, G12, G13, G14, O33




iii


Contents


Page


Abstract ....................................................................................................................................................... 1


i. introduction ................................................................................................................................ 1


ii. exogenouS verSuS endogenouS factorS in price dynamicS ........................... 3
A. Brief literature review on the relationship between news and commodity prices ............................ 3
B. Disentangling exogenous and endogenous activities in price dynamics .......................................... 4


iii. nature and characteriSticS of the anaLySed futureS contractS .......... 5


iv. the mathematicaL modeL ..................................................................................................... 9
A. Branching processes and reflexivity ................................................................................................. 9
B. Hawkes model ................................................................................................................................. 11
C. 10-minute time windows and stationarity issues ............................................................................ 12
D. Methodology ................................................................................................................................... 13


v. eStimation of the refLexivity index of commodity futureS priceS ..... 18
A. The benchmark of financial markets: E-mini S&P 500 futures ...................................................... 18
B. The evolution of endogeneity in commodity futures markets ........................................................ 20


vi. robuStneSS teStS ...................................................................................................................... 30


vii. concLuSion ................................................................................................................................... 32


referenceS ........................................................................................................................................... 33


List of tables
1 Description of the selected instruments .................................................................................................. 6
2 Summary statistics, 2005–2012 .............................................................................................................. 7
3 Uncertainty of the timestamps of events resulting from the nature of the FAST/FIX feed,
2005–2012 ............................................................................................................................................ 15
4 Fraction of 10-minute windows rejected for the calibrations of the Hawkes model
for selected intervals (Δ), 2005–2012 ................................................................................................... 17


List of figures


1 Monthly volumes of futures contracts for selected commodities, 2005m1–2012m10 .......................... 8
2 Illustration of the branching structure of the Hawkes process and events on the time axis ................. 10
3 Illustration of the high-frequency price dynamics ............................................................................... 13
4 Selected dynamics of the E-mini S&P 500 futures, 1998m1–2012m10 .............................................. 19
5 Prices, price volatility and branching ratio estimates in selected oil futures markets .......................... 21
6 Price time series of NYMEX Light Sweet Crude (front-month contract) and simple log-periodic
power law (LPPL) fits .......................................................................................................................... 24
7 Prices, price volatility and monthly branching ratio in selected agricultural futures markets,
2005m1–2012m9 .................................................................................................................................. 26
8 Monthly branching ratios computed on randomly redistributed mid-quote price changes
in selected commodity futures markets ................................................................................................ 31






Quantification of the high LeveL of endogeneity
and of StructuraL regime ShiftS in


commodity marketS
Vladimir Filimonov,* David Bicchetti,**
Nicolas Maystre** and Didier Sornette***


Abstract


We propose a “reflexivity” index that quantifies the relative importance of short-term endogeneity/
reflexivity for several commodity futures markets (corn, oil, soybeans, sugar, and wheat) and a
benchmark equity futures market (E-mini S&P 500). Our reflexivity index is defined as the average
ratio of the number of price moves that are due to endogenous interactions to the total number of
all price changes, which also include exogenous events. Estimated reflexivity levels are obtained by
calibrating the Hawkes self-excited conditional Poisson model on time series of price changes. The
Hawkes model accounts simultaneously for the co-existence and interplay between the exogenous
impact of news and the endogenous mechanism by which past price changes may influence future
price changes. Our robustness tests show that our index provides a ‘pure’ measure of endogeneity
that is independent of the rate of activity, order size, volume or volatility. We find an overall increase
of the reflexivity index since the middle of the first decade of the 21st century to October 2012, which
implies that, on a monthly basis, at least 60–70 per cent of commodity price changes are now due to
self-generated activities rather than novel information, compared to 20–30 per cent earlier. While
our reflexivity index is defined on short-time windows (10 minutes) and thus does not capture long-
term memory, we discover striking coincidence between its dynamics and that of the price hikes and
abrupt falls that developed since 2006 and culminated in early 2009.


i. introduction


The increasing importance of non-traditional participants on commodity futures markets over the last
decade – referred to as the “financialization” of commodity futures markets (Domanski and Heath, 2007)
– coincided with another major structural change in trading: the transition from an open outcry trading
platform to a computer/electronic order matching platform. These two structural changes resulted, at the
very least, in the following developments. First, market access expanded. Second, direct trading costs
declined. Third, investments tracking a commodity index became an accepted alternative investment
for institutions and pension funds. Fourth, exchange traded funds and synthetic investment vehicles that
tracked or are based on commodity indices or even single futures markets were introduced. As a result,


* Dept. of Management, Technology and Economics, ETH Zürich, Switzerland; e-mail: vfilimonov@ethz.ch.
** United Nations Conference on Trade and Development, Division on Globalization and Development Strategies,


Geneva, Switzerland; e-mail: david.bicchetti@unctad.org, nicolas.maystre@unctad.org.
*** Dept. of Management, Technology and Economics, ETH Zürich, and Swiss Finance Institute, c/o University of


Geneva, Switzerland; e-mail: dsornette@ethz.ch.




2


these changes undoubtedly contributed to the increase in the volume of trade on commodity futures
markets (Irwin and Sanders, 2012).


Arguably, the growing liquidity emanating partly from the financialization of commodity markets, coupled
with a shift in trading strategies and technological progresses, has favoured the expansion of quantitative
trading on commodity futures markets. In parallel, many commodity prices have experienced roller-coaster
rides since the middle of the first decade of the 21st century. Overall, these developments have fueled an
intense debate regarding the roles of financial investors and quantitative trading. Disagreements relate
to whether these new actors have improved the price discovery process of commodities futures markets
or whether they have made the process less effective and more unstable.


The proponents of these developments argue that the benefits are at least threefold. First, futures prices
become closer to their underlying fundamentals. Second, liquidity increases. Third, risks are transferred
to agents who are better prepared to assume it (see e.g. Stoll and Whaley, 2010, 2011; Irwin and Sanders,
2012; and references cited therein). In short, this process supports the efficient market hypothesis (EMH)
(Samuelson, 1965; Fama, 1970, 1991). By contrast, other observers argue that financial investors and
quantitative trading can have negative effects on commodity markets because they can distort the price
formation process (see e.g. UNCTAD, 2009, 2011; Tang and Xiong, 2012; Bicchetti and Maystre, 2012
and references cited therein). On some occasions, these price distortions culminated in “speculative
bubbles” (see e.g. Gilbert, 2007; Phillips and Yu, 2011; and references cited therein).1


To contribute to this debate, we analyse the microstructure of several commodity futures markets at short
time scales and provide quantitative dynamic estimates of their degree of reflexivity. More precisely, we
present a measure of the “reflexivity” or endogeneity of high-frequency price movements, defined as the
average ratio of the number of price moves that are due to endogenous interactions to the total number
of all price changes, which also include exogenous events. Robustness tests show that our measure of
endogeneity is independent of the rate of activity, order size, volume or volatility. Then, we calibrate
our measure on several commodity futures markets (corn, oil, soybeans, sugar, and wheat) and also on
a benchmark equity futures market (Emini S&P 500). The obtained reflexivity indices show that, on a
monthly basis, at least 60–70 per cent of commodity price changes are now due to self-generated activities
rather than novel information, compared to 20–30 per cent in the early 2000s.


Our results question the strict tenet of the EMH that the market absorbs in full and essentially
instantaneously the flow of information by faithfully reflecting it in asset prices. Our finding that past
price changes can trigger a large fraction of subsequent price variations implies at the very least that the
information digestion process of markets is not that efficient. While additional evidence is needed to fully
ascertain this point, our findings suggest even that Soros (1987)’s concept of “market reflexivity” may
be relevant, namely that a great deal of what occurs on financialized (commodity) markets is not much
related to fundamental information but rather to a circular loop between trading and price.


We complement our analysis by relating the endogeneity dynamics of these futures markets to their price
dynamics, particularly around the commodity price bubble that started to develop in 2006 and culminated in
mid-2008. While our reflexivity index is, by nature, designed to study only short-term price developments,
interestingly, we find on some occasions a remarkable coincidence between the price dynamics at long
time scales and the time dynamics of trades at short term intervals.


The rest of the paper is organized as follows. In section II, we briefly review the economic literature on
news and commodities, and more generally on assets and prices, highlighting the main differences between


1 According to the CFTF, a speculative bubble refers to “a rapid run-up in prices caused by excessive buying that is
unrelated to any of the basic, underlying factors affecting the supply or demand for a commodity”. See http://www.cftc.
gov/ConsumerProtection/EducationCenter/CFTCGlossary/glossary_s.




3


our approach and more traditional analyses. We also discuss possible sources of reflexivity/endogeneity
and explain why our index primarily captures short-term mechanisms of algorithmic and high-frequency
trading (HFT) and is, a priori, less adequate to study the impact of long-term mechanisms. In section III,
we present our data and then discuss some recent technological changes that appeared on exchanges, such
as the increasing influence of algorithmic trading, in particular HFT activities. In section IV, we introduce
the class of mathematical models called branching processes and explain how they can provide a measure
of endogeneity. Focusing on the Hawkes branching model, we discuss its underlying assumptions and
conditions for validity. Our “reflexivity index” is defined and the methodology of its estimation is also
explained. In section V, we present our results on the reflexivity index calibrated on commodity markets
and also a benchmark equity index. In section VI, we present some robustness of our estimation procedure.
In section VII, we conclude.


ii. exogenouS verSuS endogenouS factorS in price dynamicS


A. Brief literature review on the relationship between news and commodity prices


In principle, participants on futures markets make decisions on the basis of expectations regarding the
evolution of the price of the underlying assets over the term of their contracts. Thus, they constantly look
for new information that either corroborates or challenges their previous assessments. The uninterrupted
flow of news provides market participants with a basis for comparing expectations against reality as it
unfolds. The economic literature dealing with news and asset prices refers mostly to the reaction of the
latter to the former. In this research field, empirical studies have mostly addressed how unanticipated
“news” – mostly defined as the deviation of scheduled macroeconomic releases from the ex-ante
consensus – affects asset prices, but also their volatility (see e.g. Andersen et al., 2003). In doing so, they
only consider a small subset of the whole universe of news, which correspond to “surprises” triggered
by macroeconomic releases. In all likelihood, these surprises lead to changes of expectations. They also
provide new bases for reassessing expectations, leading to trading decisions and, probably, asset price
changes. However, this literature is rather scarce when it comes to commodity prices and its findings
are not fully conclusive. Note also that it often relies on daily data, which makes the identification of
causality more complex since several exogenous events might impact a given market within the same
day. For instance, Roache and Rossi (2010) find that 12 commodity prices are relatively insensitive to
macroeconomic news, while Hess et al. (2008) show that commodity prices are responsive only during
recessionary periods, but not during periods of economic growth.


To the best of our knowledge, only three studies use intra-day data. Yet, diverging results remain. Looking
at 15-minute intervals, for instance, Christie-David et al. (2000) find that, between 1992 and 1995, the
impact of economic surprises on the return variance of gold and silver futures prices was rather limited.
Similarly, looking at 5-minute returns between 1994 and 1997, Cai et al. (2001) find that the impact of
macroeconomic announcements is much smaller on gold compared to the impact on Treasury bond or
currency markets and that only four announcements carry statistically significant effects on gold volatility.
More recently, Elder et al. (2012) look at the impact of 19 different announcements on intra-day returns,
volatility and trading volume at 5-minute intervals between 2002 and 2008 on gold, silver and copper
futures. Their analysis reveals that macroeconomic news releases have a strong impact on all three metals
and their effects dissipate within about 60 minutes of the news release.


Another branch of the literature on news and assets prices investigates how price changes in themselves
reveal information to the market, in particular when some market participants have private information
about the value of an asset. Glosten and Milgrom (1985) and Kyle (1985) first formalized this concept
but, to our knowledge, this has never been investigated empirically on commodity markets.




4


B. Disentangling exogenous and endogenous activities in price dynamics


In the present paper, we significantly depart from these two branches of the existing literature on news
and asset prices. Rather than looking at a limited number of scheduled macroeconomic announcements,
our approach aims at capturing the entire universe of news impacting commodity futures markets. In this
sense, our methodology, which builds on Filimonov and Sornette (2012), does not distinguish between
scheduled releases and unexpected news.


Our determination of the exogenous trading activity includes all trades resulting directly from real news
as well as all trades that may occur due to idiosyncratic causes, as long as they can be considered not to
have been influenced by other trades and traders. We refer to trading activity (proxied by price changes)
as exogenous, if its origin is external, in the sense that it can probabilistically be attributed to idiosyncratic
events. Obviously, macroeconomic releases constitute one source for the exogenous dynamics, however
the latter is not solely limited to this type of information. More generally, we call exogenous any genuine
new piece of information, other than previous price changes, that forces market participants to reassess
their investments and react.


By contrast, we call endogenous all price changes that are triggered by preceding price changes as the
result of internal feedback mechanisms. The existence of triggered events embodies the mechanism of
self-excitation of the system onto itself, i.e. the influence of past price changes on future price changes.
Self-excitation and endogeneity are related to the concept of reflexivity, which has extensive roots in
philosophy and sociology, and was put forward by Soros (1987) as a useful framework for the understanding
of financial markets.


We propose the following non-exhaustive list of mechanisms for endogeneity/reflexivity:


• Technical analysis, including algorithmic trading and HFT: market participants send orders that are not
based on changes in economic fundamentals but on technical analysis of price and volume movements.


• Behavioural mechanisms and herding: changes in fundamentals trigger an avalanche of new orders
based on momentum or market sentiments that cause prices to over/undershoot.


• Optimal portfolio execution: in order to minimize market impact when buying/selling large numbers
of shares, orders are split into smaller orders that are traded incrementally based on on-going price
impacts.


• Hedging strategies increase cross-excitation and cross-correlation between markets. When combined
with portfolio execution issues, hedging strategies amplify self-excitation, as for instance in insurance
portfolio (Kyle and Obizhaeva, 2012).


• Margin and leveraged trading can result in a margin call when price moves above or below a certain
threshold. If the call is not addressed, it would result in the automatic liquidation of the leveraged
position, exacerbating price movements through a domino effect.


• Complex orders, such as stop-loss orders.


• Finally, a combination of any of the above mechanisms could increase market reflexivity.


All mechanisms presented above operate within different time frames: from days and weeks, such as
herding and behavioural mechanisms, to milliseconds, such as HFT. The present study focuses on price
evolution over small windows of 10 minutes, so as to ensure the validity of the assumption of stationarity
of the process, which is discussed further in sections IV.A and IV.C. Price dynamics at the 10-minute
time scale are essentially determined by (i) algorithmic and HFT strategies and (ii) complex orders.
To a lesser extent, it can also capture (iii) short-term human reflexivity and (iv) herding in algorithmic
strategies. The other mechanisms mentioned above are not relevant at the 10-minute time scale: longer-
term herding is responsible for bubble formation within time frames of weeks to months (and more rarely
of longer periods); optimal portfolio executions often take from tens of minutes to hours, and even days




5


for extremely large portfolios; in the case of a margin call, the clearing house usually gives one day for
retail clients to react; and hedging involves time frames also exceeding one day.


Defined as the high speed component of algorithmic trading, HFT has greatly benefited from
financialization, as well as from the transition towards electronic trading platforms. HFT was indeed
extremely limited in early 2000s, accounting for less than 10 per cent of all equity orders. In subsequent
years, it grew rapidly. In 2009, the TABB Group estimated the proportion of HFT in the United States
equity markets to exceed 60 per cent (Sussman et al., 2009), while the Aite Group (2009) provided an
estimate of 73 per cent (Iati, 2009). Although reliable estimates of HFT activities on commodity markets
do not exist, the chief executive officer of the CME Group declared that 45 per cent of volume exchanged
on the NYMEX was computer-driven,2 while other estimates point to higher figures.3


iii. nature and characteriSticS of the anaLySed futureS contractS


We base our analysis on Thomson Reuters Tick History (TRTH) data. TRTH provides financial data
for an extensive range of asset classes with more than 45 million unique instruments across more than
400 exchanges, based on the information transmitted by exchanges and market makers. TRTH contains
historical data back to January 1996 for some time series. It provides granular tick as well as lower
frequency pricing data, up to the microsecond level. Moreover, TRTH offers data on intra-day sales,
quotes, and market depth. The database also provides over-the-counter (OTC) quotes. To our knowledge,
it offers the most comprehensive pricing and reference data service, with a record of market behaviour
of 2 petabytes (2∙1015 bytes) of tick data.


In this study, we limit ourselves to a few instruments. We select some of the most liquid commodity
derivatives, namely futures on Brent crude oil (ICE – Europe), WTI crude oil (NYMEX), corn (CBOT),
soybeans (CBOT), sugar #11 (ICE – United States), wheat (CBOT) and white sugar (LIFFE). These
commodity futures contracts represent commonly used benchmarks for the world or their respective
domestic markets.


Table 1 summarizes the main characteristics of each futures contract. Each derivative contract has an
underlying physical asset described in the “Specification” column and reaches maturity on specific dates,
which we refer to as “Contract month”. Several futures contracts referring to the same underlying asset
are traded in parallel during the trading sessions but are differentiated by their maturity dates. The front
months for each future contracts usually have the greatest liquidity. For each considered commodity,
TRTH provides a so-called continuous futures contract by taking the front month and switching to the
next contract at expiration date.


The continuous ‘front-month’ futures contract (which has suffix “c1” in the TRTH notation) usually
exhibits greater trading volumes than other futures contracts with subsequent maturities. However, as the
expiration date approaches, this is no longer true as financial investors switch to the next maturity contract
to avoid delivery. The peak of these rollover processes has traditionally preceded the expiration date by
about one week. For instance, E-mini S&P 500 futures contracts are traded on a quarterly basis, and expire
on the third Friday of March, June, September and December. However, in our observations, the number
of trade on the front-month futures contract become smaller than the ones on the next maturity contract


2 David Sheppard, 3 March 2011. ‘NYMEX oil trade 45 per cent computer-driven’. Available at: http://www.reuters.
com/article/2011/03/03/us-finance-summit-nymex-volume-idUSTRE7225RV20110303 (Last accessed on 27 June 2013).
3 Note that this figure probably refers to a conservative estimate. Indeed, for WTI crude oil, day trading, for which a
large share refers to HFT, generates almost 95 per cent of trading volume. Gregory Meyer, 5 July 2011. ‘CFTC data
reveal day traders’ role in volatile oil markets’. Available at: http://www.ft.com/intl/cms/s/0/b29b2b1e-a743-11e0-b6d4-
00144feabdc0.html#axzz288MmVAIz (Last accessed on 27 June 2013).




6


eight days before the expiration dates, i.e. on the second Thursday of each of these months. Hence, the
liquidity (measured in volume) of the expiring contract is switched from the expiring contract to the next
quarter maturity at these rollover dates (see Frenk and Turbeville, 2011). For Brent and WTI futures,
monthly settlements result in rollover dates that are closer to the expiration dates as most traders typically
roll their positions two days before the expiration. In order to be consistent in our analysis of different
assets, we have excluded periods between the rollover and the expiration dates from our analysis. For
the corn, wheat and soybeans futures contracts, we could not clearly identify rollover dates because, in
some instances, the second-month contracts are more heavily traded than those of the front month. For
these contracts we have excluded five trading days before the expiration.


As can be seen from table 1, different exchanges moved the trading activities from pit trading to full
electronic platforms at different times. Brent crude oil, which was originally traded on the open outcry
International Petroleum Exchange (IPE) in London, was the first oil contract that fully switched in 2005
to the electronic platform of Intercontinental Exchange (ICE) based in London. However, the white sugar
traded in Europe at LIFFE moved already in 2000 on a full electronic platform.


As further discussed in section IV.C, we base our analysis on the so-called mid-quote price, which averages
the best bid price and the best ask price. Due to the specifics of the open outcry pits, which were mostly
driven by designated (official) market makers, the quotes on the pre-electronic exchanges could not


Table 1
Description of the selected instruments


Abbreviation* Specification
Exchange and
trading platform Contract month


Introduction of
electronic trading


Hours with
most active


trading


Brent Crude
(LCOc1)


1,000 barrels of light
sweet crude oil


ICE Europe / ICE
electronic platform


Every month 7 April 2005 BST
15:15–19:45 [a];
14:00–19:45[b]


WTI
(CLc1)


1,000 barrels of light
sweet crude oil


NYMEX / CME
Globex


Every month 4 September 2006 EST
10:00–14:45 [a];
9:00–14:45 [b]


Soybeans
(Sc1)


5,000 bushels
(~136 metric tons)


CBOT / CME Globex January, March,
May, July, August,
September,
November


1 August 2006 CDT
9:45–13:30


Sugar #11
(SBc1)


112,000 pounds ICE United Sates /
ICE electronic platform


March, May, July,
October


12 January 2007 [c] EST
8:15–13:45


Corn
(Cc1)


5,000 bushels
(~127 metric tons)


CBOT / CME Globex March, May,
July, September,
December


1 August 2006 CDT
9:45–13:30


Wheat
(Wc1)


5,000 bushels
(~127 metric tons)


CBOT / CME Globex March, May,
July, September,
December


1 August 2006 CDT
9:30–13:30


Sugar
(LSUc1)


50 metric tons LIFFE / NYSE
Euronext


March, May, August,
October, December


27 November 2000 BST
9:30–17:30 [d];
8:30–17:30 [e]


E-mini
S&P500
(ESc1)


50 x E-mini S&P 500
futures price


CME / CME Globex March, June,
September,
December


9 September 1997 EST
9:30–16:15


Source: Authors’ calculations, based on the TRTH database.
Note: *Reuters Instrument Code (RIC) is reported between parantheses.[a] Prior to 22 January 2007. [b] Since 22 January


2007. [c] Before 2 March 2008, data was disaggregated into RICs “SBc1” and “1SBc1” for pit and electronic trading
and real time bid, ask, volume and settlement values are not provided due to feed limitations. [d] Prior to 29 June 2009.
[e] Since 29 June 2009.




7


serve as a reliable source of information because time was not registered precisely. For this reason, in
our analysis, we have considered only time periods starting from the date of the complete switch to full
electronic trading for each contract (indicated in table 1). For sugar #11, we have excluded the period
prior to 2 March 2008 even though electronic trading started on 12 January 2007 at ICE United Sates. In
fact, pit trading had continued to exist in parallel at NYBOT and, unfortunately, because of NYBOT feed
limitations, the timestamps of quotes as well as volumes and settlement values are not entirely reliable
before the decommissioning of NYBOT pit trading on 28 February 2008.


Table 2
Summary statistics, 2005–2012


Volume per transaction Volume per transaction


Year Transactions Volume Average Median
90%-


quantile Transactions Volume Average Median
90%-


quantile


Brent Crude (Europe) WTI (United States)


2005 2,267,138 12,324,431 5.4 1 10 919,941 24,431,479 26.6 2 13
2006 5,723,566 17,543,910 3.1 1 5 2,468,946 29,541,698 11.9 2 8
2007 8,619,849 22,091,574 2.6 1 3 11,960,866 58,268,584 4.9 1 6
2008 13,415,057 26,408,342 2.0 1 2 21,429,745 66,766,312 3.1 1 4
2009 12,791,780 28,241,439 2.2 1 3 21,104,592 66,833,089 3.2 1 4
2010 17,694,513 38,581,454 2.2 1 3 31,570,311 79,334,457 2.5 1 3
2011 25,047,938 46,720,379 1.9 1 3 41,855,040 78,088,015 1.9 1 2
2012* 18,875,419 36,397,876 1.9 1 3 27,420,055 47,640,155 1.7 1 2


Soybeans (United States) Sugar #11 (United States)


2006 437,313 7,389,376 16.9 2 10
2007 1,520,027 11,886,079 7.9 2 10 853,963 11,082,111 13.0 3 14
2008 3,245,768 13,427,406 4.2 1 5 2,884,089 13,010,475 4.5 2 10
2009 2,887,574 13,365,613 4.7 1 6 2,167,801 12,424,883 5.7 1 7
2010 5,562,846 13,385,860 2.4 1 2 4,572,232 12,767,545 2.8 1 4
2011 7,045,022 16,435,216 2.3 1 2 4,513,119 10,867,352 2.4 1 4
2012* 5,043,826 11,191,303 2.2 1 2 3,244,271 8,864,245 2.7 1 5


Corn (United States) Wheat (United States)


2005 133,379 10,318,411 77.4 2 20 116,189 4,540,024 39.1 2 10
2006 514,309 18,215,293 35.4 2 20 306,472 6,296,176 20.5 2 10
2007 1,502,282 19,562,361 13 2 15 1,126,338 7,897,908 7.0 2 10
2008 2,472,303 22,520,498 9.1 2 11 2,060,348 8,120,508 3.9 1 5
2009 2,411,590 21,630,874 9 2 10 1,765,353 8,123,123 4.6 1 6
2010 8,132,933 27,185,254 3.3 1 4 3,887,485 9,447,008 2.4 1 3
2011 10,718,179 30,818,671 2.9 1 3 5,099,530 10,128,749 2.0 1 2
2012* 7,552,195 20,296,990 2.7 1 2 3,677,335 8,582,026 2.3 1 2


Sugar (Europe) E-mini S&P 500


2005 58,524 849,928 14.5 8 30 11,439,420 183,667,226 16.1 2 35
2006 82,688 891,134 10.8 5 23 11,095,507 223,402,685 20.1 2 48
2007 148,815 1,217,541 8.2 4 20 22,183,920 362,881,400 16.4 2 31
2008 158,151 925,481 5.9 3 12 49,488,715 551,544,452 11.1 2 23
2009 294,445 919,343 3.1 1 6 41,655,339 492,581,685 11.8 2 21
2010 400,850 977,312 2.4 1 5 107,143,664 497,545,699 4.6 1 9
2011 485,522 870,938 1.8 1 3 120,700,428 540,010,834 4.5 1 9
2012* 350,957 686,958 2.0 1 3 72,728,681 316,597,629 4.4 1 9


Source: Authors’ calculations, based on the TRTH database.
* The summary statistics shown refer to the period 1 January 2005 to 30 September 2012, except for soybeans and sugar


#11 where the first day is respectively 1 January 2006 and 1 January 2007, and for wheat (United States) and sugar
(Europe) where the last day is 30 May 2012.




8


Figure 1
Monthly volumes of futures contracts for selected commodities, 2005m1–2012m10


(Millions of contracts)


Source: Authors’ calculations, based on Thomson Reuters Tick History database.


0


5


0


5


10
WTI (United States)


0


2


4
Soybeans (United States)


0


1


2
Sugar #11 (United States)


0


5
Corn (United States)


0


1


2
Wheat (United States)


0


0.1


0.2


2005 2006 2007 2008 2009 2010 2011 2012


Sugar (Europe)


Brent Crude (Europe)




9


Table 2 summarizes the number of annual transactions and yearly volumes for each considered derivative
contract.4 The emergence of full electronic trading in the course of the first decade of the 21st century on
the considered commodity exchanges marks the beginning of an increase in the amount of transactions.
At the same time, the dynamics of volume (presented at the monthly scale also in figure 1) exhibits very
moderate growth. As a result, the average volume per transaction (A-VPT) decreased significantly between
2005 and 2012 for all considered contracts: starting at an average of 5–77 contracts per transaction in
2005, this figure declined below 3 contracts per transaction in 2012 for all considered commodities. The
more striking dynamics are observed on the median (M-VPT) and other quantiles of the distribution of
the volume per transaction. No later than 2010, M-VPT settles at 1 for all the considered commodities,
which means that at least 50 per cent of all the transactions involve only one contract. Similarly, the
90%-quantile of the volume per transaction (Q-VPT) have remained equal or below 10 contracts per
transaction from 2009 onwards for commodities. The VPT decline primarily reflects the increase of HFT
on commodity futures markets, whose typical strategies imply ultra-fast market-making with only a few
contracts per limit order.


iv. the mathematicaL modeL


A. Branching processes and reflexivity


The proposed novel econometric approach to assess and quantify the presence of the self-excitation in
price formation is built on the mathematical framework of “branching processes” (Harris, 2002). In this
framework, all events belong to one of two classes: immigrants (zero-order events) and descendants (first-,
second-, and higher order events). The exogenous immigration triggers clusters of descendants: every
zero-order event (mother) can trigger one or more first-order events (daughters), each of whom can, in
turn, become a mother-event and trigger several daughters (second-order events or grand-daughters) and
so on over many generations (see figure 2). All first-, second- and higher order events form the cluster
of aftershocks of the main event as a result of the self-excited (endogenous) generating mechanism of
the system.


To describe such cluster structure, Harris (2002) introduces the concept of “branching ratio”, denoted
as n. This key parameter corresponds to the average number of daughter events of first generation per
mother event. When the branching ratio is small (n ≪ 1), the dynamic is stationary and mostly driven
by the external (exogenous) immigrants in the system, as most of the clusters contain only one or few
events. When the average number n of daughters per mother increases, clustering increases, reflecting
the increasingly important role of the self-excitation mechanisms in the system dynamics. When the
branching ratio is close to one (n <~ 1), an external stimulation of the system by zero-order (e.g. news-
driven) events is followed by many subsequent endogenous events. Finally, when the branching ratio is
above one (n >1), implying that each price change triggers on average more than one future price change,
the dynamic becomes non-stationary. With finite probability, the system explodes in an infinite number
of events without need for a permanent supply of exogenous events.


The key parameter n has an extremely important implication on the analysis of price evolution. If n < 1
and the rate of arrival of truly exogenous events is constant, the branching ratio n is exactly equal to the
average fraction of endogenous events over the whole population of events (Helmstetter and Sornette,
2003; Filimonov and Sornette, 2012). In other words, n is equal to the fraction of events that are triggered


4 Due to feed limitations, TRTH does not contain reliable information of trading volumes for soybeans and sugar #11
contracts prior to the introduction of electronic trading. Therefore, for these contracts we use data starting 2006 and 2007
respectively.




10


by the internal feedback mechanisms described above and (1 − n) represents the fraction of price changes
triggered by exogenous events. Note also that, as n approaches 1, most of the activity becomes endogenous
(or reflexive). In addition, the total activity increases sharply because numerous endogenous events will
follow one particular exogenous event. As a quantitative illustration, for n = 0.8, the observed number
of trades is 5 (precisely 1/[1 − n]) times the number of trades that would exist if each trade was only
reacting directly to an incoming exogenous trigger (e.g. news).


A priori, the excess endogenous trading described by the branching model could be interpreted as reflecting
the “tatonnement” process of convergence of the price towards the fundamental price, and not taken as
a diagnostic of a potential source of inefficiency or instability. This interpretation must be tampered by
taking into consideration the following facts. First, it can be shown that the convergence time to any
“true” price is also proportional to 1/(1−n), which increases without bounds as n increases towards 1,
reflecting an increasing loss of efficiency. Rather than agreeing rapidly on the “correct” price after the
arrival of some unanticipated news, the traders trade longer and longer as n → 1, seemingly hesitating
on what price to settle. As the branching ratio n increases from, say 0.2 to 0.8, not only does the activity
increase by a factor of 4, but the convergence time to the true price is multiplied also by this factor 4. This
supports the interpretation that, as n increases, endogeneity makes the market less efficient. Moreover, not
only does the rate of price change and the convergence time diverge proportionally to 1/(1 − n), but the
variance of the event rate also diverge as n → 1. The susceptibility to external shocks diverges similarly.
All these singular behaviours (in the mathematical sense) point to a growing instability of the system as
the branching ratio increases.


In what follows, we will refer to n using both terms “reflexivity index” (which emphasizes its conceptual
meaning) and “branching ratio” (which emphasizes its technical definition).


Figure 2
Illustration of the branching structure of the Hawkes process (top)


and events on the time axis (bottom)


Note: Different markers correspond to different clusters. The numbers below an event denotes its order within the cluster. This
picture corresponds to the branching ration n = 0.88.


Time


0 0 01 1 1 1 1 1 1 12 2 2 2 2 22 2 23 3 3 342




11


B. Hawkes model


There are several ways to determine n. One is to reverse-engineer the clusters and calculate the average
number of direct descendants to any given event. Statistical techniques allow probabilistic reverse-
engineering clusters by reconstructing ensembles of scenarios for the top structure in figure 1 from the
known bottom timeline (see Sornette and Utkin, 2009; and references therein for discussion of methods
and their limitations).


A simpler way benefits from the exact mapping between the clustered process (figure 2) and the self-
excited model of Hawkes (1971), which was originally used in the study of earthquake aftershocks (see
e.g. Vere-Jones, 1970; Ogata, 1988). Bowsher (2002) (published later with corrections in Bowsher, 2007)
has applied the Hawkes process to high-frequency financial data. Nowadays, the Hawkes point process
has become the “gold standard” of self-excited models to describe discontinuous financial data. It has a
wide range of applications going from modelling high-frequency order flows (Hewlett, 2006; Bauwens
and Hautsch, 2009) and the construction process of the order book (Large, 2007; Toke, 2011; Cont, 2011),
to modelling extreme events clustering at daily and hourly scales (Embrechts et al., 2011), estimating
Value-at-Risk (Chavez-Demoulin et al., 2005) or modelling correlated default times in a portfolio of firms
(Errais et al., 2010; Azizpour et al., 2011).


The Hawkes point process can be regarded as the generalization of the non-homogeneous Poisson process,
whose intensity λ(t) (defined such that λ(t)dt is the expected value of the number of events in the time
interval [t, t+dt[) not only depends on time t but also on the history of the process.5 Within the Hawkes
model, the intensity of a process is conditional on history and has the form


λ(t) = μ(t) + ∑ti<t h(t – ti) (1)


where ti are the timestamps of the events of the process, μ(t) is a background intensity that accounts for
exogenous events (not dependent on history) and h(t) is a memory kernel function that weights how much
past events influence the generation of future events and thus controls the amplitude of the endogenous
feedback mechanism. As can be seen from (1), the Hawkes model clearly isolates the external influences
on the system, μ(t), from the internal feedback mechanisms, h(t). In addition, the linear structure of λ(t)
of the Hawkes model allows it to be mapped exactly onto a so-called branching process (see Daley and
Vere-Jones, 2008) and its key parameter n, which directly equals the “branching ratio” corresponding to it.


Traditionally, the memory kernel is assumed to be exponentially decaying in time:


h(t) = α exp (– t) (2)


τ


It is usually parametrized by two variables α > 0 and τ > 0. The choice of an exponential kernel is motivated
by targeting the short-term speculative mechanisms of reflexivity that operate on the scales of minutes
and less (see section II.B). We have validated this choice numerically using a goodness-of-fit analysis
(see section IV.D.2).


5 Without going into precise mathematical definitions, point processes are special types of random processes, for which
the realization consists of isolated events and the modelled variable is the timestamp (and coordinate as well as marks
if applicable) of each event. The typical null hypothesis in modelling point processes is the so-called Poisson process in
which events occur independently of one another with a constant average arrival rate λ. Having no correlation structure,
the Poisson point process cannot describe the wide range of empirical stylized facts of real order flows. Thus, two large
classes of self-excited point processes have traditionally been used to account at least partially for some stylized facts that
are characteristic of high-frequency price data. The first one is the so-called Autoregressive Conditional Durations (ACD)
model (Engle and Russell, 1997, 1998) and its extensions, which describes the inter-event durations with a GARCH-type
equation. The second one is more parsimonious and flexible and is called the self-excited Hawkes model (Hawkes, 1971).




12


The calibration of the Hawkes model traditionally requires a Maximum Likelihood Estimation (MLE)
method. The log-likelihood function is known in closed form for Hawkes processes (see Ogata, 1978;
Ozaki, 1979 for an analytical expression of the likelihood function). Moreover, if interested only in the
determination of the branching ratio n, it is simpler to estimate the parameters α and τ of the kernel h(t)
in (2) using the following normalization:


n = ∫ ∞ h(t)dt = ατ (3)
0


Then, the MLE method provides a statistical estimation of α and τ, and therefore of n, in addition of μ.


C. 10-minute time windows and stationarity issues


We base our analysis on “mid-quote prices” (see illustration in figure 3), which equal the average
between best bid and best offer prices. In contrast to the last transaction price, which is subjected to the
“microstructure noise” (see e.g. Aït-Sahalia et al., 2005), the mid-quote price is a better proxy for the asset
value, given the available information (Hasbrouck, 1991; Engle, 2000).6 It has also become the “default
measure” to monitor price movements in high-frequency finance (see e.g. Bouchaud et al. (2009) for an
extensive review).


In order to calibrate the Hawkes model, we need to verify that stationarity holds. While trading activity
is not really stationary as it has significantly increased over the last decade, it has done so over long-term
secular time scales (years) that do not affect our estimation. The intraday U-shaped “seasonality”, with
amplitudes almost doubling at the beginning and at the end of each daily trading session, also occurs
on a time scale of hours, and has been checked via synthetic tests to ensure that it does not impact our
estimation procedure in short time windows (e.g. of 10 minutes) (Filimonov and Sornette, 2012). Finally,
the existence of clustering of exogenous triggering events (such as news) is known to bias upward the
branching ratio n.7 But the broadcasted news typically exhibit clustering around some idiosyncratic or
expected events, which typically happen at time scales of hours and longer. For instance, news on a
developing drought or of a mis-functioning oil platform generally do not evolve over minutes but hours
and days. These different effects thus do not affect our short-term reflexivity estimates.


By contrast, the assumption of stationarity cannot hold for some particular short windows during which
major macro-economic news announcement occur (such as the FOMC rate decision or the EIA weekly


6 The choice of a proxy for the price movements at high frequency (minute, second and sub-second time scales) matters
and depends on the particular application. At any given moment t, one may distinguish three different prices: (i) the last
transaction price ptr(t), at which the previous transaction was executed, (ii) the best ask price a(t) and (iii) the best bid
price b(t) at which market participants may immediately correspondingly buy and sell an asset. Best bid and best ask prices
are usually aggregated in the so-called mid-quote price, which averages the two: pm(t)=a(t) +b(t)/2 (see figure 3). The
bid and ask prices reflect demand and supply of the liquidity providers, respectively. The transaction price reflects actions
of liquidity takers. And mid-quote price changes result from actions of all market participants, both liquidity providers
and takers. The transactions are triggered when a market order arrives. In case of a buy market order, the transaction is
executed at the best ask price, while a sell market order triggers a transaction at the best bid price. Since the sequence of
order arrivals is stochastic with the sign of order being a random variable, the last transaction price will jump from best
bid to best ask price and back even without changes in the balance between supply and demand. This stochastic behaviour,
which is called “bid-ask bounce”, represents a kind of “noise source” to the price.


The idea that the last transaction price in high-frequency financial data is a poor proxy of the unobservable asset’s value,
because it is subjected to the additive “microstructure noise” (see e.g. Aït-Sahalia et al., 2005; Black, 1986). In contrast to
the last transaction price, the mid-quote price is free from the bid-ask bounce and changes only when the balance between
supply (liquidity providers) and demand (liquidity takers) is modified.
7 This has been quantified by modelling news clustering also as a Hawkes process or using the Autoregressive Conditional
Durations (ACD) model (Engle and Russell, 1997, 1998) and other standard models of clustered point processes.




13


report). Before the announcement, the trading almost “freezes” to a halt and liquidity evaporates from the
limit order book due to uncertainty perceived by traders. Immediately after the announcement, the activity
shoots up (see e.g. Almgren, 2012). We thus exclude these time windows containing announcements of
the FOMC (eight times per year) and of the EIA (weekly), to be consistent with the model assumption
of constant background (exogenous) activity. In addition, we excluded the days when trading was closed
before the end of hours of active trading (table 1) or with daily volumes less than the 5 per cent quantile
of daily volumes for each given year.8 Summing up, we calibrate our model within 10-minute windows,
which are both short enough to ensure the approximate validity of stationarity (constant parameters, μ, n, τ)
and long enough to contain sufficiently many events to perform reliable calibration.9 We have also checked
that our results are robust with respect to changes of the time window duration to 20 or 30 minutes.


D. Methodology


1. Dealing with the TRTH uncertainties of timestamp recording


We base our analysis on the TRTH database which provides high-frequency data of trades and quotes
with microsecond resolution of the timestamps. Despite the fact that ticks in TRTH are stamped with


8 In any case, since the results presented below are based on monthly averages, the presence in our estimations of some
small number of “outliers” due to news announcements would not change the overall statistics. The tight interquartile
ranges around the monthly average that we observe for all of commodities support this hypothesis.
9 10-minute intervals are wide enough to capture a significant part of the endogenous memory of the system: indeed,
using the exponential kernel, our estimations give a characteristic memory time much smaller than minutes. The number
of mid-quote price changes amounts typically to more than 100–200 events over a typical 10-minute window, which is
sufficient to perform a reliable calibration. In the most active periods it can reach up to several tens of thousands.


Figure 3
Illustration of the high-frequency price dynamics


Note: The black line corresponds to the last transaction price, the thick and the thin grey lines correspond to best ask and best
bid prices respectively and the dashed black line corresponds to the mid-quote price. Black circles denote transactions
and grey squares denote timestamps of mid-quote price changes.


Time


Price
Last transaction price
Best bid price
Best ask price
Mid-quote price
Transaction
Mid-quote price change




14


microsecond resolution, a rather large number of quote changes have identical timestamps. In the most
recent years, we can observe up to several hundreds quote changes during active trading hours for the same
timestamp. The origin of this phenomenon lies in the nature of the data feed from the exchange, which
is obtained by the FAST/FIX protocol. The protocol bundles multiple updates of multiple instruments
within a single message by an algorithm designed by the exchange. Then, the package is sent to the
Thomson Reuters collection system, and TRTH timestamps relate to the time when the messages reach
the collection system, but not to the time when the transactions were actually executed and recorded by
the exchange. Since the exchange time, coded in the FAST/FIX protocol, is stamped with a resolution
of seconds, the actual time of any tick is uncertain within a range that is larger than or equal to the time
between two consecutive FAST/FIX packages. This range varies from tens of milliseconds in recent years
to several hundreds of milliseconds or even seconds in the early 2000s.


An additional source of uncertainty to the timestamps is introduced by the latency of the message traveling
from the exchange to the Thomson Reuters collection system and by the overhead brought by processing
the FAST/FIX protocol on both sides. However, both factors introduce a shift to the timestamp, which
is constant when the latency does not fluctuate. Such constant time shifts would not change the analysis
and could be omitted in principle. In reality however, both actors may vary in time, but the order of
magnitude of these variations is much smaller than the time between consecutive packages and thus
could be neglected. The typical time for package processing is of the order of tens of microseconds.
The latency is usually also of the order of tens of milliseconds. This suggests a rough estimate of their
variations, typically of the order of milliseconds, which is much smaller than the uncertainty introduced
by the bundling of updates to a single message.


Two possibilities can be considered to deal with the uncertainty in the timestamps resulting from FAST/
FIX protocol. One is to consider only the timestamps provided by the exchange (with a resolution of
seconds) as a reliable source of data. The other is to use enriched millisecond timestamps of the TRTH,
while accounting for the uncertainty due to bundling updates in FAST/FIX packages. In this paper, for the
sake of caution, we follow the second option by relying on the non-zero difference of timestamps between
consecutive transactions or updates of quotes as the proxy for the uncertainty in arrival times. Tables 3A
and 3B provide respectively the annual average and median uncertainties of the event timestamps for our
different considered instruments. Starting in the range of 200–300 milliseconds in the middle of the first
decade of the 21st century, the average timestamp uncertainties have decreased progressively over the
years. In 2012, the average and median durations between two consecutive FAST/FIX packages that were
recorded with different timestamps by the TRTH reached a relatively low range of 103–242 milliseconds
and 22–135 milliseconds, respectively.


In order to make the data compatible with the Hawkes model for which the probability of having multiple
events with identical timestamps is equal to zero, we follow the methodology developed by Filimonov
and Sornette (2012). Specifically, we randomly redistribute the TRTH timestamps around their recorded
values within an interval of duration Δ. In doing so, we implicitely assume that each event that occurs
within the interval of width Δ is independent of all the others within the same interval (but not between
different intervals). This processing step tends to underestimate the endogeneity levels presented below.


The intuition that Δ should be chosen to be of the order of the typical duration between consecutive FAST/
FIX packages has been validated by numerical tests. Similarly to Filimonov and Sornette (2012), we have
verified the procedure on synthetic time series obtained by numerical synthesis of the Hawkes process (1)
with parameters (μ,n,τ) close to the calibrated values of the real data. The results of such synthetic tests
and comparisons with the estimation on real data has revealed quantitative limitations of the proposed
method. In particular, the distortion of the distribution of inter-event times becomes significant and, as
a consequence, the estimation of the parameters of the Hawkes process becomes unreliable when Δ is
chosen to be significantly smaller than the typical waiting time between consecutive FAST/FIX packages.
As a rule of thumb, Δ should be chosen to be more than the median duration (table 3B). As seen from
table 3B, for most years, the median timestamp uncertainty is of the order or below 100 milliseconds,




15


while its average value is of the order or below 200 milliseconds. This suggests that a reasonable value
for Δ is 200 milliseconds. In order to check the robustness of the method applied to real data, we have
also used the values Δ=50 milliseconds, Δ=100 milliseconds and Δ=300 milliseconds, where the last value
corresponds to the upper bound of the average uncertainty (table 3A). As an extreme case, we have also
considered Δ=1 second, which corresponds to the resolution of the exchange time.


2. Data construction and testing the goodness-of-fit of our calibration


Using the TRTH dataset, we construct 10-minute intervals for every day from 2005 to 2012 within the
hours of active trading10 (see table 1) with one minute time step. Within each 10-minute interval, we
identify the moments {ti} when the mid-quote price changes (irrespectively to its direction), considering
the so-called point process of the timestamp of events as represented by grey squares on the time axis
in figure 3. The maximum likelihood estimator (Ogata, 1978; Ozaki, 1979) allows us to calibrate the
Hawkes model (1)–(2) on these sequences {ti}.


10 Due to differences in the geographic locations of these trading venues, these hours differ from the so-called “Regular
Trading Hours”.


Table 3
Uncertainty of the timestamps of events resulting from


the nature of the FAST/FIX feed, 2005–2012
(Milliseconds)


(A) Average uncertainty


Contract 2005 2006 2007 2008 2009 2010 2011 2012


Brent (Europe) 332 222 105 98 107 115 165 167
WTI (United States) --- 326 208 133 144 137 141 110
Soybeans (United States) --- 267 240 174 192 146 125 141
Sugar #11 (United States) --- --- --- 235 199 183 243 242
Corn (United States) --- 268 267 186 207 164 142 144
Wheat (United States) --- 287 281 211 213 146 147 141
Sugar (Europe) 309 272 303 344 230 212 200 185
E-mini S&P 500 173 195 168 112 129 87 92 103


(B) Median uncertainty


Contract 2005 2006 2007 2008 2009 2010 2011 2012


Brent (Europe) 227 118 35 26 24 30 65 68
WTI (United States) --- 199 80 62 61 62 59 22
Soybeans (United States) --- 149 130 71 77 32 22 23
Sugar #11 (United States) --- --- --- 112 58 43 127 135
Corn (United States) --- 151 174 75 106 45 32 26
Wheat (United States) --- 174 179 91 86 29 30 22
Sugar (Europe) 223 197 190 245 119 85 84 69
E-mini S&P 500 127 121 79 51 60 31 32 41


Source: Authors’ calculations, based on the TRTH database.
Note: Dashed lines (---) correspond to the time periods before the introduction of electronic trading for the given contract (see


table 1).




16


Each calibration is completed by a goodness-of-fit analysis, to test the hypothesis that the data is generated
by the Hawkes model. Standard quantification of the goodness-of-fit of the data by the Hawkes process
uses the so-called residual analysis (Ogata, 1988). This consists in studying the residual process, defined
as the nonparametric transformation of the initial series of the event time stamps ti into


(4)


where ̂λ (t) is the conditional intensity of the Hawkes process (1) estimated with the maximum likelihood
method. As it was shown by Papangelou (1972), under the null hypothesis that the data has been generated
by the Hawkes process (1) with kernel (2), the residual process ξj should be Poisson (memoryless) with
unit intensity. The goodness-of-fit can then be verified both by (i) visual cusum plot or Q-Q plot analysis
and (ii) rigorous statistical tests, such as independence tests applied to the sequence of ξj and/or tests of
the exponential distribution of the transformed inter-event times ξj – ξj–1, which amounts to testing the
uniform distribution of the random variables Uj=1–exp[–(ξj – ξj–1)] in the interval [0,1]). We have used
the Kolmogorov-Smirnov test in order to test the uniformity of the distribution of Uj’s.


Each 10-minute data interval is characterized by the estimated parameters (μ,n,τ) and a p-value of the
goodness-of-fit. We reject the null hypothesis if the p-value is smaller than the confidence level of 0.05.
Table 4 presents the fraction of time-windows for which the null hypothesis was rejected, for a set of
future contracts, at different years and for different values of Δ. We also exclude from the analysis the
years prior to the introduction of electronic trading for each analysed commodity (see table 1), due to the
weak reliability of the corresponding quote timestamps when the open pit still existed.11 Those years are
marked with dash lines (---). The reliability of the timestamp (Δ) should be comparable with the typical
inter-package times. We have excluded from the analysis time periods where the median inter-package
times (table 3B) were larger than Δ. Those time intervals are marked in the table 4 with stars (***).


As can be seen from tables 4(A) to 4(E), the quality-of-fit is usually good. In total, for all the analysed
commodities over the period 2005–2012 for 10-minute intervals and with Δ=200 milliseconds (msec),
we reject 452’514 estimations out of a total of 3’332’016 estimations, corresponding to a rejection rate
of 13.6 per cent. For smaller Δ’s, the match of the Hawkes model with the data worsens: for Δ=100 msec,
we reject 576’532 out of 2’605’129 estimations, a rejection rate of 22.1 per cent; for Δ=50 msec, we reject
434’662 out of 1’165’761 estimations, a rejection rate of 37.3 per cent. By contrast, for Δ=300 msec, we
reject 303’420 out of 3’375’079 estimations, a rejection rate of 9.0 per cent; for Δ=1 second, we reject
only 70’126 out of 3’375’079 estimations, a rejection rate of 2.1 per cent. Due to the strong distortion of
the statistics of inter-event intervals occurring for small Δ’s and the generally poor match of the Hawkes
model with the data (as quantified by the Kolmogorov-Smirnov test), we have not presented results for
Δ<100 msec. However, we must acknowledge that the results obtained for Δ=50 msec lie within the
confidence intervals with those obtained for larger Δ’s. Despite very good match between model and data
for Δ=1 second, we will see that the use of such large Δ (which lasts 3–10 times more than the typical
waiting time between packages) results in a significant overestimation of the reflexivity index n.


Overall, these robust results support the choice of the exponential kernel (2) in the specification of the
Hawkes model (1). Let us stress that the results of the analysis within 10-minute windows are robust with
respect to the choice of the kernel. In particular, our tests have shown that using long memory (power
law) kernel results only in small biases in the estimations of the n, but does not change the overall secular
dynamics presented in the following sections.


From the calibration of the branching ratio n in each 10-minute interval, we introduce a robust measure
of the secular (long-term) evolution of the endogeneity in commodity markets, our “reflexivity” index, by


11 As discussed above, for sugar #11 we have additionally excluded the time period before 6 March 2008, when electronic
trading on the ICE platform coexisted with pit trading at NYBOT.


ξi= ∫ λt
t i
0


ˆ t) dt =μt i+ α ∑ expt j<t i


t i – t j
τ (




17


Table 4
Fraction of 10-minute windows rejected for the calibrations of


the Hawkes model for selected intervals (Δ), 2005–2012
(Per cent)


2005 2006 2007 2008 2009 2010 2011 2012 Total


(A) Δ=50 milliseconds


Brent (Europe) *** *** 32.6 41.3 31.4 31.4 *** *** 34.2
WTI (United States) --- *** *** *** *** *** *** 34.3 34.3
Soybeans (United States) --- *** *** *** *** 33.2 28.2 31.0 30.8
Sugar #11 (United States) --- --- --- *** *** 34.5 *** *** 34.5
Corn (United States) --- *** *** *** *** 33.4 48.9 29.9 38.5
Wheat (United States) --- *** *** *** *** 33.5 32.8 28.4 31.9
Sugar (Europe) *** *** *** *** *** *** *** *** ***
E-mini S&P 500 *** *** *** *** *** 45.5 60.8 37.0 49.0
Total *** *** 32.6 41.3 31.4 36.0 45.6 33.0 37.3


(B) Δ=100 milliseconds


Brent (Europe) *** *** 15.1 18.9 20.5 20.1 14.7 13.2 17.2
WTI (United States) --- *** 10.1 6.8 7.6 17.6 9.9 22.4 12.0
Soybeans (United States) --- *** *** 14.0 25.7 23.8 19.7 22.0 21.0
Sugar #11 (United States) --- --- --- *** 16.5 25.7 *** *** 21.5
Corn (United States) --- *** *** 28.3 *** 22.9 35.4 23.9 28.2
Wheat (United States) --- *** *** 26.5 30.3 25.7 24.9 20.7 26.2
Sugar (Europe) *** *** *** *** *** 36.1 38.1 42.7 38.1
E-mini S&P 500 *** *** 9.8 9.1 9.3 39.6 51.7 28.0 24.8
Total *** *** 11.6 15.6 16.9 27.4 29.0 24.5 22.1


(C) Δ=200 milliseconds


Brent (Europe) *** 24.3 6.3 12.1 15.0 13.0 5.3 5.2 11.4
WTI (United States) --- 6.4 3.8 3.1 3.2 8.7 5.9 14.2 6.1
Soybeans (United States) --- 3.4 8.3 8.8 13.6 14.1 11.7 12.1 11.0
Sugar #11 (United States) --- --- --- 12.0 9.9 16.8 30.0 26.5 19.2
Corn (United States) --- 5.7 7.4 16.0 10.2 11.8 18.2 15.5 13.1
Wheat (United States) --- 3.3 8.4 16.0 16.6 15.8 14.5 11.8 13.4
Sugar (Europe) *** 25.7 13.9 *** 13.4 22.4 26.1 33.0 23.6
E-mini S&P 500 1.2 10.0 7.1 4.1 4.5 30.7 34.3 19.5 14.4
Total 1.2 13.2 6.7 9.3 10.3 17.6 18.9 17.8 13.6


(D) Δ=300 milliseconds


Brent (Europe) 11.1 16.4 3.9 8.3 11.1 9.1 3.0 3.4 8.0
WTI (United States) --- 3.5 2.5 1.9 2.0 5.9 3.6 9.7 4.0
Soybeans (United States) --- 2.6 5.9 6.3 7.6 9.2 7.3 7.6 7.1
Sugar #11 (United States) --- --- --- 7.1 6.8 10.7 19.1 18.7 12.5
WTI (United States) --- 3.5 2.5 1.9 2.0 5.9 3.6 9.7 4.0
Wheat (United States) --- 2.3 5.6 10.0 9.5 10.1 8.7 7.4 8.3
Sugar (Europe) 16.9 24.6 10.5 7.8 9.5 14.1 18.7 28.5 16.3
E-mini S&P 500 0.9 8.8 6.7 2.5 2.7 23.6 32.7 14.1 12.4
Total 6.0 9.4 4.5 5.2 6.0 11.5 13.1 12.9 9.0


(E) Δ=1 second


Brent (Europe) 2.4 2.0 0.9 1.3 2.2 1.2 0.6 0.8 1.4
WTI (United States) --- 0.9 0.4 0.3 0.3 0.7 0.5 1.2 0.6
Soybeans (United States) --- 0.9 1.4 1.7 1.5 1.9 1.6 1.3 1.6
Sugar #11 (United States) --- --- --- 1.3 1.3 1.7 2.1 3.0 1.8
Corn (United States) --- 1.1 1.4 2.2 1.4 2.0 2.4 2.7 2.0
Wheat (United States) --- 0.6 1.1 1.1 1.5 1.8 1.4 1.2 1.3
Sugar (Europe) 10.6 15.1 2.7 1.7 2.1 1.8 4.6 16.8 4.6
E-mini S&P 500 0.3 1.8 2.2 0.5 0.3 6.0 8.0 4.0 3.1
Total 1.4 1.9 1.3 1.2 1.3 2.2 3.0 3.8 2.1


Source: Authors’ calculations, based on the TRTH database.
Note: TRTH timestamps were randomly redistribute around their recorded values within selected interval of duration (Δ). The


fraction of 10-minute windows was rejected with 95% confidence on the basis of the Kolmogorov-Smirnov test (see text).
--- corresponds to the years before the introduction of electronic trading for the given contract (see table 1). *** denotes
the years when the corresponding reliability of timestamps (Δ) is not applicable (see text).




18


calculating the average of n over all intervals in each calendar month. We also calculate the interquartile
range (IQR) of the branching ratio n within each month period, to assess its robustness.12


v. eStimation of the refLexivity index of commodity futureS priceS


A. The benchmark of financial markets: E-mini S&P 500 futures


Before presenting our analysis of the commodity markets, we revisit the analysis of Filimonov and
Sornette (2012), which originally used the E-mini S&P 500 futures contract data up to 29 August 2010,
between 1998m1 and 2012m10. Being introduced in 1997 as a supplement to the regular S&P 500 futures
contracts with a reduced size of 50 times the value of the index, the E-mini has attracted many small
investors and has become one of the most actively traded derivatives in the world. We take this contract
as a benchmark for the subsequent analyses of commodity futures contracts.


Figure 4(a) presents the two-month volume and trading activity (measured in number of mid-quote price
changes), while figure 4(b) shows daily volatility and price dynamics for the E-mini S&P 500 futures
contracts. Together with the dynamics of these traditional measures of activity, figures 4(c) and (d)
show, respectively, the dynamics of the estimated background intensity, μ̂ , and branching ratio, n̂ , over
the same time period. The estimates for each different Δ (100, 200, 300 milliseconds) are practically
undistinguishable. This observation together with the narrowness of the IQR confirm both the relevance
of the Hawkes model as an excellent data descriptor and the robustness of our estimation procedure. Both
observations will be later verified with the data analysis performed on commodities futures. Finally, let
us note that considering Δ=1 sec, which corresponds to the uncertainty of exchange timestamps, results
in slightly higher branching ratios, but does not change its overall dynamics.


Looking at the levels of these variables, figure 4 shows some changes in the number of mid-price
changes in panel (a); the daily volatility in panel (b); and the background intensity μ̂ in panel (c) exhibit
synchronized peaks that usually coincide with major episodes of market instabilities. Indeed, one of the
first peak coincides with the burst of the ICT dot-com bubble in early 2000 (Johansen and Sornette, 2000).
Note also the synchronized behaviour during the following bearish regime as well as during the financial
crisis that started in 2007, including its culmination with the Lehman Brothers bankruptcy (Sornette and
Woodard, 2010). Note that the increase of trading activity from 1998 to 2012, as proxied by volume in
figure 4(a), is not accompanied by an increase of the background intensity μ̂ of exogenous events in the
market. This makes intuitive sense since μ(t) should reflect the genuine news impacting the market.


In contrast with figures 4(a), (b) and (c), the time evolution of the branching ratio, ̂n , presented in figure 4(d)
exhibits a very different behaviour.13 Importantly, one should note that the branching ratio is not simply
another measure of trading activity or the frequency of price changes. Indeed, figure 4 illustrates the
existence of completely different dynamics of the branching ratio compared with measures of activity such
as volume or mid-quote price changes. We address this point in greater detail when we discuss several
robustness tests in section VI. For now, we only highlight the following findings:


12 Interestingly, when considering either all estimates within one month period, or only the estimates that could not be
rejected with the Kolmogorov-Smirnov test, the averages and the IQR intervals remain similar. However, to be consistent,
we have excluded from the averages those estimates for which the null hypothesis of the Hawkes model as the generating
process for the data could be rejected (corresponding to estimates with p-value below 0.05)
13 Note that the present analysis slightly differs from the one presented in Filimonov and Sornette (2012).




19


Figure 4
Selected dynamics of the E-mini S&P 500 futures, 1998m1–2012m10


Source: Authors’ calculations, based on Thomson Reuters Tick History (TRTH) database.
Note: Volatility was computed using daily prices with the Garman & Klass estimator. Each point in panels (c) and (d) represents


2-month average of 10-minute estimates. Due to timestamp uncertainties in TRTH, the timestamps were randomly
redistribute within four intervals of duration, Δ. Squares correspond to estimates with Δ = 100 msec. Crosses with black
line refer to Δ = 200 msec. Circles refer to Δ = 300 msec and dots with grey line refer to Δ = 1 sec. The shaded area
gives the interquartile range obtained with the same two months estimates for Δ = 200 msec. In the analysis we have
considered only estimates performed within hours of active trading (see table 1).


0


2


4


6


0


50


100


150


M
id


-q
uo


te
p


ric
e


ch
an


ge
s


(m
ill


io
n)


V
ol


um
e


(m
ill


io
n)


(a) Bimonthly volume and mid-quote price changes


Volume


Mid-quote price changes
(right scale)


500


1,000


1,500


2,000


0


0.05


0.1


0.15


P
ric


e


V
ol


at
ili


ty


(b) Daily closing price and volatility


Volatility


Price (right scale)


0.0


0.2


0.4


0.6


0.8


1.0


B
ac


kg
ro


un
d


ac
tiv


ity


(c) Bimonthly background intensity estimates (^µ, see text)


0.0


0.2


0.4


0.6


0.8


1.0


1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012


B
ra


nc
hi


ng
ra


tio


(d) Bimonthly branching ratio estimates (^n, see text)




20


(i) Between 1998 and 2004, the monthly14 trading volume increased almost 36 times (from 316’401
contracts in February 1998 to 11’428’371 contracts in February 2004). However the branching ratio
increased only slightly from 0.35 to 0.4 during this period.


(ii) Similarly, despite an almost doubling of the volume from 25’890’923 in June 2007 to 55’251’608
in August 2007, the branching ratio decreased from 0.6 to 0.45 during this period.


(iii) The same period a year later, June–August 2008, could be considered another example that illustrates
the decoupling between trading activity and branching ratio. The number of transactions increased
3 times (from 1’346’928 to 4’191’227) and the number of mid-quote price changes doubled (from
230’022 to 580’220) over the same period of time. Nevertheless, this did not lead to an increase of
the branching ratio, which, by definition, is estimated based on mid-quote price changes.


(iv) The dramatic surge of the branching ratio from 0.5 in September 2009 to 0.75 in March 2010 by
contrast, coincided with a moderate increase of volume (from 31’574’403 to 43’320’946) and of
mid-quote price changes (from 219’918 to 266’014).


(v) Finally, one could observe that the spike in volume and background activity in June 2010 did not
affect the branching ratio at all, and even a similar spike in September 2011 is associated with a
sharp decline of the ratio.


The branching ratio, measuring the level of endogeneity, increased regularly from 2002 onwards and
peaked first with the beginning of the bear market of the E-mini in 2007/2008. The successive policy
interventions led to a decrease in the level of endogeneity until August 2009. Afterwards, the branching
ratio rapidly rose again to reach a high plateau from the beginning of 2010 onwards with a small transient
decline at the end of 2011. This coincides with the discussion on the debt ceiling in the United States and
a deepening of the eurozone crisis. The decline of the endogeneity level between 2008 and mid-2009
coincides with a series of financial and economic interventions, when fundamentals, like liquidity provision
to avert a credit freeze on financial capital markets and stimulus packages to revive aggregate demand,
were prevalent features. Although it is beyond the scope of this paper to explain the effect of quantitative
easing policies on endogeneity, one can note that the sharp rise of the endogeneity level coincides with
the Fed’s announcement, in mid-2009, that it would purchase the full US$ 300 billion of Treasury coupon
securities eliminating the ambiguity of the “up to” language in the first announcement of March 2009
(see e.g. D’Amico and King, 2012). In parallel, animated discussions about the shape of the economic
recovery, either V, U, L or W, from mid-2009, added to economic uncertainties as characterized by the
risk-on/risk-off behaviours. These economic uncertainties as well as the quantitative easing policies have
remained prevalent in the subsequent years until 2012, while risks of credit freeze in financial capital
markets have receded substantially. The combination of economic uncertainties and unlimited liquidity
could rationalize the high plateau of the endogeneity level measured on the E-mini S&P 500 futures.


B. The evolution of endogeneity in commodity futures markets


In figures 5 and 7, we report our estimates of the branching ratio for several commodities. Similar to
the endogeneity level of the E-mini S&P 500 futures, we usually find the average levels of endogeneity
to exceed 50 per cent for all considered commodity markets since the middle of the first decade of the
21st century. Moreover, we observe that endogeneity levels are greater in 2012 than when our estimates
start. Nevertheless, these increases have not necessarily been monotonous. In the case of oil, both series,
Brent and WTI, show a gradual increase before reaching a peak in late-2008 to early-2009. Afterwards,


14 Note that here and further below we discuss monthly volumes and numbers of transactions, as well as average branching
ratios over one month interval. However, for the sake of clarity in figure 4 we plot dynamics of two-month values.




21


Figure 5
Prices, price volatility and branching ratio estimates in selected oil futures markets


Source: Authors’ calculations, based on Thomson Reuters Tick History (TRTH) database.
Note: Volatility was computed using daily prices with the Garman & Klass estimator. Branching ratio estimates (n̂) were


computed using the flow of mid-quote price changes. Each point represents 1-month average of 10-minute estimates.
Due to timestamp uncertainties in TRTH, the timestamps were randomly redistribute within four intervals of duration,
Δ. Squares correspond to estimates with Δ = 100 msec. Crosses with black line refer to Δ = 200 msec. Circles refer to
Δ = 300 msec and dots with grey line refer to Δ = 1 sec. The shaded area gives the interquartile range obtained with
the same two months estimates for Δ = 200 msec. In the analysis we have considered only estimates performed within
hours of active trading (see table 1).


0


50


100


150


0


0.05


0.10


0.15


P
ric


e


V
ol


at
ili


ty


(a) Brent crude (Europe), 2005m4–2012m9


Volatility


Price (right scale)


Daily closing prices and price volatility


0.3


0.5


0.7


0.9


B
ra


nc
hi


ng
ra


tio


Monthly branching ratio estimates (^n)


0


50


100


150


0


0.05


0.10


0.15


P
ric


e


V
ol


at
ili


ty


(b) WTI (United States), 2006m9–2012m9


Volatility


Price (right scale)


Daily closing prices and price volatility


0.3


0.5


0.7


0.9


2005 2006 2007 2008 2009 2010 2011 2012


B
ra


nc
hi


ng
ra


tio


Monthly branching ratio estimates (^n)


2005 2006 2007 2008 2009 2010 2011 2012




22


they have partly receded. By contrast, the level of endogeneity of the soft commodities exhibit a marked
oscillating pattern around a upward-sloped or constant long-term trend.


In addition, where our data go back to 2005 or 2006, we usually observe a period of about half a year
in the course of 2006 or 2007 when the branching ratio escalates sharply to higher levels from which it
does not recede. In particular, we observe this phenomenon for the Brent crude oil in figure 5(a) during
2006, when the monthly averages of its branching ratio move from roughly 0.4 to 0.6, and for the white
sugar futures market in Europe in figure 7(a) in the course of 2007, when this figure rose sharply from
about 0.3 to 0.55. These phenomena are similar to the pattern of the E-mini S&P 500 futures markets
over 2005 and 2006. Nevertheless, for the commodities, these episodes seem to have taken place over
a shorter time span.


As mentioned above, we cannot compute the branching ratios prior to the introduction of full electronic
trading and of sufficient liquidity in the middle of the first decade of the 21st century. Nevertheless, in all
likelihood, the levels of endogeneity in commodity markets in the late 1990s and in the early 2000s were
not greater than the one of the E-mini S&P 500 futures market at that time. This conjecture makes us
believe that the already-high endogeneity levels that we observe for all commodity markets in the second-
half of the first decade of the 21st century have not been a permanent feature in the period prior to the
introduction of full electronic trading and the availability of reliable tick data on commodity derivatives.


It should also be recalled that, at first sight, our reflexivity indices are not particularly designed to capture
longer-term herding mechanisms, which are responsible for bubble formation on time scales of months
(or even longer periods) . Since we need to calibrate the model in running windows of 10 minutes, our
branching ratio does by definition not have a long-term memory. However, we surprisingly find that the
mechanisms working at longer time scales sometimes seem to cascade down to the shorter intervals on
which we compute our indices. In specific cases discussed below, we show in fact that growing average
levels of endogeneity during several months sometimes coincide with bubble-bust cycles.


In addition, some common shocks seem to impact commodity markets simultaneously. For example:


• Among the commodities examined in this paper, many United States commodities exhibit a decline
of the branching ratio for the period around June/July 2011 (WTI, wheat, corn and soybeans). This
period coincides with discussions on the United States debt ceiling and fears that no agreement could
trigger a new worldwide economic downturn. It relates also to oil-reserve data releases by the IEA
Members and better than expected weather conditions in the United States.


• Interestingly, the second half of 2012 exhibits a sharp and synchronized increase of the branching
ratio for all United States traded commodities examined here, possibly on expectations of QE3.


• Oil, including Brent and WTI, seem to follow the same pattern as the E-mini from their endogeneity
peak of 2008 to the low of 2009, which coincides with the August 2009 hint of further quantitative
easing (see discussion above in section III.B)


• For the WTI, corn and wheat, one can observe a slight decline in the branching ratio in early 2008,
i.e. just before the bubble burst. In the United Sates, ethanol is mostly produced from corn and some
substitution effects (oil/corn and corn/wheat) could explain this common feature in the branching ratio.


After these general observations, we concentrate on each different commodity market specifically.


1. Crude oil: Brent and WTI


We start our analysis of commodity futures with oil prices, which exhibited a record rise followed by a
spectacular crash in 2008. The Brent daily close price peaked at US$ 146.08 (daily high of US$ 146.69)
per barrel on 3 July 2008. Six months later, Brent prices reached a trough of US$ 44 (daily low of
US$ 43.03) on 19 December 2008, a level not seen since 2004.




23


Figure 5 presents the dynamics of the price and of the daily volatility (estimated with the Garman & Klass
open-high-low-close estimator (Garman and Klass, 1980)) for two major futures contracts on light sweet
crude oil: Brent Crude, figure 5(a), and WTI, figure 5(b). Along with the price dynamics and the price
volatility, the evolution of the estimated branching ratio (effective degree of reflexivity) is presented.
Different symbols on the plots correspond to different values of Δ: 100, 200, 300 milliseconds and 1 second.


Figure 5 documents the following regimes for both Brent Crude (Europe) and WTI (United States):


• The branching ratio has shown an upward trend over the whole bubble period until early 2009, having
an intermediate peak in July 2008 coinciding with the end of the oil bubble.


• The branching ratio exhibited three large periods of stabilization, which were preceded by a small
drop: in Q1-2007, Q1-2008 and from mid-2008 to the end of 2008.


• In the last period of the run-up (December 2007–April 2008), the branching ratio showed a pronounced
drawdown for the WTI.


• The branching ratio started accelerating again until the price peak in July 2008.


• The branching ratio stayed high (at values of 0.7 for WTI and 0.75–0.78 for Brent) during the whole
period of the price fall until the bottom in December 2008, even exhibiting a maximum at the price
bottom higher than its previous peak reached on July 2008. This illustrates that the branching ratio
is also independent from the price trajectory, in addition to being unrelated to volume or mid-quotes
changes (see the discussion on the E-mini in section III.B).


• Thereafter, the branching ratio starts decreasing until mid-2009. Afterwards, the dynamics of the
branching ratio for Brent and WTI slightly diverged:


(i) The branching ratio for Brent was falling until December 2010 (see the note below on the sharp
fall) and then changed to a sideway trend.


(ii) The branching ratio for WTI was falling until mid-2009, and started rising again, peaked on
April–May 2010, before a sharp (but small) fall in May 2010 occurred. This is similar to the pat-
tern observed on the E-mini between mid 2009 to mid 2010 (see the discussion on the E-mini in
section III.B).


• The branching ratio sharply decreased for the Brent at the end of 2010 (also visible on the WTI)
although the price increased. This might be attributed to the unusual cold weather in Europe that lift
up oil demand unexpectedly and reduced herding mechanism temporarily.


• Beginning 2011, the branching ratio increased sharply for the WTI and also for the Brent following
the start of the Arab Spring and speculation on oil output of some producing countries like Libya.15


The most remarkable result obtained from the calibration of the branching ratio is its very large increase
during the period when oil prices started to accelerate. The fact that our methodology identifies a growing
reflexivity during the ascent of the price and, even more so, during its collapse, is particularly interesting
in view of other analyses that documented strong evidence for the existence of a bubble during that
period. Since the beginning of 2008, a growing number of specialists,16 bankers17 and academics18 were
considering the possibility that oil may have entered a bubble regime. The tormenting question was: how


15 It is interesting to note that while the cross-market correlations between these commodity futures and the United States
equity markets collapsed during that period and suggested a growing role of the fundamentals (Bicchetti and Maystre,
2012), the endogeneity levels on each specific market grew during that period.
16 See e.g. Zumbrun J, Soros tells congress to pop an oil bubble, Forbes, 3 June 2008.
17 Credit Suisse, The Investment Committee Meeting of 27 May 2008.
18 See e.g. Siegel J and Henisz W, What’s Behind the Flare-ups in Oil Prices? Jeremy Siegel and Witold Henisz Weigh
In, Knowledge@Wharton, 28 May 2008; also see Krugman P, More on oil and speculation (The Conscience of a Liberal),
The New York Times, 13 May 2008.




24


to justify the quadrupling of oil prices since 2003? Some attributed it mainly to the growing demand
from China and India, a claim that former Chinese President Jiang Ze-Min himself debunked at least for
China (see figure 2 with caption in English in Jiang, 2008). Comparing the values on world liquid fuel
supply and demand reported by the International Energy Agency (IEA) and by the United States Energy
Information Administration (EIA), Sornette et al. (2009) noted that, until the end of 2005, both agencies
reported consistent numbers showing that supply was systematically exceeding demand. Since 2006, there
was a significant discrepancy between the numbers presented by the two agencies, ushering a period of
uncertainty or opaque reporting, with no clear conclusion on whether an excess demand versus supply
was the cause of the appreciation of oil prices. One can argue that the lack of clarity of the oil supply-
demand situation during that period ushered in a period of growing speculation both in the literary sense
of “forming conjectures” and in the financial sense, based on the general fact that the more imprecise is the
estimation of the fundamental value of an asset, the more room there is for “stories” and “new economy”
thinking that can justify speculative bubble prices (Kindleberger and Aliber, 2005; Sornette, 2003).


Sornette et al. (2009) further support the hypothesis that the 2007–2008 oil price run-up was amplified
by speculative behaviours of the type found during a bubble-like expansion. They analysed oil prices
in United States dollar and in other major currencies and found clear evidence of speculation. Based
on the mechanism of positive feedbacks and the concept of emergent phase transitions (or bifurcation)
to another regime, using analogies with statistical physics and complexity theory, Sornette et al. (2009)
used an approach that diagnoses bubbles as transient super-exponential regimes (Sornette, 2003). In a
nutshell, the methodology aims at detecting the transient phases where positive feedbacks operating on
some markets or asset classes create local unsustainable price run-ups. The mathematical signature of
these bubbles is a log-periodic power law (LPPL) (see e.g. Sornette and Johansen, 1998; Johansen and
Sornette, 1999; Johansen et al. 2000; Sornette and Johansen, 2001). The power law finite-time singular
process models the faster-than-exponential growth culminating in finite time at some critical time . The
log-periodic oscillations reflect hierarchical structures (Johansen and Sornette, 1999; Johansen et al., 2000)
as well as competition between the trading dynamics of fundamental value and momentum investors (Ide
and Sornette, 2002). Figure 6 shows the calibration of the LPPL model to the oil price (NYMEX Light


Figure 6
Price time series of NYMEX Light Sweet Crude (front-month contract) and


simple log-periodic power law (LPPL) fits


Note: The shaded box shows the 80 per cent confidence interval of the forecast performed at the time indicated by the vertical
dashed line. See Sornette et al. (2009) for details.


N
Y


M
E


X
L


ig
ht


S
w


ee
t C


ru
de


, U
S


$


Date
2002
0


50


100


150


200


250


50


100


150


200


250


300


2003 2004 2005


March 2008


Actual peak date
(3 July 2008)


Original analysis date
(27 May 2008)


Shaded region is
80% confidence interval


of the projected peak date


September 2008


2006 2007 2008 2009




25


Sweet Crude, Contract 1, from the Energy Information Administration of the United States Government).19
The shaded box shows the 80 per cent confidence interval of the critical time tc indicating the end of the
bubble. Note that this analysis was performed ex-ante before the oil price did peak and was presented as
a genuine real-time prediction, which turned out to be successful, as recounted by Sornette et al. (2009).
The overall conclusion of this analysis is that the geopolitical events that unfolded in 2007 and 2008 have
participated in raising the level of uncertainty, which worked as a fertilizer for speculation, leading to oil
prices increasingly decoupling from fundamental valuation (the hallmark of a bubble).


Our present analysis of the effective degree of reflexivity estimated with high-frequency data, summarized
in figure 5, has shown that, during the bubble period, herding between investors existed not only at scales
of years but was also accompanied by short-term herding of the algorithmic trading strategies. Combining
the evidence of figures 5 and 6, we conclude that the positive feedback mechanisms working at large
time scales, which are at the origin of the oil bubble, sometimes cascaded down to the minute time scales
and were reflected in the abnormal increase of the branching ratio that occurred concomitantly with the
development of the bubble and its burst. Such cascade processes (Arneodo et al., 1998) are captured by the
concept of multifractality that has been found to provide a remarkably powerful description and predictor
of asset return dynamics (Muzy et al., 2001; Calvet and Fisher, 2008; Sornette et al., 2003; Lux, 2008).


2. Soft commodities: soybeans, sugar, corn and wheat


(a) Sugar (Europe and the United States)


One can distinguish four main regimes in the dynamics of the branching ratio for sugar (Europe) shown
of figure 7(a):


• Before 2007, the branching ratio is hovering around 0.3, with a rather large standard deviation due to
the limited size of the data set resulting from relatively low trading activity.


• Starting at the beginning of 2007, the branching ratio increases rapidly and doubles in less than three
quarters, stabilizing around the value 0.5–0.6 in the third quarter of 2007.


• From the fourth quarter of 2007 till mid 2011, the branching ratio is practically stable and remains in
the range 0.5–0.6 with some excursions higher up.


• From mid-2011 to the end of 2011, the branching ratio increases and passes over 0.75. Thereafter, it
decreases but remains in the range 0.6–0.7.


The rough pattern is similar for sugar #11 shown in figure 7(b), except that (i) the story is shorter, (ii) there
is a divergence in endogeneity dynamics with respect to sugar (Europe) in spring 2009: sugar (United
States) experiences a spike in endogeneity levels from around 0.55 to 0.7. The peak of the branching
ratio observed for sugar (Europe) at the very end of 2011 is absent for sugar #11 and its dynamics in
2012 is characterized by a monotonous increase of the branching ratio in the second part of 2012 to the
extremely high level of 0.8 (possibly on expectations of QE3 discussed above).


The divergence in endogeneity dynamics between the two benchmarks cannot be fully resolved with our
model. For instance, our model does not allow fully exploring the potential of cross-market endogeneity.
However, by reviewing some common features as well as some differences between the two benchmarks,
we suggest some leads for further research.


19 This figure was reproduced from figure 5 in Sornette et al. (2009).




26


Figure 7
Prices, price volatility and monthly branching ratio in


selected agricultural futures markets, 2005m1–2012m9


Source: Authors’ calculations, based on Thomson Reuters Tick History (TRTH) database.
Note: Volatility was computed using daily prices with the Garman & Klass estimator. Branching ratio estimates (n̂) were


computed using the flow of mid-quote price changes. Each point represents 1-month average of 10-minute estimates.
Due to timestamp uncertainties in TRTH, the timestamps were randomly redistribute within four intervals of duration,
Δ. Squares correspond to estimates with Δ = 100 msec. Crosses with black line refer to Δ = 200 msec. Circles refer to
Δ = 300 msec and dots with grey line refer to Δ = 1 sec. The shaded area gives the interquartile range obtained with
the same two months estimates for Δ = 200 msec. In the analysis we have considered only estimates performed within
hours of active trading (see table 1).


2005 2006 2007 2008 2009 2010 2011 2012


0


200


400


600


800


1,000


0


0.02


0.04


0.06


0.08


0.10


P
ric


e


V
ol


at
ili


ty


(a) Sugar (Europe)


Volatility


Price (right scale)


0.2


0.4


0.6


0.8


B
ra


nc
hi


ng
ra


tio


Monthly branching ratio estimates (^n)


0


10


20


30


40


50


0


0.02


0.04


0.06


0.08


0.10


P
ric


e


V
ol


at
ili


ty


(b) Sugar #11 (United States)


Volatility


Price (right scale)


Daily prices and price volatility


0.4


0.5


0.6


0.7


0.8


0.9


2005 2006 2007 2008 2009 2010 2011 2012


B
ra


nc
hi


ng
ra


tio


Monthly branching ratio estimates (^n)


Daily prices and price volatility




27


Figure 7 (continued)
Prices, price volatility and monthly branching ratio in


selected agricultural futures markets, 2005m1–2012m9


Source: Authors’ calculations, based on Thomson Reuters Tick History (TRTH) database.
Note: Volatility was computed using daily prices with the Garman & Klass estimator. Branching ratio estimates (n̂) were


computed using the flow of mid-quote price changes. Each point represents 1-month average of 10-minute estimates.
Due to timestamp uncertainties in TRTH, the timestamps were randomly redistribute within four intervals of duration,
Δ. Squares correspond to estimates with Δ = 100 msec. Crosses with black line refer to Δ = 200 msec. Circles refer to
Δ = 300 msec and dots with grey line refer to Δ = 1 sec. The shaded area gives the interquartile range obtained with
the same two months estimates for Δ = 200 msec. In the analysis we have considered only estimates performed within
hours of active trading (see table 1).


0


500


1,000


1,500


2,000


0


0.02


0.04


0.06


0.08


P
ric


e


V
ol


at
ili


ty


(c) Soybeans (United States)


Volatility


Price (right scale)


Daily prices and price volatility


0.3


0.4


0.5


0.6


0.7


0.8


2005 2006 2007 2008 2009 2010 2011 2012


B
ra


nc
hi


ng
ra


tio


Monthly branching ratio estimates (^n)


0


200


400


600


800


0


0.02


0.04


0.06


0.08


P
ric


e


V
ol


at
ili


ty


(d) Corn (United States)


Volatility


Price (right scale)


0.3


0.4


0.5


0.6


0.7


0.8


2005 2006 2007 2008 2009 2010 2011 2012


B
ra


nc
hi


ng
ra


tio


Monthly branching ratio estimates (^n)


Daily prices and price volatility




28


First of all, the quality of the underlying is different: sugar #11 is the world benchmark for raw sugar,
while sugar (Europe) is the world benchmark for refined or white sugar.20 Moreover, until mid-2012,
the sugar (Europe) closing time was ahead of the New York closing time by about one and half hours.
Sometimes, to catch up with the evolution of the New York price of the preceding day, this resulted in
some price spikes on the opening. After controlling for different hours of trading activity, we find that
these spikes, which result from the arbitrage trade between the two benchmarks, cause a decline in the
endogeneity level on sugar (Europe) but do not influence the endogeneity dynamics. By construction of
the branching ratio, arbitrage trades are captured as being exogenous although they might be the result of
cross-market endogeneity.21 Yet, these difference in dynamics remain puzzling, despite the difference in


20 Although the European-based derivative is traded in London, its price is not influenced by the quotas imposed by the
European Union.
21 We cannot exclude that cross market endogeneity might induce price changes in another fundamentally related market
although these prices changes may not be triggered by an exogenous event. This raises the possibility of a contagion effect
from highly financialized markets to other more fundamental driven markets, which would become in turn less efficient
than otherwise. However, this working hypothesis is beyond the research of this current study, and we currently unable
to estimate the impact of cross-excitation. Our estimates of endogeneity then is even more conservative as they do not
take into account the possible contagion effects. Indeed, the model will consider each event, that was triggered due to the
cross-excitation with a coupled marked, as exogenous; which results in overestimation of the background (exogenous)
activity and underestimation of the overall endogenous dynamics.


Figure 7 (concluded)
Prices, price volatility and monthly branching ratio in


selected agricultural futures markets, 2005m1–2012m9


Source: Authors’ calculations, based on Thomson Reuters Tick History (TRTH) database.
Note: Volatility was computed using daily prices with the Garman & Klass estimator. Branching ratio estimates (n̂) were


computed using the flow of mid-quote price changes. Each point represents 1-month average of 10-minute estimates.
Due to timestamp uncertainties in TRTH, the timestamps were randomly redistribute within four intervals of duration,
Δ. Squares correspond to estimates with Δ = 100 msec. Crosses with black line refer to Δ = 200 msec. Circles refer to
Δ = 300 msec and dots with grey line refer to Δ = 1 sec. The shaded area gives the interquartile range obtained with
the same two months estimates for Δ = 200 msec. In the analysis we have considered only estimates performed within
hours of active trading (see table 1).


0


500


1,000


1,500


0


0.05


0.10


0.15


P
ric


e


V
ol


at
ili


ty


(e) Wheat (United States)


Volatility


Price (right scale)


0.3


0.4


0.5


0.6


0.7


0.8


2005 2006 2007 2008 2009 2010 2011 2012


B
ra


nc
hi


ng
ra


tio


Monthly branching ratio estimates (^n)


Daily prices and price volatility




29


quality and trading hours during the observed period, because both derivatives prices are highly correlated.
Finally, the fluctuation of the branching ratio in 2010–2012, which is not synchronized between Europe
and United States, deserves a closer consideration beyond the scope of this paper. Nevertheless, the New
York market is much deeper than London, as volumes can be 12 times more important (see table 2). As a
result, New York is the favourite venue for high-frequency traders and algorithmic funds. This may also
explain the lower endogeneity level observed in London. However, at the moment, to what the extent
these points might result as possible explanations for the different endogeneity dynamics is left as an
open question for future research.


(b) Corn, soybeans and wheat (United States)


The dynamics of the prices and volatilities of soybeans (United Sates), corn (United Sates), and wheat
(United Sates) shown of figures 7(c), 7(d) and 7(e), respectively, share many features. They exhibit a very
large peak in mid-2008, only surpassed by very recent price surges for corn and soybeans, followed by
price declines until mid-2010. These peaks in mid-2008 coincide with the peak of oil prices, discussed
above, and are symptomatic of the commodity bubble that developed in 2007 and 2008. One can also
notice a precursory peak in the first quarter of 2008, which is especially pronounced for soybeans (United
Sates) and is actually a dominant price feature for wheat.


Interestingly, in contrast to the behaviour of the branching ratio for oil, whose increase accompanied the
growth of the oil price bubble, the branching ratios for these commodities remained in the range 0.5–0.6,
with some spikes before the price peak and spikes associated with the price correction following the peak
in mid-2008. However, given that the branching ratio could not be computed prior to the end of 2006 for
these commodities and based on the branching ratio measured on the E-mini S&P 500 and sugar (Europe),
one can reasonably assume that the reflexivity level is likely to be around 0.3–0.4 for the earlier years.
Therefore, the level measured just before the commodity bubble burst is already relatively high and does
not exhibit the jump seen on the Brent or the E-mini in early 2006.


The branching ratios of corn, soybeans and wheat remained approximately constant after this bubble
episode, in the range 0.4–0.6 from mid-2009 to mid-2010 for soybeans and wheat and in the range
0.5–0.65 for corn from mid-2008 to mid-2010. The branching ratios of both soybeans and corn exhibited
a sharp increase from 0.5 to 0.6 for soybeans and even to 0.7 for corn and wheat, in the third quarter of
2010, which can be associated with a change to a phase of rising prices. In fact, in the summer of 2010,
Russia announced an export ban for wheat and Ukraine followed by announcing export restrictions.22
Although the export ban is a fundamental event, the increase in the endogeneity level is concomitant to the
increased interest of financial speculators (see UNCTAD, 2011, for an explanation of fundamental shocks
raising an increasing interest from financial speculators). As wheat, soybeans and corn are substitutes
for feed grain, the export restrictions created a shock that impacted simultaneously these markets. The
substitutability between these commodities through their use as feedstock creates a correlation between
them, in addition to raising financial speculation interests (UNCTAD, 2011).23


Thereafter, the branching ratio for soybeans showed a steady decline from 0.6 to slightly above 0.5 in
October 2012. In contrast, the branching ratio for corn exhibited much larger volatility with drops down
to 0.4 and peaks up to 0.6 from 2011 to September 2012.


22 See Reuters article “Snap analysis – Race for Russia’s grain business after export ban” (http://www.reuters.com/
article/2010/08/05/uk-russia-grain-export-ban-idUKTRE6744E720100805) and BBC article “Ukraine sets grain export
quotas following drought” (http://www.bbc.co.uk/news/business-11495369).
23 See also: http://www.bloomberg.com/news/2010-08-08/wheat-speculators-slowing-bets-on-more-gains-after-best-
month-since-1973.html and footnote 21 about contagion effect.




30


vi. robuStneSS teStS


The branching ratio represents a standalone measure of reflexivity, which is not affected by a simple
increase or decrease of trading activity (measured by the number of transactions or volume) or by price
changes. As discussed in section IV, the input for the calibration of the Hawkes model is the series of
timestamps of mid-quote price changes, independently of their directions. Thus, the branching ratio
is insensitive to the presence and direction of trends, whether the price is rising, falling or moving
sideways. An increase of the branching ratio qualifies an increase of self-excitation in the price formation
mechanisms and, as explained in section IV.A, could signal the development of short-term instabilities
and of incoming crises.


Similarly to the effect of the direction of price moves, neither transactions nor volume enter directly
into the formulation of the Hawkes model, since individual transactions do not necessary result in a
change of the mid-price. As an example, doubling the number of transactions by splitting each of them
into two independent transactions (to keep the daily volume constant) does not affect the dynamics of
the mid-price at all. Similarly, keeping the number of transactions constant and doubling the volume of
each of them (doubling the volume of each incoming market order) while simultaneously doubling the
volume of all incoming limit orders again would not change the dynamics of the mid-price. The number
of events (mid-quote price changes) also does not directly affect the parameters of the model except for
the background intensity parameter μ. For instance, doubling the number of events by superimposing two
identical clusters from figure 2 (or two clusters generated with identical parameter set (n,τ)) will result
in doubling the background intensity μ that quantifies the rate of exogenous (zero-order) events in the
system, but the branching ratio n will not change.


The above is theory, but does it hold in practice, in particular in the statistical estimation of the branching
ratio with limited data and for different parameters? In order to reject the possibility that the observed
dynamics of the branching ratio could reflect an increase of trading activity, we perform the following
test. Fixing the number of mid-quote price changes per day, we redistribute these events in time such
that, within one day, their dynamics was described by a Poisson process. This “redistribution” of the
time series amounts to keeping the price trajectories, the daily volume, the number of price and mid-
quote price changes per day unchanged, i.e. keeping the same trajectories as shown in figures 5 and 7
while distorting time such that the intervals between consecutive mid-quote price changes within one
day become uncorrelated and exponentially distributed. Then, we perform exactly the same procedure
as described in section IV. Namely, we divide each day in 10-minute intervals, rounded timestamps to
nearest end of sub-interval of size Δ of 100, 200, 300 milliseconds and 1 second (which would correspond
to introducing uncertainty in timestamps). Then, we implement the procedure described in section IV
and estimated the parameters (μ, n, τ) of the Hawkes process (1) with an exponential kernel (2) within
each of these intervals.


Figure 8 presents the results of this robustness test. Despite the increase of activity (measured, for instance,
in the number of transactions) and the increase of trading volume (see table 2), as well as the existence
of a highly nontrivial seasonal volume dynamics (see figure 1), the random shuffling of the time stamps
have completely erased the self-excited structure of the time series. Indeed, the estimated branching ratio
in the randomized time series is consistently found very small, as it should. Its average and median values
are always n̂ ≲0.08 and the 75 per cent quantile is about 0.1. One can thus clearly reject the hypothesis
that the branching ratio is sensitive to, or equivalently provides another measure of, trading activity and
trading volume. This quantitative result supports the key property of the Hawkes model, which is that
the branching ratio is not determined by the average rate of events but by the degree of self-excitation
of the system.




31


Figure 8
Monthly branching ratios computed on randomly redistributed mid-quote price changes in


selected commodity futures markets


Source: Authors’ calculations, based on Thomson Reuters Tick History (TRTH) database.
Note: Monthly branching ratios were computed on randomly redistributed flows of mid-quote price changes. Each point represents


a 1-month average of 10-minute estimates. In addition, the randomly redistributed timestamps were a second time randomly
redistribute within four intervals of duration, Δ. Squares correspond to estimates with Δ = 100 msec. Crosses with black
line refer to Δ = 200 msec. Circles refer to Δ = 300 msec. The shaded area gives the interquartile range obtained with
the same two months estimates for Δ = 200 msec. In the analysis we have considered only estimates performed within
hours of active trading (see table 1).


Brent crude (Europe), 2005m8–2012m4


0


0.1


0.2


WTI (United States), 2006m9–2012m4


0


0.1


0.2


Soybeans (United States), 2006m7–2012m4


0


0.1


0.2


Sugar (United States), 2006m7–2012m4


0


0.1


0.2


Corn (United States), 2008m3–2012m4


0


0.1


0.2


Wheat (United States), 2006m8–2012m4


0


0.1


0.2


2005 2006 2007 2008 2009 2010 2011 2012
0


0.1


0.2
Sugar (Europe), 2005m1–2012m4




32


vii. concLuSion


We have analysed the evolution of the degree of short-term endogeneity for commodity futures markets
since the second half of the first decade of the 21st century, when the selected commodity exchanges
moved from pit trading to full electronic platforms. On average, our estimates show that more than one
out of two price changes is triggered by another preceding price change, which can be interpreted as the
actions of “self-reinforcing” mechanisms. In particular cases, we have documented excursions of the
reflexivity index up to levels in the range of 80–85 per cent and higher, as for example in Brent and WTI
futures over the run-up and crash periods of 2008; sugar #11 contract in the second half of 2012 or the
benchmark E-mini contracts over 2010–2012. As discussed, even these high estimates are conservative,
since they take into account only mechanisms that operate at scales below ten minutes and neglect
contagion effects across assets.


Our robustness tests show that our measure of endogeneity is independent of other factors that have also
experienced significant changes over the last decade. More importantly, it is also independent of the
background intensity of exogenous events in these markets. Interestingly, we do not observe a long-term
increase of this variable parallel to the developments that we observe for the other variables, such as
transaction volumes. This suggests that the rate of genuine news impacting the market, reflected in our
measure of background intensity, has remained relatively constant over the analysed period.


As discussed in section III.A, the documented high levels of endogeneity are likely to make the price
formation process less efficient, because higher endogeneity implies a slower convergence towards
fundamental (equilibrium) value. Moreover, high levels of endogeneity imply the existence of a large
susceptibility of the system with respect to external shocks. When the price discovery process is dominated
by the internal feedback mechanisms (as opposed to external novel information), this endogenous trading
may “amplify” even a small initial shock to the system, turning it into the large crash or rally. The most
illustrative example is indeed the “flash-crash” of May 6, 2010, when an initial large market order triggered
a flurry of activities by high-frequency traders resulting in a large drop of the price of S&P 500 E-mini
futures. As shown in Filimonov and Sornette (2012), this event was accompanied (and even preceded)
by a fast increase of the branching ratio to the 95 per cent level.


Over the past few years, in part due to the development of algorithmic and HFT, mini “flash-crashes” and
anomalous events regularly shock markets across the world, and commodity markets are no exception.
Considering just the WTI futures contracts over the single year of 2012, several wild excursions were
experienced, such that price plunged/spiked with swings of 1–3 per cent in a matter of seconds or minutes.24
Our preliminary systematic unpublished analysis of such events using the methodology of the present
article indicates that many of them share the same footprint as observed in the dynamics of the reflexivity
index as shown during the E-mini “flash crash” event analysed in Filimonov and Sornette (2012).


While our reflexivity index does not have a long-term memory, interestingly, we found that it can still
provide some interesting insights when the mechanisms working at longer time scales cascade down to
shorter terms, as occurred during the oil bubble that culminated in mid-2008 and deflated until early-2009.
At that time, according to our analysis, the monthly average of short-term endogeneity of oil futures
market reached a level that was achieved by the financial market (proxied by the E-mini contracts) only
two years later in 2010.


Summarizing, our results suggest that the financialization of commodity markets has not been accompanied
with an increase of market efficiency compared with the period preceding the rise of electronic trading.
Despite significantly increasing trading and market participation, we have observed signatures that suggest


24 The most dramatic, yet still not fully understood, drop was on 17 September 2012 when the WTI price fell 3.1 per
cent in 50 seconds and then rebounded in a way similar to the E-mini dynamics during its “flash crash” event.




33


that the price discovery process has become in fact less efficient, in stark contrast to the usual postulated
hypothesis. Our findings therefore question the rationale for expanding a market participation base, as
happened with the deregulation of the commodity market.


referenceS


Aït-Sahalia Y, Mykland PA and Zhang L (2005). How often to sample a continuous-time process in the presence
of market microstructure noise. Review of Financial Studies, 18(2): 351–416.


Aite Group (2009). New World Order: The High Frequency Trading Community and Its Impact on Market Structure.
Aite group.


Almgren R (2012). High-Frequency Event Analysis in Eurex Interest Rate Futures. Working paper, Quantitative
Brokers.


Andersen TG, Bollerslev T, Diebold FX and Vega C (2003). Micro effects of macro announcements: Real-time
price discovery in foreign exchange. American Economic Review, 93(1): 38–62.


Arneodo A, Muzy JF and Sornette D (1998). “Direct” causal cascade in the stock market. The European Physical
Journal B, 2(2): 277–282.


Azizpour S, Giesecke K and Schwenkler G (2011). Exploring the sources of default clustering. Working paper,
Stanford University.


Bauwens L and Hautsch N (2009). Modelling financial high frequency data using point processes. In: Mikosch T,
Kreiss, JP, Davis RA and Andersen TG, eds. Handbook of Financial Time Series. Berlin, Springer: 953–976.


Bicchetti D and Maystre N (2012). The synchronized and long-lasting structural change on commodity markets:
evidence from high frequency data. UNCTAD Discussion Paper No. 208, UNCTAD, Geneva. Published as:
Bicchetti D and Maystre N (2013). The synchronized and long-lasting structural change on commodity markets:
evidence from high frequency data. Algorithmic Finance, 2(3-4). DOI 10.3233/AF-13028.


Black F (1986). Noise. The Journal of Finance, 41(3): 529–543.
Bouchaud JP, Farmer JD and Lillo F (2009). How markets slowly digest changes in supply and demand. In: Handbook


of Financial Markets: Dynamics and Evolution. Amsterdam, North Holland: 57–160.
Bowsher CG (2002). Modelling security market events in continuous time: Intensity based, multivariate point


process models. Nuffield College Economics Discussion Papers.


Bowsher CG (2007). Modelling security market events in continuous time: Intensity based, multivariate point
process models. Journal of Econometrics, 141(2): 876–912.


Cai J, Cheung YL and Wong MCS (2001). What moves the gold market? Journal of Futures Markets, 21(3): 257–278.
Calvet LE and Fisher AJ (2008). Multifractal Volatility: Theory, Forecasting, and Pricing. Amsterdam, Academic


Press-Elsevier.


Chavez-Demoulin V, Davison AC and McNeil AJ (2005). Estimating value-at-risk: A point process approach.
Quantitative Finance,5(2): 227–234.


Christie-David R, Chaudhry M and Koch TW (2000). Do macroeconomics news releases affect gold and silver
prices? Journal of Economics and Business, 52(5): 405–421.


Cont R (2011). Statistical modeling of high frequency financial data: Facts, models and challenges. IEEE Signal
Processing Magazine, 28(5): 16–25.


D’Amico S and King T (2012). Flow and stock effects of large-scale asset purchases: evidence on the importance
of local supply. Staff working papers in the Finance and Economics Discussion Series. Divisions of Research
& Statistics and Monetary Affairs. Washington, DC, Federal Reserve Board.


Daley DJ and Vere-Jones D (2008). An Introduction to the Theory of Point Processes. Volume II: General theory
and structure, 2nd Edition. New York, Springer.


Domanski D and Heath A (2007). Financial investors and commodity markets. BIS Quarterly Review, 5: 1–15.
Duhigg C (2009). Traders profit with computers set at high speed. The New York Times, 24 July. Available at: http://


www.nytimes.com/2009/07/24/business/24trading.html.




34


Eisler Z and Kertész J (2006). Size matters: Some stylized facts of the stock market revisited. The European Physical
Journal B, 51(1): 145–154.


Elder J, Miao H and Ramchander S (2012). Impact of macroeconomic news on metal futures. Journal of Banking
& Finance, 36(1): 51–65.


Embrechts P, Liniger T and Lu L (2011). Multivariate Hawkes processes: An application to financial data. Journal
of Applied Probability, 48(A): 367–378.


Engle RF (2000). The econometrics of ultra-high-frequency data. Econometrica, 68(1): 1–22.


Engle RF and Russell JR (1997). Forecasting the frequency of changes in quoted foreign exchange prices with the
autoregressive conditional duration model. Journal of Empirical Finance, 4(2-3): 187–212.


Engle RF and Russell JR (1998). Autoregressive conditional duration: A new model for irregularly spaced transaction
data. Econometrica, 66(5): 1127–1162.


Errais E, Giesecke K and Goldberg LR (2010). Affine point processes and portfolio credit risk. Journal on Financial
Mathematics, 1(1): 642–665.


Fama EF (1970). Efficient capital markets: A review of theory and empirical work. The Journal of Finance, 25(2):
383–417.


Fama EF (1991). Efficient capital markets: II. Journal of Finance, 46(5): 1575–1617.


Filimonov V and Sornette D (2012). Quantifying reflexivity in financial markets: Toward a prediction of flash
crashes. Physical Review E , 85(5): 056108.


Frenk D and Turbeville W (2011). Commodity index traders and boom/bust in commodities prices. Better Markets
working papers.


Garman MB and Klass MJ (1980). On the estimation of security price volatilities from historical data. The Journal
of Business, 53(1): 67–78.


Gilbert CL (2007). Speculative influences on commodity futures prices. UNCTAD Discussion Paper No. 197,
UNCTAD, Geneva.


Glosten LR and Milgrom PR (1985). Bid, ask and transaction prices in a specialist market with heterogeneously
informed traders. Journal of Financial Econometrics, 14(1): 71–100.


Green TC (2004). Economic News and the Impact of Trading on Bond Prices. The Journal of Finance, 59(3):
1201–1233.


Harris TE (2002). The Theory of Branching Processes. Mineola NY, Dover Phoenix Editions.


Hasbrouck J (1991). Measuring the information content of stock trades. The Journal of Finance, 46(1): 179–207.


Hawkes AG (1971). Point spectra of some mutually exciting point processes. Journal of the Royal Statistical Society.
Series B (Methodological), 33(3): 438–443.


Helmstetter A and Sornette D (2003). Importance of direct and indirect triggered seismicity in the ETAS model of
seismicity. Geophysical Research Letters, 30(11): 1576.


Hess D, Huang H and Niessen A (2008). How do commodity futures respond to macroeconomic news? Financial
Markets and Portfolio Management, 22(2): 127–146.


Hewlett P (2006). Clustering of order arrivals, price impact and trade path optimisation. In Workshop on Financial
Modeling with Jump processes, Ecole Polytechnique.


Iati R (2009). The real story of trading software espionage. The TABB Group. Available at: http://www.
advancedtrading.com/algorithms/the-real-story-of-trading-software-espio/218401501


Ide K and Sornette D (2002). Oscillatory finite-time singularities in finance, population and rupture. Physica A:
Statistical Mechanics and its Applications, 307(1-2): 63–106.


Irwin SH and Sanders DR (2012). Financialization and structural change in commodity futures markets. Journal
of Agricultural and Applied Economics, 44(3): 371–396.


Ivanov PCC, Yuen A, Podobnik B and Lee Y (2004). Common scaling patterns in intertrade times of U. S. stocks.
Physical Review E, 69(5): 056107.


Jiang ZM (2008). Reflections on energy issues in China. Journal of Shanghai Jiaotong University (Science), 13(3):
257–274.




35


Jiang ZQ, Chen W and Zhou WX (2009). Detrended fluctuation analysis of intertrade durations. Physica A: Statistical
Mechanics and its Applications, 388(4): 433–440.


Johansen A, Ledoit O and Sornette D (2000). Crashes as critical points. International Journal of Theoretical and
Applied Finance, 3(2): 219–255.


Johansen A and Sornette D (1999). Critical crashes. Risk, 12(1): 91–94.


Johansen A and Sornette D (2000). The Nasdaq crash of April 2000: Yet another example of log-periodicity in a
speculative bubble ending in a crash. The European Physical Journal B - Condensed Matter and Complex
Systems, 17(2): 319–328.


Kindleberger CP and Aliber R (2005). Manias, Panics, and Crashes: A History of Financial Crises (5th Edition).
Hoboken NJ, Wiley.


Kyle AS (1985). Continuous auctions and insider trading. Econometrica, 53(6): 1315–1335.


Kyle AS and Obizhaeva AA (2012). Large bets and stock market crashes. College Park Working Paper.


Large J (2007). Measuring the resiliency of an electronic limit order book. Journal of Financial Markets, 10(1): 1–25.


Lux T (2008). The Markov-switching multifractal model of asset returns: GMM estimation and linear forecasting
of volatility. Journal of Business and Economic Statistics, 26(2): 194–210.


Marsan D and Lengline O (2008). Extending earthquakes’ reach through cascading. Science, 319(5866): 1076–1079.


Muzy JF, Sornette D, Delour J and Arneodo A (2001). Multifractal returns and hierarchical portfolio theory.
Quantitative Finance, 1(1): 131–148.


Ogata Y (1978). The asymptotic behaviour of maximum likelihood estimators for stationary point processes. Annals
of the Institute of Statistical Mathematics, 30(1): 243–261.


Ogata Y (1988). Statistical models for earthquake occurrences and residual analysis for point processes. Journal
of the American Statistical Association, 83(401): 9–27.


Oswiecimka P, Kwapien J and Drozdz S (2005). Multifractality in the stock market: price increments versus waiting
times. Physica A: Statistical Mechanics and its Applications, 347: 626–638.


Ozaki T (1979). Maximum likelihood estimation of Hawkes’ self-exciting point processes. Annals of the Institute
of Statistical Mathematics, 31(1): 145–155.


Papangelou F (1972). Integrability of expected increments of point processes and a related random change of scale.
Transactions of the American Mathematical Society, 165: 483–506.


Perelló J, Masoliver J, Kasprzak A and Kutner R (2008). Model for interevent times with long tails and multifractality
in human communications: An application to financial trading. Physical Review E, 78(3): 036108.


Phillips PCB and Yu J (2011). Dating the timeline of financial bubbles during the subprime crisis. Quantitative
Economics, 2(3): 455–491.


Politi M and Scalas E (2008). Fitting the empirical distribution of intertrade durations. Physica A: Statistical
Mechanics and its Applications, 387(8-9): 2025–2034.


Roache SK and Rossi M (2010). The effects of economic news on commodity prices: Is gold just another commodity?
The Quarterly Review of Economics and Finance, 50: 377–385.


Samuelson PA (1965). Proof that properly anticipated prices fluctuate randomly. Industrial Management Review, 6:
41–49.


Scheffer M (2009). Critical Transitions in Nature and Society. Princeton NJ, Princeton University Press.


Sornette D (2003). Why Stock Markets Crash: Critical Events in Complex Financial Systems. Princeton NJ, Princeton
University Press.


Sornette D (2006). Critical Phenomena in Natural Sciences. Chaos, Fractals, Selforganization and Disorder:
Concepts and Tools. Berlin, Springer.


Sornette D and Johansen A (1998). A hierarchical model of financial crashes. Physica A: Statistical Mechanics and
its Applications, 261(3-4): 581–598.


Sornette D and Johansen A (2001). Significance of log-periodic precursors to financial crashes. Quantitative Finance,
1(4): 452–471.


Sornette D, Malevergne Y and Muzy JF (2003). What causes crashes? Risk, 16(2): 67–71.




36


Sornette D and Utkin S (2009). Limits of declustering methods for disentangling exogenous from endogenous events
in time series with foreshocks, main shocks, and aftershocks. Physical Review E, 79(6): 061110.


Sornette D and Woodard R (2010). Financial bubbles, real estate bubbles, derivative bubbles, and the financial and
economic crisis. Proceedings of APFA7 (Applications of Physics in Financial Analysis), Conference series
entitled Applications of Physics in Financial Analysis focuses on the analysis of large-scale economic data.


Sornette D, Woodard R and Zhou WX (2009). The 2006–2008 oil bubble: Evidence of speculation, and prediction.
Physica A: Statistical Mechanics and its Applications, 388(8): 1571–1576.


Soros G (1987). The Alchemy of Finance: Reading the Mind of the Market. New York NY, John Wiley & Sons.
Stoll HR and Whaley RE (2010). Commodity index investing and commodity futures prices. Journal of Applied


Finance, 20(1): 7–46.
Stoll HR and Whaley RE (2011). Commodity index investing: Speculation or diversification? The Journal of


Alternative Investments, 14(1): 50–60.
Sussman A, Tabb L and Iati R (2009). US Equity High Frequency Trading: Strategies, Sizing and Market Structure.


TABB group.
Tang K and Xiong W (2012) Index Investment and Financialization of Commodities, Financial Analysts Journal,


68(6): 54–74.
Toke IM (2011). “Market making” in an order book model and its impact on the spread. In: Abergel F, Chakrabarti


BK, Chakraborti A and Mitra M, eds. Econophysics of Order-Driven Markets, Milan, Springer-Verlag: 49–64.
UNCTAD (2009). Trade and Development Report, 2009. Responding to the global crisis: Climate change mitigation


and development. United Nations publications. Sales No. E.09.II.D.16, New York and Geneva.


UNCTAD (2011). Price Formation in Financialized Commodity Markets: The Role of Information. New York and
Geneva, United Nations, June.


Vere-Jones D (1970). Stochastic models for earthquake occurrence. Journal of the Royal Statistical Society. Series
B (Methodological), 32(1): 1–62.


Vere-Jones D and Ozaki T (1982). Some examples of statistical estimation applied to earthquake data I. Cyclic
Poisson and self-exciting models. Annals of the Institute of Statistical Mathematics, 34(1): 189–207.


Zhuang J, Ogata Y and Vere-Jones D (2002). Stochastic declustering of space-time earthquake occurrences. Journal
of the American Statistical Association, 97(458): 369–380.




37


unctad Discussion PaPers


No. Date Author(s) Title


211 October 2013 André Nassif,
Carmem Feijó and
Eliane Araújo


Structural change and economic development: Is Brazil
catching up or falling behind?


210 December 2012 Giovanni Andrea
Cornia and Bruno
Martorano


Development policies and income inequality in selected
developing regions, 1980–2010


209 November 2012 Alessandro Missale
and Emanuele
Bacchiocchi


Multilateral indexed loans and debt sustainability


208 October 2012 David Bicchetti and
Nicolas Maystre


The synchronized and long-lasting structural change on
commodity markets: Evidence from high frequency data


207 July 2012 Amelia U. Santos-
Paulino


Trade, income distribution and poverty in developing
countries: A survey


206 December 2011 André Nassif,
Carmem Feijó
and Eliane Araújo


The long-term “optimal” real exchange rate and
the currency overvaluation trend in open emerging
economies: The case of Brazil


205 December 2011 Ulrich Hoffmann Some reflections on climate change, green growth
illusions and development space


204 October 2011 Peter Bofinger The scope for foreign exchange market interventions
203 September 2011 Javier Lindenboim,


Damián Kennedy and
Juan M. Graña


Share of labour compensation and aggregate demand
discussions towards a growth strategy


202 June 2011 Pilar Fajarnes An overview of major sources of data and analyses
relating to physical fundamentals in international
commodity markets


201 February 2011 Ulrich Hoffmann Assuring food security in developing countries under the
challenges of climate change: Key trade and development
issues of a fundamental transformation of agriculture


200 September 2010 Jörg Mayer Global rebalancing: Effects on trade flows and employment
199 June 2010 Ugo Panizza,


Federico Sturzenegger
and Jeromin Zettelmeyer


International government debt


198 April 2010 Lee C. Buchheit and
G. Mitu Gulati


Responsible sovereign lending and borrowing


197 March 2010 Christopher L. Gilbert Speculative influences on commodity futures prices
2006–2008


196 November 2009 Michael Herrmann Food security and agricultural development in times of
high commodity prices


195 October 2009 Jörg Mayer The growing interdependence between financial and
commodity markets


194 June 2009 Andrew Cornford Statistics for international trade in banking services:
Requirements, availability and prospects


193 January 2009 Sebastian Dullien Central banking, financial institutions and credit creation
in developing countries


192 November 2008 Enrique Cosio-Pascal The emerging of a multilateral forum for debt
restructuring: The Paris Club


191 October 2008 Jörg Mayer Policy space: What, for what, and where?




38


No. Date Author(s) Title


190 October 2008 Martin Knoll Budget support: A reformed approach or old wine in new
skins?


189 September 2008 Martina Metzger Regional cooperation and integration in sub-Saharan Africa
188 March 2008 Ugo Panizza Domestic and external public debt in developing


countries
187 February 2008 Michael Geiger Instruments of monetary policy in China and their


effectiveness: 1994–2006
186 January 2008 Marwan Elkhoury Credit rating agencies and their potential impact on


developing countries
185 July 2007 Robert Howse The concept of odious debt in public international law
184 May 2007 André Nassif National innovation system and macroeconomic policies:


Brazil and India in comparative perspective
183 April 2007 Irfan ul Haque Rethinking industrial policy
182 October 2006 Robert Rowthorn The renaissance of China and India: implications for the


advanced economies
181 October 2005 Michael Sakbani A re-examination of the architecture of the international


economic system in a global setting: Issues and proposals
180 October 2005 Jörg Mayer and


Pilar Fajarnes
Tripling Africa’s Primary Exports: What? How? Where?


179 April 2005 S.M. Shafaeddin Trade liberalization and economic reform in developing
countries: structural change or de-industrialization?


178 April 2005 Andrew Cornford Basel II: The revised framework of June 2004
177 April 2005 Benu Schneider Do global standards and codes prevent financial crises?


Some proposals on modifying the standards-based approach
176 December 2004 Jörg Mayer Not totally naked: textiles and clothing trade in a quota


free environment
175 August 2004 S.M. Shafaeddin Who is the master? Who is the servant? Market or


Government?


174 August 2004 Jörg Mayer Industrialization in developing countries: some evidence
from a new economic geography perspective


173 June 2004 Irfan ul Haque Globalization, neoliberalism and labour
172 June 2004 Andrew J. Cornford The WTO negotiations on financial services: current


issues and future directions
171 May 2004 Andrew J. Cornford Variable geometry for the WTO: concepts and precedents
170 May 2004 Robert Rowthorn and


Ken Coutts
De-industrialization and the balance of payments in
advanced economies


169 April 2004 Shigehisa Kasahara The flying geese paradigm: a critical study of its
application to East Asian regional development


168 February 2004 Alberto Gabriele Policy alternatives in reforming power utilities in
developing countries: a critical survey


167 January 2004 Richard Kozul-Wright
and Paul Rayment


Globalization reloaded: an UNCTAD Perspective


166 February 2003 Jörg Mayer The fallacy of composition: a review of the literature
165 November 2002 Yuefen Li China’s accession to WTO: exaggerated fears?
164 November 2002 Lucas Assuncao and


ZhongXiang Zhang
Domestic climate change policies and the WTO




39


No. Date Author(s) Title


163 November 2002 A.S. Bhalla and S. Qiu China’s WTO accession. Its impact on Chinese
employment


162 July 2002 Peter Nolan and
Jin Zhang


The challenge of globalization for large Chinese firms


161 June 2002 Zheng Zhihai and
Zhao Yumin


China’s terms of trade in manufactures, 1993–2000


160 June 2002 S.M. Shafaeddin The impact of China’s accession to WTO on exports of
developing countries


159 May 2002 Jörg Mayer,
Arunas Butkevicius
and Ali Kadri


Dynamic products in world exports


158 April 2002 Yılmaz Akyüz and
Korkut Boratav


The making of the Turkish financial crisis


157 September 2001 Heiner Flassbeck The exchange rate: Economic policy tool or market price?
156 August 2001 Andrew J. Cornford The Basel Committee’s proposals for revised capital


standards: Mark 2 and the state of play
155 August 2001 Alberto Gabriele Science and technology policies, industrial reform and


technical progress in China: Can socialist property rights
be compatible with technological catching up?


154 June 2001 Jörg Mayer Technology diffusion, human capital and economic
growth in developing countries


153 December 2000 Mehdi Shafaeddin Free trade or fair trade? Fallacies surrounding the theories
of trade liberalization and protection and contradictions in
international trade rules


152 December 2000 Dilip K. Das Asian crisis: Distilling critical lessons
151 October 2000 Bernard Shull Financial modernization legislation in the United States –


Background and implications
150 August 2000 Jörg Mayer Globalization, technology transfer and skill accumulation


in low-income countries
149 July 2000 Mehdi Shafaeddin What did Frederick List actually say? Some clarifications


on the infant industry argument
148 April 2000 Yılmaz Akyüz The debate on the international financial architecture:


Reforming the reformers
147 April 2000 Martin Khor Globalization and the South: Some critical issues
146 February 2000 Manuel R. Agosin and


Ricardo Mayer
Foreign investment in developing countries: Does it
crowd in domestic investment?


145 January 2000 B. Andersen,
Z. Kozul-Wright and
R. Kozul-Wright


Copyrights, competition and development: The case of
the music industry


Copies of UNCTAD Discussion Papers may be obtained from the Publications Assistant, Macroeconomic and
Development Policies Branch (MDPB), Division on Globalization and Development Strategies (DGDS), United
Nations Conference on Trade and Development (UNCTAD), Palais des Nations, CH-1211 Geneva 10, Switzerland;
e-mail: gdsinfo@unctad.org; fax no: +41 22 917 0274.


UNCTAD Discussion Papers are accessible on the website at http://unctad.org.






Login