Information theoretic approaches to characterizing complex systems, part 1: complexity of reconstructed epsilon machines

Introduction

In earlier posts, I opined that the behaviour of various climate subsystems showed greater complexity in the Late Pleistocene than in the early Pleistocene. This opinion is shaped by observations of the behaviour of these varying subsystems (Himalayan monsoon strength, global ice volume, and oceanographic conditions) inferred from proxy records up to about 2 million years in length.

Any such argument would be strengthened by a number. So the challenge I will address over the next few posts in this series will be--how to characterize the complexity of the output of a dynamic series by a single number. After all, in order to compare complexity between two periods, it would be helpful to have a single parameter to compare.

The principal information theoretic concept we shall use is Shannon's (1949) measurement of entropy. The trick is deciding how to apply this parameter.

Entropy of the epsilon machine for the ice volume proxy

Let's look at epsilon machine reconstructions of the ice volume proxy.

Three separate first-order epsilon machines describe portions of the Early Pleistocene variations in the ice volume proxy. A1 represents a minimum ice state, A2 is roughly what goes for an interglacial at present, and A3 is the maximum ice state of the early Pleistocene, which would pass for a minor glacial event in the late Pleistocene.

The Mid-Pleistocene epsilon machine looks more complex.

The Late Pleistocene epsilon machine reconstruction for the ice volume looks to be more complex than any of the Early Pleistocene reconstructions. But is it? How can we tell?

The approach we will try here is to characterize the complexity by the entropy of all of the state transitions. Entropy is expressed as -Σp(i)log p(i) for all values of p(i).* Entropy is considered to be a measure of the "information" in a stream of data. This expression is normally applied in systems where Σp(i) = 1, a condition not met in the figures above. The probabilities of each pathway leading from each of the predictive states adds up to 1; so that the total of all "probabilities" adds up to the number of predictive states in the epsilon machine (two or three in the Early Quaternary, four in the mid Quaternary, six in the late Quaternary.


Do we add up the probabilities as they appear? Should we divide all probabilities by the total number of predictive states so we end up with Σ p(i) = 1? Should we weight the various probabilities to reflect the relative importance of an individual predictive state?


Let's see what happens.


First off, consider a system based not on the proxy data, but on a model. Say, the Late Quaternary global ice volume model of Paillard (1998).

Quite provocative given the current state of the economy! Actually, the I stands for interglacial regime, the M for mild glacial regime, and the F for full glacial regime. The system bumps along from state to state, but there are no probabilities listed as there is only one possible successor state from each predictive state.


Is the above system more complex, less complex, or the same as one with a single state--say, "I".


From a dynamic system, which would use topological arguments--both systems would be equally complex (or  simple, in this case), as there is no choice of successor state from each predictive state. From an information sense, it is not at all clear the two systems are the same.


I M F I M F I M F I M F I M F I M F . . . 


I I I I I I I I I I I I I I I I I I I I I I . . .


It depends on whether you allow yourself to 'group' the Is, Ms, and Fs into words, which repeat. From a geological perspective, there is a difference in the complexity between the two systems--having three separate predictive states is different than having a single repeated predictive state. 


However, if we calculate entropy [-Σp(i)log p(i)] for all the states in both systems, we come up with a value of zero. This is because the probability of each transition (I → M, M→ F, F → I or I → I in the second example) is 1. If, on the other hand, we establish the probability of each of the transitions (I → M, M → F, and F → I) as 1/3, then the entropy is 1.585, as compared to zero for the system with a single predictive state.


The implied complexity is 3x greater for the I M F system as compared to I. Seems reasonable. Now let's consider the epsilon machine construction for the Early Quaternary ice volume proxy.

There are two possibilities to recalculating the probabilities of each transition for α1, for instance: we could divide each of the probabilities by by the number of predictive states (remembering that the probability on the unlabelled A1 → A3 arrow is 1), or we could multiply the probability of each transition by the probability of the originating predictive state. In the interval from 1870-1700 ka, we find p(A1) = 0.2, p(A2) = 0.4, p(A3) = 0.4.


By method 1, the entropy for α1 is 2.13. By method 2, the entropy for α1 is 2.17. Not too different.


By both methods, the entropy for both α2 and α3 is 1.


For the Mid Pleistocene, the entropy for α4 (by method 1) is 3.23. We observe p(A1) = 0.19, p(A2) = 0.125, p(A3) = 0.31, p(A4) = 0.375, so by method 2, entropy for α4 is 3.21. Again, not much different from method 1.


For the Late Pleistocene, the entropy of α5 (by method 1) is 3.42. We observe p(A1) = 0.027, p(A2) = 0.243, p(A3) = 0.243, p(A4) = 0.297, p(A5) = 0.162, p(A6) = 0.027. The entropy of α5 is 2.85, which is considerably lower than by method 1. I think this is because observations of A1 and A6 are rare, as these predictive states are only observed once each during the Late Pleistocene.


Entropy of epsilon machines for paleomonsoon strength proxy


Now consider the reconstructed epsilon machines for the paleomonsoon strength proxy. We shall only use method 2 in calculating entropy.

Three predictive states in the Early Quaternary, dominated by M1 and M2. Given p(M1) = 0.50, p(M2) = 0.41, p(M3) = 0.09, then the entropy of μ1 is 1.83.

In the Late Quaternary, there are six predictive states, with observed probabilities as follows: p(M1) = 0.4, p(M2) = 0.2, p(M3) = 0.17, p(M4) = 0.1, p(M5) = 0.1, p(M6) = 0.03. The entropy of μ2 is 3.67.


Conclusions


Method 1 is the easier calculation, however method 2 is a better calculation. However, method 1 can be used as long as the distribution of predictive states is not too far from even.


In summary


   Time                                             Entropy (ice volume)       Entropy (paleomonsoon)


Late Pleistocene                                    2.85                                    3.67


Mid-Pleistocene                                     3.21                                   1.83


Early Pleistocene                                   1-2.1                                   1.83


By this test, the behaviour of the climate system has been more complex in the Late Pleistocene than it was in the Early Pleistocene.


In our next installment, we look at how we can use the characterize the complexity for the probability density calculation for each window shown here, to give us a nice smooth graph of complexity of the climate system through time.


References



Crutchfield, J. P., 1994. The calculi of emergence: Computation, dynamics, and induction. Physica D 75: 11-54.
Gipp, M. R., 2001. Interpretation of climate dynamics from phase space portraits: Is the climate system strange or just different? Paleoceanography, 16, 335-351.
Kukla, G., Z. S. An, J. L. Melice, J. Gavin, and J. L. Xiao, 1990. Magnetic susceptibility record of Chinese loess. Trans. R. Soc. Edinburgh Earth Sci., 81: 263-288.
Paillard, D., 2001. Glacial cycles: Toward a new paradigm. Reviews of Geophysics, 3: 325-346.
Shackleton, N. J., A. Berger, and W. R. Peltier, 1990. An alternative astronomical calibration of the Lower Pleistocene timescale based on ODP site 677, Trans. R. Soc. Edinburgh, Earth Sci., 81: 251-261.
Shannon, Claude (1949). "Communication Theory of Secrecy Systems". Bell System Technical Journal 28 (4): 656–715.



* We calculate all logarithms in a base of 2, in accordance with the nerds who came up with this concept.

Flash crash the nat gas!

As shown on posts on Zero Hedge, (correction, these are originally from Nanex) there have been some bizarre patterns in the trading of natural gas in the past couple of days. The charts below are from the evening of June 8, 2011 and come from the first of the two articles linked above:

At first glance this looks like nearly perfect chaos.

The last figure represents a one-dimensional projection of a Lorenz butterfly curve, shown in its glory below.

In reality the trading data isn't as nice as it first appears. There has been a bit of playing around with the time axis on the first two plots. I have subsequently digitized the data with trades at half-second intervals (but I'll outline the caveats below).

The axis on the bottom is time in seconds, starting from 19:40:37 on 08-Jun-11. The digitization is at half-second intervals, because the analysis below requires evenly spaced data. There were some difficulties, however. There was not always a trade right on the half- or full-second mark. Frequently the two nearest trades on either side of the desired time interval were at the same price so that it would be reasonable to use that price. Sometimes, however, the two nearest prices were quite far apart--for these I used a midpoint between bid and ask at the moment--however arguably this is not a price, particularly when we observe that many trades took place either above the ask or below the bid. Additionally, there are intervals where the midpoint between the bid and ask is actually undefinable, as during the interval from 19:42:04 to 19:42:06 where the bid price fluctuates in a complex fashion while the ask remains constant. So there are risks in this analysis.

The two dimensional lagged state space (using a 2-s lag to minimize mutual information) looks as below:

Not quite as beautiful as the Lorenz butterfly curve. I smoothed the data through a 3-point moving average filter as the original really looked like hell. The sinusoidal waves that slowly increase in size are reflected in the two-d state space as a spiral, drawn from the centre outwards, until the curve flies off into a new area of phase space.

The conventional dynamic explanation of the nat gas trading curve would be of a system in equilibrium--but the equilibrium is unstable, and the wobbles get progressively larger until the system shifts to a new equilibrium. Such a dynamic interpretation is incorrect. The Lorenz equilibrium is all equilibrium. What we perceive as a sudden shift in equilibrium is actually part of the equilibrium state.

Disequilibrium--it's new equilibrium.

From a trading perspective, the nat gas trading is more straightforward. Someone is able to profit from variability. The proper positioning of puts and calls may allow you to benefit by a large move (either up or down) in the price of a stock or commodity. After that, you can try to hit any buy or sell stops if you are able to drive the price up or down, (or in this case, both ways, until the sell stops were hit triggering a cascade in price). If there had been more buy stops, then the price would have melted up. It was simply a matter of luck that the price melted down.

As in all markets there are two (or more) participants. Leaving the actual trades for a minute and scrutinizing the fluctuations in the bid and ask, we see a complicated psychological game being played. I see many of the tricks that I have seen in thinly traded gold stocks priced by a market maker. You see the bid (or trade price) creep up towards the ask price, then run away, perhaps hoping to draw the ask price down.

But the time frame is completely different. I used to see this play out over the course of a trading day (my favourite was when I would get a partial fill of, say 500 shares of some penny stock before the price would swing away, and could imagine the market maker saying "do you want to pay full transaction fees for a $100 sale, or are you going to meet my price and fill your order?"). In this case, these games are being played on a split-second scale.

Game theory has been digitized and is running on a level of complexity that leaves TIT FOR TAT in the dust. Unfortunately my textbook on the subject is being used by my seven-year-old, so I can't get into strategy games between different players.

Many of the price rises happen while trades are occurring above the ask price. How does that happen. Is the asker crossing transactions with fictional entities to lure the buyer? Similarly, many trades occur below the bid price as the price is falling. Are these fictional crosses? Are these real trades or just a gimmick to lure in another party (hurry up and buy--look how fast price is falling!).

The chaotic appearance of this function is simply an emergent property of the gaming algorithms.

Lastly, I will point out that my analysis is roughly seven orders of magnitude slower than the frequency of some of the trades.

Modeling and the axiomatic approach to science

I have previously written on the problems that arise from taking an axiomatic approach to science.

To review once again--science proceeds through the testing of falsifiable hypotheses, which, if falisified, are rejected. If numerous attempts to falsify an hypothesis fail, we begin to accept that the hypothesis explains some set of observations, and this explicatory power moves into the scientific pantheon as a theory.

The axiomatic approach is quite different than the normal scientific approach of testing hypotheses. The question today is whether the modeling approach lends itself to hypothesis-testing or is purely deduction from axioms. My interest in this question stems from the recent GAC symposium on "Earth climate: past, present, and future", moderated by Dr. Andrew Miall.

Our question today concerns climate modeling. Is it inherently axiomatic (i.e., based on non-falsifiable non-reducible logical statements)? Arguments in favour of this view include: 1) whereas an inductive scientist may admit a relationship between a pair of parameters, once the relationship is included in a deterministic model, that relationship is quantified; 2) once the relationship between two parameters is quantified, that quantification tends not to change unless challenged by specific observations compared to the model; 3) specific comparisons usually cannot be used to test any one particular quantified relationship in a complex model--there are ambiguities; 4) the models (here I am specifically looking at GCM models) tend to be very difficult to test, as the required simultaneous observations are difficult to come by.

The Thursday morning session was contentious, with talks by Ian Plimer, Jan Veizer, Hans Svensmark (pre-recorded, unfortunately), each of which could be construed as "climate denying". These talks focussed on points or particular weakness in the climate warming arguments, particularly the effects of solar and cosmic effects on climate. Interestingly, the Svensmark talk dealt with very long-term effects (140 my time-scale), which may be reflected in the 28 My cyclicity in mass extinctions Raup and Sepkoski's work on mass extinctions.

[If the extinctions are instead related to the passage of the solar system through relatively denser portions of the galactic arms (140 my cyclicity), and if extinctions were related to this event (instead of Nemesis), the spectral power observed in the 28-my periodicity could be explained as a harmonic of 140-my cyclicity.]

Norman Kalmanovitch discussed observations of the blackbody radiation escaping from the Earth increasing in tandem with warming, suggesingt that warming is externally driven rather than due to enhanced trapping of heat by greenhouse gases (whereupon one would expect escaping radiation to decrease with warming).

The theme of solar influence on climate continued in the afternoon session. Important points of the afternoon session were that the Little Ice Age was not entirely a European phenomena, but appears in paleoclimatic records of other disparate areas around the world; and climate models tend to overestimate the variation in temperature.

The audience was much smaller for the Friday morning session, which dealt mainly with paleoclimatic records.

The first two talks of the session showed the importance of realistic understanding of future climate change, as accurate predictions of future thickness of the ice roads that service several diamond mines in the Northwest Territories of Canada has a clear economic impact.

My talk was largely a reiteration of previous posts.

The last talks dealt with earlier portions of the global record.

The debate which followed was enjoyable but one-sided. As commonly happens, potential participants to these conferences draw their cues from the moderator and the list of talks, and those with a pro-IPCC viewpoint basically stayed away from this session.

On the basis of the geological record, it is reasonable to conclude that increasing atmospheric CO2 will lead to increased global temperature. How much of an increase is the question.

Even as you construct a climate model, you may be aware that there is a large range of possible values expressing this correlation. But when you create a model, you must select a value, even if you don't really believe it. You may not strongly defend your choice of parameter. But if your model becomes famous, and a lot of other people begin to use it, then you defend the model vigorously, not so much because your own reputation is at stake, but rather the reputations of all of your followers. This is the influence of the human condition on the scientific method.

Unemployment data dives further into the black hole

Today we reprise the image from Captain Bernanke's black hole misadventure to show the nature of the ongoing "recovery".

I've increased text size for the last few months. Note the real plunge after January '11. The last few points are affected by a change in methodology wherein the BLS no longer caps unemployment duration at two years, but rather at five years.

The blue curve near the bottom represents the data using the old methodology (for you purists who insist on consistent methodologies).

The terrifying correction in silver . . .

. . .doesn't seem so bad.

The last month's activity (yellow curve) still shows we are a long way from the breakout of a Lyapunov-stable area for the gold-silver ratio that had held for the last ten years. It will take a lot more technical damage to bring us back to the former area of stability.

In dynamic systems, once you cross the separatrix, it is difficult to put the system back into the same state. It happened in 1998 after the Warren Buffet affair, but there may have been furious behind-the-scenes activity at that time. For this breakout to fail, the gold-silver ratio will have to return to approximately 50. So keep heart.