Efficient Markets and Deep Learning

I'm going to posit a theory about markets which will be hard to prove or disprove, except, perhaps, in hindsight, which may take many years.

I believe markets may have become substantially more efficient in recent years due to deep learning. Because I am not an expert on either thing (efficient markets, deep learning), and because I am lazy, and because I believe it will thankfully be clearer this way, I am going to keep the theory mercifully brief.

Efficient markets theory ("EMT") states that market prices of securities tend to reflect all known pertinent information about those securities. You can also state it by expressing its opposite, which is that any disregard for information will be 'arbitraged' away by other market participants - this is the magnetizing mechanism by which markets are pulled towards 'efficiency.' People love (or at least, used to love) arguing about how strictly 'efficient' they are. Basically, is it EMT, or EMT, or EMT? But, that is rather beside the point, here.

Deep learning is a form of machine learning, or artificial intelligence, which detects and adapts to patterns in a recursive manner. Experts, please unmercifully assault that definition for being too broad, or too narrow, or whatever.

The stock market is interesting because, if you believe people are motivated by greed (which is to say, money), its feedback loop between effort (security selection) and reward (making money) is shorter than virtually any other feedback loop that delivers money as a reward. Therefore, if you believe people are motivated by greed, it stands to reason they'll prefer this feedback loop to others, all else being equal. That means they'll employ a lot of effort to gain an edge. Or, put another way, they'll employ tactics here, first, at least until it stops working.

I don't think any of the above should be very controversial.

Deep learning is interesting because it seems to solve certain classes of problems better than other methods, and notably, better than human judgment - data search, speech recognition, the game "Go". How can we broadly classify these categories? Well, for one, it seems to work well with so-called "dynamic systems," which tend not to move in straight lines, but bounce between "multiple equilibria," which is to say, behave a certain way until they don't. The stock market is a dynamic system. The stock market is also notoriously irrational - meaning, it is a dynamic system whose equilibria are a function of human judgment, which is notoriously flawed. Importantly, human judgement isn't just flawed - it's flawed in predictable ways, which means that rules-based systems can be expected to identify those patterns, to some degree.

What does it add up to? If I'm right about the above (I might not be!), is deep learning being used in the stock market? Almost certainly. What would the effect be? I suspect it would lead to more efficient markets.

Market efficiency is a function of the marginal buyer. If 99% of the capital is controlled by 'base case' rules, and 1% allocated in some 'better' manner, well, market efficiency might be fairly low. As the 'better' percentage rises, efficiency increases. How soon does the market truly get efficient? As a whole, I'm not sure. For a single security, it gets efficient when the better manner reaches a threshold amount of the trading volume (NOT the overall holding of the security).

I think my argument makes some sort of conceptual sense, but I can't quantify it. And it's true, admittedly, that I may simply be experiencing confirmation bias linked in my head to a convenient fear, during a period of time when value stocks - by most quantitative measures - have underperformed the broad market. This underperformance has happened before. I feel this underperformance because my investment style skews towards value, by those same factors.

For what it's worth (and to skate away from that topic a fair distance), my gut tells me that since I can't quantify the risk, I should be pursuing more private investments, where the market efficiency is more likely to still be low. I could conjure plenty of supporting justification for such a decision, but that seems like a silly and unnecessary thing to do. If I am right, the above theory is all the justification that should be necessary.